/srv/irclogs.ubuntu.com/2012/05/15/#launchpad-dev.txt

lifelesswgrant: flat and flat2 ?00:04
wgrantlifeless: 2 is just a second temporary index on that table00:05
wgrantgin (access_grants) and gin (access_policies) respectively.00:06
wgrantlifeless: So, BugSummary v2 needs to avoid the journal write slowness that we see today, which probably means not having an exploded journal, instead just journalling the new and old states. This leaves a couple of options for displaying up to date numbers: rely on garbo-frequently to keep the table up to date, and hope that users don't notice the numbers are a couple of minutes out of date; or use array indices to quickly count the ...00:10
wgrant... relevant non-exploded rows in the journal.00:10
wgrantThoughts?00:10
=== vednis is now known as mars
lifelesswgrant: do we know why its slow?01:05
lifelesswgrant: when we originally did it it was under some time pressure01:06
lifelessso it probably hasn't had even a single pass optimisation run over it01:06
wgrantlifeless: Say I have an 80-task Linux task, because the kernel team are evil.01:14
wgrantIt has 6 tags.01:14
wgrantEach change to the bug is likely to generate 960 journal rows01:15
wgrantIf it's a security bug, it might be private.01:15
wgrant5 subscribers01:15
wgrant5000 journal rows01:15
wgrantThis goes into the list of "things that probably don't scale very well"01:17
StevenKwallyworld, wgrant: No complaints about my mail? I'm shocked.01:23
wgrantlifeless: eg. each change to bug #732628 that requires a BugSummary update (eg. adding a tag) takes 650ms to journal 770 rows.01:26
_mup_Bug #732628: TOCTOU in mount.ecryptfs_private <kernel-cve-tracking-bug> <patch> <eCryptfs:Fix Released by kirkland> <ecryptfs-utils (Ubuntu):Fix Released by kirkland> <linux (Ubuntu):Fix Released> <linux-ec2 (Ubuntu):Invalid> <linux-fsl-imx51 (Ubuntu):Invalid> <linux-linaro (Ubuntu):Won't Fix> <linux-lts-backport-maverick (Ubuntu):Invalid> <linux-lts-backport-natty (Ubuntu):Invalid> <linux-lts-backport-oneiric (Ubuntu):Invalid> <linux-mvl-dove01:26
wgrantAdd 3 tags, 2970 rows total.01:26
wallyworldStevenK: haven't read it yet01:37
StevenKTypical. :-P01:37
wallyworldStevenK: seems a reasonable summary. devil will be in the detail01:39
lifelesswgrant: how many are -1+1 pairs ?01:44
wgrantlifeless:  count | count01:46
wgrant-------+-------01:46
wgrant     1 |  165001:46
wgrant    -1 |  132001:46
lifelessso, lots02:07
lifelessis there some way we could filter those out ?02:07
wgrantSure.02:08
lifelessIIRC we call two helpers, one to un-count thebug and one to recount it02:08
wgrantWe can use our sensible data access layer to not do this in the DB :)02:08
wgrant*cough*02:08
lifelesssimple but apparently high overhead02:08
wgrantlifeless: The difficulty is that we don't know when the update is done.02:09
wgrantSo we can't avoid the writes.02:09
lifelesswe can reduce them02:10
wgrantIf our data access layer existed we could implement it there.02:10
lifelessindeed02:10
lifelessso, at the moment02:10
lifelessUPDATE FOO02:10
lifeless-> trigger which gets old, new02:10
wgrantYes02:10
lifelessold gets dereferneced, new gets referenced02:10
lifelessthat trigger could optimise02:10
wgrantHow?02:11
wgrantIt can't potentially delete the old ones.02:11
wgrantOh, I guess it could.02:11
wgrantSince they're not committed yet.02:11
wgrantThat would almost halve the writes.02:12
lifelesswe could have the helpers return the tuples, not add them, calculate in-memory, then write out the remainder02:12
wgrantWe could.02:16
lifelesswould that be sufficient?02:16
wgrantThere's no persistent cross-statement storage for that purpose, apart from temp tables.02:17
lifelessits all one statement02:17
lifelessthe UPDATE (or add/delete in the case of tags)02:18
wgrantTrue. No cross-trigger storage, then.02:18
lifelesswill it truely optimimal it pathological cases? (email command sequences) - no, but does that matter?02:19
wgrantlifeless: INSERTing two tags will fire each trigger twice.02:20
wgrantHow do you propose to optimise that?02:20
lifelessdo we need to? do we know we need to?02:21
wgrantI don't understand how your optimisation thing can work at all.02:21
lifelessINSERT INTO BugTag knws it only needs the +1s02:21
wgrantOh, for that case, sure.02:22
wgrantBut now I change an attribute of the bug.02:22
lifelessthats an UPDATE02:22
wgrantAnd it has to journal both sides.02:22
lifelesswhy ?02:22
wgrantBecause it's removing from some counts, and adding to others?02:22
lifelessit has to journal -1s for the no longer relevant summary rows02:23
lifelessbut it doesn't have to journal -1s for every summary row02:23
wgrantAnyway, do we know that this journalling approach has benefits over my proposal?02:23
wgrantMine is a lot easier on writes and probably easier on reads too.02:24
lifelessso your proposal is 'journal the old,new inputs' ?02:25
wgrantYes.02:25
lifelessthat will be pretty big too02:25
wgrantThe number of writes scales linearly with number of tasks.02:26
lifeless+ subscribers + tags02:26
wgrantAnd each will only be slightly larger than a single existing journal row02:26
wgrantYes.02:26
wgrantWell, not subscribers, grants/polciies.02:26
lifelesswhich is a lot larger than a single journal row02:26
wgrantFor things with a lot of grants or tags, the row will be a fair bit larger, yes.02:27
lifeless2*bugtask + 2*tags + 2*policies02:27
lifelessso this case, will be 160 bugtasks, + the other bits02:28
wgrantCurrently it's bugtask*tags*(policies+subscribers)02:28
lifeless* summary width02:28
lifeless*202:28
lifeless(for non-delete/add cases)02:29
lifelessjournaling the state will be 2*(bugtasks*bigtask width+area(tags) + policies*grant width)02:30
lifelesswgrant: anyhow, I've no objection if you want to journal an array of bugtasks + array of tags + array of policies and adjust accordingly.02:32
lifelesswgrant: I'm not convinced its better (or worse)02:32
lifelesswgrant: but thats because I haven't modelled it sufficiently02:32
wgrantlifeless: The journalled thing would be a subset of bugsummary + array of tags.02:32
wgrantSo roughly (subset of bug, subset of bugtask, [tags], [grants], [policies])02:33
lifelessso when a tag changes, you'll journal multiple rows?02:36
wgrantlifeless: Yes, since I can't efficiently index into inhomogenous arrays like an array of task attributes.02:37
wgrant(or even filter, if we have few enough rows to not care about indices)02:38
lifelessso, how much space does 160 of these rows take02:38
lifelessfor someone adding a tag and deleting a tag02:39
wgrantI can never remember the precise disk format of arrays.02:43
wgrantBut IIRC it's just a 4 byte size followed by the fields.02:43
wgrantSo the row size increase for the bug I mentioned above will be (#tasks * (task width + 4 bytes + all the tags)).02:44
wgrantCurrently we have approx (#tasks * #tags * (task width + avg tag size))02:44
wgrantAlso the base postgres per-row overhead is not insignificant :)02:47
wgrantThe postgres overhead makes up >30% of the average bugsummary tuple size now.02:48
wgrantLikely around 50%, in fact.02:49
wgrantAverage bugsummary tuple is 62 bytes (including header), header is 27 bytes + a couple of bytes + 8 byte alignment.02:50
lifelesscould we do away with bugsummary ?03:23
wgrantNo.03:24
StevenKlifeless: After you spent all that time writing it, and then rewriting it six times?03:25
lifelessI suspect, due to writing just a few bugsummaries, that coalescing around statements would be a bigger win overall, but it may be more work03:27
lifelessStevenK: Its a tool, not an end goal.03:27
lifelessStevenK: dropping it if its no longer a win would be entirely appropriate03:27
wgrantlifeless: Coalescing around statements?03:28
StevenKlifeless: I'm teasing03:29
lifelesswgrant: what I described before03:29
lifelesswgrant: anyhow, the constraints I see are:03:30
wgrantlifeless: I don't see how thta does away with bugsummary...03:30
lifeless - alterations do not timeout03:30
lifeless - things work.03:30
lifelessdone03:30
lifelessbeyond that I've no particular view on it03:30
wgrant13:23:39 < lifeless> could we do away with bugsummary ?03:31
wgrant13:27:21 < lifeless> I suspect, due to writing just a few bugsummaries, that coalescing around statements would be a bigger win overall, but it may be more work03:31
wgrantAre those two statements unrelated?03:31
lifelessno03:35
lifelessyou said no03:35
lifelessso I went back to discussing how to evolve it03:35
wgrantHm05:06
wgrantDoes https://launchpad.net/ubuntu/+patches do anything that a bug search for latest_patch_uploaded IS NOT NULL ordered by latest_patch_uploaded and showing a new "Patch age" field wouldn't do?05:07
wgrantI think I might add the 4 lines of code needed to support that on +bugs, and delete the view.05:09
StevenKIs it linked from anywhere, and do people use it?05:10
wgrantIt is linked from the portlet, and people do use it.05:10
wgrantI would change the link to point to a search, like the rest of them already do.05:10
StevenKOh, right.05:10
StevenKSounds good, then.05:10
wgrantAh, hm.05:12
wgrantThe Patch age column is a link with a tooltip.05:12
wgrantAnd dynamic bug listings pull all the data every time, so it would be a little inefficient to pull all the LFAs.05:12
StevenKsteven@undermined:~% indicator-weather05:31
StevenKAnother instance of this program is already running05:31
StevenKzsh: exit 1     indicator-weather05:31
StevenKRARGH05:31
* StevenK stabs it.05:31
bigjoolsI gave up  with that05:32
StevenKIt keeps crashing and bringing up apport too05:33
lifelesssent an error report ?05:33
lifelesswgrant: can annotate the LFAs in a post-load hook, but yea, not trivial05:34
wgrantlifeless: We can, but it's still a fair bit more data to get from the DB and send to the browser.05:35
wgrantFor something that's used on probably 0.1% of requests.05:35
StevenKlifeless: I've sent two or three05:35
lifelesswgrant: right, thats why I'm saying load it only when used05:36
wgrantlifeless: The infrastructure doesn't support that :(05:37
wgrantWould need significant changes.05:37
lifelesswgrant: I don't think so: we do a very similar thing for linkification05:38
lifelesswgrant: get page -> after render -> send request -> markup results05:39
wgrantlifeless: Oh, true, could do it like that.05:39
lifelessis it pretty and integrated and all that - no05:40
lifelesswould it be nice to do that? yes.05:40
lifelessbut ...05:40
lifelessit would be doable and fairly small to code05:40
lifelessand likely more efficient overall05:40
wgrantYeah.05:40
lifelesswgrant: voice call - bugsummary2 ?05:41
=== jtv1 is now known as jtv
wgrantSure05:41
wgrantI'm online05:42
lifelessdoes db-link-sppu bypass auditord?07:28
stubOh my god, its full of jargon...07:29
StevenKlifeless: No?07:29
StevenKIt is complementary, it just doesn't depend on it07:29
lifelessStevenK: ah, it just being on the bug that the service was discussed on confused me.07:35
lifelessStevenK: I thought it was one of the things we'd be storing in auditord07:35
StevenKlifeless: It is certainly related to the bug.07:35
StevenKlifeless: No, we will be storing details relating to the PU in auditor, so we need a way to get at the PU from SPPH.07:36
lifelesscool cool07:36
stubThe PG 9.2 feature list in the beta2 announcement does look nice. Even the short version.07:36
wgrantOh, that's actually been released?07:39
wgrantI was looking through commit logs to see what had changed.07:39
wgrantBecause the only reference to a changelog was "oh, I'm here's a draft announcement, but I'm still writing the changelog"07:39
wgrantHeh: "I have completed the Postgres 9.2 release notes I started seven days ago."07:40
daniloshi all, anyone able to review https://code.launchpad.net/~danilo/launchpad/bug-999040/+merge/105637 for me?07:42
wgrantstub: Array stats, yay!07:44
wgrantand sensible prepared statement plans.07:44
wgrantAnd of course index-only scans, but they go without saying.07:45
stub"Allow analyze statistics to be collected for foreign tables" could be interesting07:46
wgrantI hope we never use foreign tables :)07:47
wgrantAlso, DROP INDEX CONCURRENTLY.07:47
StevenKOooh07:47
StevenKWANT07:47
wgrantAnd quick CHECK adding07:47
wgrantThe ALTER TABLE lock reduction stuff didn't make it in, though.07:48
adeuringgood morning07:52
wgrantstub: "Have pg_upgrade create a script to incrementally generate more accurate optimizer statistics"07:59
wgrantSo no need for waiting for ANALYZE07:59
daniloswgrant, gmb: you guys are still listed as on-call reviewers for Tuesday on https://dev.launchpad.net/ReviewerSchedule, so I hope you don't mind if I pester you to get https://code.launchpad.net/~danilo/launchpad/bug-999040/+merge/105637 reviewed :)08:01
wgrantdanilos: Done08:05
daniloswgrant, thanks, will you land it for me as well please? (as for LOC addition, before we started this work we [salgado] removed a bunch of LOC, and I am pretty sure we are still a net negative; I'd have to check though)08:06
wallyworldwgrant: question. in a test case, i can makeAccessArtifactGrant for a bug but for a branch i also have to call makeAccessPolicyArtifact in order for findArtifactsByGrantee to work. why is that?08:08
wgrantwallyworld: Branches don't have triggers to set up the APAs.08:09
wgrantwallyworld: Bugs do (for now)08:09
wallyworldok. will put a note in the test. thanks08:10
wgrantdanilos: Sure.08:10
daniloswgrant, thanks08:10
wgrantdanilos: You need to set a commit message.08:11
daniloswgrant, sorry, done08:15
wgrantdanilos: It's on its way.08:16
daniloscool, ta08:16
=== almaisan-away is now known as al-maisan
=== gmb_ is now known as gmb
daniloswgrant, btw, I believe bug 998052 is still at 'fix committed' instead of released08:34
_mup_Bug #998052: Upcoming Work View now showing all desired work items <qa-ok> <Launchpad itself:Fix Released by dooferlad> < https://launchpad.net/bugs/998052 >08:34
wgrantdanilos: Ah, it was deployed but rolled back shortly afterwards due to an unrelated regression. It should be redeployed in a few hours if all goes well.08:36
daniloswgrant, ok, that explains it, thanks08:36
=== Topic unset by gmb on #launchpad-dev
gmbOh arse10:18
=== gmb changed the topic of #launchpad-dev to: http://dev.launchpad.net/ | On call reviewer: gmb | Firefighting: - | Critical bugs: 3.47*10^2
gmbI fail at IRC. Again.10:18
wgrantBlame jetlag.10:19
wgrantAlways jetlag.10:19
=== rick_h_ changed the topic of #launchpad-dev to: http://dev.launchpad.net/ | On call reviewer: gmb, rharding | Firefighting: - | Critical bugs: 3.47*10^2
wgrantdanilos: ec2 seems to hate you. Let me retry it.12:10
daniloswgrant, :(12:10
daniloswgrant, I'll pretend to like it if that's going to help12:11
allenaprick_h_: Thanks for the review! I'll get to work :)12:37
rick_h_allenap: thanks for all the fresh code12:40
rick_h_talk about a wake up in the morning12:40
rick_h_and I feel I lacked as a reviewer to not have anything to complain about on something new/large like that :P12:41
rick_h_jcsackett: ping13:06
jcsacketthola rick_h_13:06
rick_h_jcsackett: so couple of things13:06
jcsackettnever good, that. :-P13:06
rick_h_1) I bet the jsbuild_watch doesn't work for you because you don't have the combo loader FF enalbed in dev?13:06
rick_h_that auto build only builds into the combo load directory and doesn't rebuild launchpad.js13:07
rick_h_too slow to do that13:07
jcsackettaah.13:07
jcsackettdidn't realize that.13:07
jcsackettand no, i didn't set any feature flags.13:08
rick_h_2) Just a heads up, in checking the YUI widget source, the default for the visible property is true, wonder if that was giving you grief after seeing the comments on not listening to the html/etc13:08
jcsackettrick_h_: that's one thing, but i set it to false in ATTRS.13:08
rick_h_jcsackett: ok, that makes me feel better (in a way) and sorry I didn't think of it until last night13:08
jcsackettrick_h_: dig, so i'll set that up, b/c i'm doing more js hacking. :-P13:08
rick_h_jcsackett: yea, I feel like this is a combo issue, at some point I hope to poke at it. I was going to note in my reivew that you didn't need to have the ATTR, but I saw you were setting it false vs true13:09
rick_h_jcsackett: awesome, hopefully it goes smoother then13:09
jcsackettrick_h_: as to the visible, the *other* thing is that listening "on" means the visible change event has happened, but it's not guaranteed visible *has* changed yet.13:09
jcsackettthus, listening to "after" means you're trigger based on the change, but the change is definitely done.13:09
rick_h_jcsackett: yea, there's also afterChange13:09
rick_h_jcsackett: right13:09
jcsackettrick_h_: yeah, it seems stable with this.after('visibleChange'). i owe reading longpoll code for figuring that one out. :-P13:10
rick_h_heh13:10
rick_h_cool, well glad it's fitting together better. Hate it when things *should* work but don't13:11
jcsackettrick_h_: it's not they don't work--it's that i insufficiently understand the parameters in which they *should*. :-P13:14
jcsackettrick_h_: incidentally, apologies for the blitzkrieg of js branches in the queue today. :-P13:14
rick_h_jcsackett: hah, all good, these look pretty now that there's nice module/code reuse13:14
jcsackettyeah, i'm way happier with them.13:14
jcsackettrick_h_: reply to your comments on privacy-banner branch. in particular as regards the build/ in tests. happy to talk here after you've read the comments, or we can converse via MP. :-)13:59
rick_h_jcsackett: k, loading it up14:00
rick_h_jcsackett: so the build thing was something we chatted about in our squad when doing combo loader/test updates.14:02
deryckrick_h_, abentley, adeuring -- sorry, missed it was stand-up.  coming now....14:02
rick_h_the idea is that those are actually 'deps' and should come from a static/standard location like the download-cache kind of idea14:02
rick_h_deryck: ooh, me too14:02
jcsackettrick_h_: except that totally breaks code/reload. i don't know if bin/test rebuilds js everytime, but if you're checking things in the browser (which is way easier than trying to read the error messages from bin/test) you now have to rebuild every time.14:03
rick_h_jcsackett: and that we want to make sure that files don't break during 'build' so that they fail in tests as they would in production14:03
rick_h_if something causes a file to not get put into the build directory right, tests will pass, but production usage will fail14:04
rick_h_jcsackett: right, but the _watch should fix that up tbh14:04
rick_h_and combo loader is coming14:04
rick_h_so it's kind of a convention to help make sure that building will work correctly along with tests passing I guess14:05
rick_h_you're probably right that a -dev discussion is a good idea14:05
jcsackettrick_h_: right, but _watch has undocumented provisos to it working (as we demoed :-P) and combo loader isn't going to fix coding/reloading a test_html file, is it?14:05
rick_h_jcsackett: right, that's what I mean. I'm telling you the idea behind the enforcement, but admitting it's not something that's been vetted/put into stone14:06
jcsackettrick_h_: i think if you want this to be a policy to keep people from doing it the way i have, we *def* need to talk about it and get group buy in. :-)14:06
rick_h_jcsackett: no, but you're not linking to a test file, but a dependency which is copied to the build directory14:06
jcsackettrick_h_: dig. to be clear, i got the idea behind your request--i'm just pointing out there are some non-trivial disadvantages.14:06
rick_h_jcsackett: I'm not totally clear on the issue though14:06
rick_h_so you change code, reload, your test is linking to deps in a build dir14:07
rick_h_you're not editing the deps, just the current module under test which is ok to be relative since it's right there14:07
rick_h_the only time it requires a rebuild is if you edit a dependency which should have its own test file14:08
jcsackettrick_h_: hm. i suppose in this instance, i was dealing with test files that were/are poorly separated. so yeah, my experience may be nonstandard.14:08
rick_h_jcsackett: yea, I guess the idea is that deps come from build and only the code 'under test' is linked directly as a build step sanity check14:09
jcsackettrick_h_: fair. okay, i'll make those changes since outside of something like what i was doing where i had to modify and test a bunch of files it isn't actually such an issue.14:10
rick_h_jcsackett: ok, sorry, stand up. The rest of that thanks for the notes/feedback to the questions there.14:24
jcsackettrick_h_: all good. i'm in the process of changing over to build paths.14:24
rick_h_jcsackett: thanks, does the idea click after the discussion? I'll look to add some notes somewhere on the jsbuild_watch stuff and can bring up the build link discussion to -dev14:25
rick_h_you're getting to be the first user that didn't write the stuff so lucky you with the growing pains :)14:25
jcsackettrick_h_: yeah, i think the problem is it is still a gotcha if you're doing work over several modules and their tests.14:25
jcsackettbut that's a less common workflow, so i'm making the update.14:25
jamestunnicliffegmb: Before I ask for a review of the changes I have made for Bug #999717, I need to get a test written. It will require a JS test I think, but I would appreciate some pointers.15:08
_mup_Bug #999717: upcomingwork view blueprints should be initially expanded if incomplete <Launchpad itself:In Progress by dooferlad> < https://launchpad.net/bugs/999717 >15:08
gmbjamestunnicliffe, Okay. Let me take a look at the branch.15:23
=== al-maisan is now known as almaisan-away
gmbjamestunnicliffe, So, I think that rather than doing the detection of not-100%-completeness in the JS we can do it in the view code and make the testing a bit easier. Submit it for review and I'll suggest a patch for it (easier than me explaining it line-by-line in IRC).15:35
jamestunnicliffegmb: Will do, thanks.15:36
gmbnp15:36
=== matsubara is now known as matsubara-lunch
=== matsubara-lunch is now known as matsubara
=== Topic unset by gmb on #launchpad-dev
gmbAAAAAAAA17:09
=== gmb changed the topic of #launchpad-dev to: http://dev.launchpad.net/ | On call reviewer: - | Firefighting: - | Critical bugs: 3.47*10^2
=== rick_h_ changed the topic of #launchpad-dev to: http://dev.launchpad.net/ | On call reviewer: rharding | Firefighting: - | Critical bugs: 3.47*10^2
sinzuirick_h_, do you have time to review https://code.launchpad.net/~sinzui/launchpad/halt-when-done-2/+merge/10585917:11
rick_h_sinzui: loading now17:11
* deryck steps away to eat17:13
rick_h_sinzui: r=me17:17
sinzuithank you17:17
maxbIs there a polite way of attracting the attention of any webops who may be around?18:56
maxb(re #launchpad current conversation; psusi wants to know why he can't access bug 999790 despite receiving email about it)18:56
sinzuirick_h_, do you have time to review https://code.launchpad.net/~sinzui/launchpad/daily-build-form/+merge/10588419:39
adeuringabentley: this is the stuff i have so far for the test if a job with a retry error is queued again: http://paste.ubuntu.com/989542/ . Note the "poor man's debug attempt" in BaseRunnableJob.xxxRunExtraCommit() (the exception): Allows me to check if the fleature flag is correctly set. WHat I get in a test run: Exception: waaaaa2 TestJobWithRetryError False None . In other words: the feature flag "vanished"... Any idea?19:52
abentleyadeuring: looking.19:54
abentleyadeuring: Feature flags are memoized, so it's possible that it could start with None, then the flag could get updated, but the worker would still have the stale value.  Are you running this test by itself or with other tests?20:00
adeuringabentley: just the two tests in the module. The flag is set in both tests...20:00
adeuringabentley: and the flag is also missing if i run just the one test...20:02
abentleyadeuring: My new task_init code just landed in devel, and that explicitly clears the feature flag cache, so could you try merging devel?20:03
adeuringabentley: ok, I'll try it.20:03
james_whas anyone succeeded in getting the python-oops-tools tests passing locally?20:04
rick_h_sinzui: loading20:04
james_wI can't get them to pass before I've changed any code20:05
adeuringabentley: yay, that did it!20:06
rick_h_james_w: sinzui r=me20:06
abentleyadeuring: great!20:06
james_wpossibly because of the db being emptied after each test20:07
sinzuithank you rick_h_20:07
james_wlifeless, https://code.launchpad.net/~james-w/python-oops-tools/remove-lp-branding/+merge/10589220:28
=== rick_h_ changed the topic of #launchpad-dev to: http://dev.launchpad.net/ | On call reviewer: - | Firefighting: - | Critical bugs: 3.47*10^2
=== matsubara is now known as matsubara-afk
=== Ursinha` is now known as Ursinha
=== Ursinha is now known as Guest8786
wgrantsinzui: Bah, I thought I tracked down all the ES5 array ops :(21:20
gary_posterlifeless, I have to run to dinner but fwiw, I'm still seeing the odd -tags.  They don't hurt anything except themselves, but... http://pastebin.ubuntu.com/989769/  gotta run21:47

Generated by irclog2html.py 2.7 by Marius Gedminas - find it at mg.pov.lt!