[01:11] <wgrant> StevenK: I'm out for a while, but I have a lead and can reproduce it on ec2. Hopefully get it fixed soonish
[02:52] <StevenK> wgrant: The subscription YUI issue is because it is the only call in the codebase that calls Y.substitute() with a 3rd argument of a function to do the replacement.
[02:53] <StevenK> According to upstream, that means "We have a harder migration", which is pointing to Y.Template, but Y.Template was only added in 3.8.0 and I can't figure out how to have two code paths depending on the YUI version.
[02:53] <wgrant> StevenK: Where's the call?
[02:54] <StevenK> wgrant: lib/lp/bugs/javascript/subscription.js line 849
[02:59] <wgrant> StevenK: Can't you just manually preprocess the keys?
[03:01] <StevenK> wgrant: As in call Y.Lang.sub() with additional_vars as well?
[03:03] <wgrant> StevenK: The function just takes the key and the value
[03:03] <wgrant> So you should be able to replicate the present behaviour by just manually mapping the function over vars, and giving Y.Lang.sub the result
[03:04] <wgrant> StevenK: Can you try running YUIAppServerLayer locally?
[03:04] <wgrant> I've managed to reproduce it once on a single test locally, and two ec2 runs last night failed with it, but I can't repro it on ec2 manually. I guess it must be a race.
[03:07] <StevenK> wgrant: The function also uses additional_vars from the outer function
[03:08] <wgrant> StevenK: Sure, is that a problem?
[03:32] <StevenK> -    return Y.substitute(subscription.reason, subscription.vars, var_replacer);
[03:32] <StevenK> +    vars = var_replacer(subscription.vars);
[03:32] <StevenK> wgrant: Like that?
[03:32] <StevenK> +    return Y.Lang.sub(subscription.reason, vars);
[03:33] <StevenK> It even fits inline
[03:33] <StevenK> wgrant: YUIAppServerLayer against 3.5.1 or 3.9.1?
[03:34] <wgrant> StevenK: 3.5.1
[03:34] <wgrant> StevenK: var_replacer takes a key and a value
[03:34] <wgrant> you'll need to map it over the keys and values of cars
[03:34] <wgrant> vars
[03:35] <StevenK> So it does
[03:35] <StevenK> Where did the keys come from?
[03:36] <wgrant> I'm confused
[03:36] <wgrant> subscription.vars should be a map
[03:46] <StevenK> wgrant: The plot thickens. key is only used for additional_vars
[03:48] <wgrant> StevenK: Sure, because it already has the value from subscriptions.vars
[03:48] <wgrant> It's given the (key, value) pair from subscription.vars
[03:49] <StevenK> Ah ha, we do have to call it twice
[03:49] <StevenK> It will only act on additional_vars if vars is undef
[03:51] <wgrant> That presumably means you want to merge the keys of subscription.vars and additional_vars
[03:52] <StevenK> bin/test -vv --layer YUIAppServerLayer --subunit against 3.5.1 looks fine
[03:54] <wgrant> Yeah :(
[03:54] <wgrant> I might just disasble the layer by default for now...
[03:55] <wgrant> Does it work with 3.9.1 too?
[03:55] <wgrant> I have YUIAppServerLayer repeating on four machines now, hopefully one of them will give me a failure soon...
[03:55] <StevenK> wgrant: Your leads all ended up being nothing? :-(
[03:57] <StevenK> wgrant: 3 failures with 3.9.1
[03:57] <StevenK> Looks suspcisiously like the buildbot failures
[04:08] <StevenK> Hmmm
[04:08] <StevenK> I can't just call it with subscription.vars since that's an object too
[04:39] <StevenK> wgrant: I have all failures sorted for 3.9.1, except for the 3 XHR failures
[04:59] <wgrant> StevenK: Can you paste the failures?
[05:00] <wgrant> I will be very happy if you can reproduce them :)
[05:00] <StevenK> wgrant: I can copy the subunit stream to carob or so if you wish
[05:00] <wgrant> StevenK: I'm more interested in whether they are reproducible
[05:00] <wgrant> My four continuous runs have yielded nothing so far :/
[05:01] <StevenK> Re-running --layer YUI against 3.9.1
[05:03] <StevenK> wgrant: It happens against on a re-run
[05:03] <StevenK> *again
[05:04] <wgrant> StevenK: Same errors? Success on test_yuixhr_fixture_facet, a timeout before test start on test_milestone_creation, and undefined errors on the other two?
[05:04] <StevenK> Yes
[05:04] <wgrant> Can you push up your 3.9.1 branch so I can try to get them locally?
[05:05] <StevenK> wgrant: I've split it into two
[05:05] <StevenK> You should be able to merge the one I'm about to push, change default to 3.9.1 and make jsbuild
[05:07] <StevenK> wgrant: lp:~stevenk/launchpad/sprinkle-in-yui-391
[05:09] <wgrant> :S
[05:09] <wgrant> All four still pass
[05:10] <wgrant> On both my machines
[05:12] <StevenK> wgrant: build/js/yui points to 3.9.1?
[05:13] <wgrant> Yes
[05:14] <StevenK> wgrant: Odd. They still fail for me
[05:17] <wgrant> actual    = ['Failure in /home/wgrant/launchpad/lp-branches/yuixhr-3.5/lib/lp/app/javascript/choiceedit/tests/test_choiceedit.html.nullable_choice_edit.test_action_icon: Add icon is not visible when it should be\nExpected: inline-block (string)\nActual: inline (string)']
[05:17] <wgrant> That's the only failure I get with 3.9.1
[05:17] <wgrant> (ignore the branch name)
[05:19] <StevenK> wgrant: What other changes have you made?
[05:19] <wgrant> Nothing. That's just the branch that switched the default.
[05:19] <wgrant> It's identical to devel
[05:20] <wgrant> The ec2 runs where I can't manually reproduce it *are* devel
[05:20] <wgrant> :/
[05:21] <StevenK> wgrant: sprinkle-in-yui-391 doesn't change the default, but okay
[05:21] <StevenK> wgrant: Are you able to try on Quantal, rather than Lucid?
[05:22] <wgrant> Oh yeah, I changed the default too. but that's it
[05:23]  * wgrant tries on precise
[05:31] <wgrant> Still works fine :/
[05:32] <wgrant> And that precise container is even amd64
[05:35] <StevenK> wgrant: This saddens me
[05:37] <wgrant> StevenK: Can you reproduce the test_milestone_creation issue in a browser?
[05:38] <wgrant> (copy yuitest.zcml from configs/testrunner-appserver to configs/test-playground, bin/run -r librarian -i test-playground, visit https://launchpad.dev/+yuitest, select test_milestone_creation... it will fail because it can't reset the database, but you should at least see it running the tests)
[05:38] <StevenK> wgrant: On 3.5.1 or 3.9.1?
[05:39] <wgrant> StevenK: Doesn't really matter, if it fails on both
[05:40] <StevenK>     lp.registry.javascript.tests.test_milestone_creation
[05:40] <StevenK> Want to re-run your test?
[05:40] <wgrant> You don't get a test console at the bottom after a few seconds?
[05:41] <StevenK> Nope
[05:41] <wgrant> :D
[05:41] <wgrant> Check the browser's network/JS console
[05:41] <wgrant> For errors
[05:42] <StevenK> Hmm
[05:42] <StevenK> GET meta.js ended up with a 500
[05:42] <wgrant> Oho
[05:43] <wgrant> You've run combobuild, right?
[05:43] <StevenK> The path is bong
[05:43] <StevenK> https://launchpad.dev/+yuitest/build/js/lp/meta.js
[05:44] <wgrant> That should be fine
[05:44] <wgrant> +yuitest/build forwards to the comboloader dir
[05:44] <wgrant> So it doesn't have to use a real comboloader
[05:44] <StevenK> Passed:2 Failed:2 Total:4 (0 ignored)
[05:45] <wgrant> What'd you change?
[05:45] <StevenK> I ran make combobuild
[05:45] <wgrant> Ah
[05:45] <wgrant> Do your tests pass now?
[05:45] <wgrant> Make sure you shut down test-playground before you rerun YUIAppServerLayer
[05:47] <StevenK> test: lp/registry/javascript/tests/test_milestone_creation
[05:47] <StevenK> WARNING: gnome-keyring:: couldn't connect to: /run/user/steven/keyring-hlnCV9/pkcs11: No such file or directory
[05:47] <StevenK> successful: lp/registry/javascript/tests/test_milestone_creation
[05:47] <StevenK> Ah ha
[05:47] <wgrant> Damn.
[05:47] <wgrant> Can you paste the failures you got before the combobuild?
[05:47] <StevenK> Just make clean && make and you should have no combobuild ?
[05:48] <StevenK> Or rm -rf build/js && make jsbuild
[05:48] <StevenK> Failed tests:      0
[05:49] <wgrant> I'd expect a missing meta.js to cause all four suites to fail to start
[05:49] <wgrant> but omfg
[05:49] <wgrant> that was it
[05:50] <wgrant> It's just a missing meta.js
[05:50] <wgrant> fuuuuu
[05:50]  * wgrant lands fix
[05:50] <StevenK> Haha
[05:50] <wgrant> Thanks
[05:50] <StevenK> What is the fix, then?
[05:50] <wgrant> I guess the other tests specify just enough direct deps to get them stat
[05:50] <wgrant> build needs to depend on combobuild
[05:50] <wgrant> The reason it worked whenever I tried it on ec2 is that I ran 'make' before testing...
[05:51] <StevenK> That will blow up
[05:51] <StevenK> You can't rely on convoy being available
[05:51] <StevenK> Which was the whole reason combobuild was split out
[05:51] <wgrant> Ah, right
[05:51] <wgrant> Might just put that in check, then
[05:52] <wgrant> I think buildbot and ec2 both use check, right?
[05:52] <StevenK> Not sure
[05:52] <StevenK> ec2 does
[05:52] <StevenK> Since I just checked
[05:52] <wgrant>         command = ['make', 'check']
[05:52] <wgrant> Yeah
[05:53] <wgrant> Hmm
[05:53] <wgrant> buildbot won't
[05:53] <wgrant> Because it does it parallel
[05:53] <wgrant> lxc-clean codehosting-dir build schema
[05:55] <StevenK> We could do it in build, but the only way I can think to flag if convoy is available is -f /usr/share/convoy/convoy.wsgi or something equally hideous
[05:56] <wgrant> Well
[05:57] <wgrant> StevenK: Is there a reason to not do it in build now, other than the fact that python-convoy isn't installed everywhere?
[05:57] <wgrant> Easiest thing to do may just be to add it to launchpad-dependencies
[05:57] <wgrant> Otherwise we can change the buildbot config
[05:59] <StevenK> wgrant: We don't need to do so everywhere
[05:59] <StevenK> Only the frontends really need it
[05:59] <wgrant> But is there any reason to not?
[05:59] <StevenK> Save a little time
[05:59] <StevenK> I guess
[05:59] <wgrant> We already do the JS building etc. unnecessarily on the backends and non-web machines
[05:59] <StevenK> Not anymore we won't
[06:00] <StevenK> wgrant: I think python-convoy is only on banana/nutmeg and not everywhere
[06:00] <wgrant> Sure
[06:00] <StevenK> But we sync built trees, no?
[06:00] <wgrant> Yes.
[06:00] <wgrant> But then they're remade at the destination
[06:01] <StevenK> Hah
[06:01] <StevenK> wgrant: I seem to recall you objecting to needing convoy everywhere, but I could be recalling wrong
[06:02] <wgrant> So do I, but I can't find the IRC logs.
[06:02] <StevenK> wgrant: If we want convoy everywhere, we just move combobuild into jsbuild
[06:02] <wgrant> Yes.
[06:02] <wgrant> I suspect my objection was just that the easiest way to get the world unbroken was to split combobuild back ouit
[06:02] <StevenK> Right
[06:03] <wgrant> Now that combobuild is mandatory, having it separated from jsbuild probably doesn't make sense.
[06:03] <wgrant> But buildbot should have convoy installed, so we could unbreak it now by merging them with a guard.
[06:03] <wgrant> And then push out the dependency changes, or stop building the JS everywhere, later.
[06:05] <wgrant> I don't remember if we actually use make build on prod anywhere
[06:05] <StevenK> I *think* banana and nutmeg have been hacked to also run combobuild
[06:05] <StevenK> And HTF knows what asuka does any more
[06:06] <wgrant> staging uses make build
[06:06] <wgrant> rocketfuel-built on carob doesn't have any JS stuff
[06:07] <wgrant> Looking at deploymgr configs, codebrowse uses make build
[06:07] <wgrant> For no reason
[06:08] <wgrant> I don't think...
[06:08] <StevenK> stevenk@carob:~$ dpkg -l python-convoy
[06:08] <StevenK> No packages found matching python-convoy.
[06:08] <wgrant> But other than that it's just banana/nutmeg
[06:08] <wgrant> StevenK: Yeah, but carob has no JS at all. It must just use compile, not build
[06:08] <StevenK> Ah
[06:08] <wgrant> Actually, it probably only uses build_eggs
[06:08] <wgrant> Not even compile
[06:09] <wgrant> And asuka must already have python-convoy, since its comboloader works
[06:09] <wgrant> So it might just be codebrowse that needs fixing to not call build
[06:09] <StevenK> I think the upgrade scripts call combobuild
[06:09] <wgrant> Upgrade scripts/Z
[06:09] <wgrant> ?
[06:10] <StevenK> The staging/qas new revision scripts
[06:10] <wgrant> Yeah, they do
[06:10] <wgrant> modules/launchpad/files/staging_restore.sh:    ssh launchpad@asuka "make -C /srv/staging.launchpad.net/staging/launchpad-new build combobuild LPCONFIG=staging" >> $LOGFILE 2>&1
[06:10] <wgrant> modules/launchpad/templates/qastaging-update.erb:make combobuild LPCONFIG=qastaging
[06:11] <StevenK> Right, so those can die
[06:11] <StevenK> How do you want to proceed?
[06:11] <wgrant> We need to think of anything else that might use build
[06:11] <wgrant> And kill it
[06:11] <wgrant> I think it's just babaco that needs fixing
[06:11] <StevenK> Does it actually need anything from the built tree?
[06:12] <wgrant> NAFAIK
[06:12] <wgrant> It uses CSS from launchpad.net, I believe
[06:12] <wgrant> Yeah
[06:13] <wgrant> So I think it can just use compile
[06:13] <wgrant> Which leaves only the frontends and staging using build
[06:13] <wgrant> Which means we can merge combobuild into jsbuild
[06:13] <StevenK> wgrant: I can destroy the combobuild target and move to jsbuild for my sprinkle-in-yui-391 branch if you wish
[06:13] <wgrant> We'll want to leave combobuild as an alias for jsbuild for now, I guess
[06:13] <wgrant> Until we remove references to it
[06:14] <StevenK> That's one puppet branch only, though
[06:14] <wgrant> Yes
[06:14] <wgrant> But two phases
[06:14] <wgrant> Mm, I guess not
[06:14] <StevenK> And we probably need one anyway to add python-convoy
[06:14] <wgrant> If we just deploy it at the right time
[06:16] <StevenK> wgrant: I have a meta-lp-deps branch that promotes convoy
[06:17] <StevenK> In fact that shows a problem with 0.120
[06:17] <wgrant> StevenK: What's the problem?
[06:17] <StevenK> -  python-egenix-mxtools, python-amqplib, python-gmpy, libpq-dev, unzip, lzma
[06:17] <StevenK> -  ${misc:Depends}
[06:17] <StevenK> +  python-egenix-mxtools, python-amqplib, python-gmpy, libpq-dev, unzip, lzma,
[06:17] <wgrant> (and while we don't necessarily need convoy everywhere, it's so trivial that we might as well just install it everyhwere)
[06:17] <StevenK> +  python-convoy, ${misc:Depends}
[06:17] <wgrant> Oops
[06:17] <StevenK> Haha
[06:17] <wgrant> That was mine, wasn't it?
[06:17] <StevenK> Yeah
[06:18] <wgrant> But I bet misc:Depends expands to nothing
[06:18] <wgrant> So it didn't actually matter
[06:18] <StevenK> Yes, so it's fine
[06:30] <StevenK> wgrant: So I should push up my death-of-combobuild commit to sprinkle-in-yui-391 and my meta-lp-deps branch?
[06:31] <wgrant> StevenK: Please do
[06:31] <wgrant> StevenK: That branch doesn't make 3.9.1 the default, right? Just adds it as an alternate version?
[06:32] <StevenK> wgrant: Right
[06:32] <StevenK> And fixes the 3.9.1 carnage in the JS
[06:32] <wgrant> Right
[06:32] <wgrant> Though that is safe back to 3.5
[06:32] <wgrant> And possibly even 3.3
[06:32] <StevenK> I checked 3.5.1, but 3.3.0
[06:32] <StevenK> wgrant: Shall I just land meta-lp-deps?
[06:33] <wgrant> 3.3.0 is dead, fortunately :)
[06:33] <wgrant> Indeed
[06:33] <StevenK> meta-lp-deps r149 pushed
[06:45] <wgrant> StevenK: Is there an MP coming up?
[06:46] <wgrant> sprinkle-in-yui-391 looks good, but I can't approve it :)
[06:46] <StevenK> https://code.launchpad.net/~stevenk/launchpad/sprinkle-in-yui-391/+merge/157805
[06:48] <StevenK> wgrant: Oh?
[06:48] <wgrant> StevenK: Hm, what's with the test_choiceedit change?
[06:49] <wgrant> That didn't fail on the last buildbot run
[06:49] <StevenK> That was the failure
[06:49] <wgrant> Won't that break on 3.5?
[06:49] <StevenK> On 3.9.1
[06:49] <wgrant> And only pass on 3.9?
[06:50]  * StevenK tests again
[06:52] <StevenK> wgrant: Failed tests:      0
[06:52] <wgrant> StevenK: But how
[06:52] <wgrant> StevenK: Are you sure your symlinks are correct?
[06:53] <StevenK> lrwxrwxrwx 1 steven steven 9 Apr  9 14:56 build/js/yui -> yui-3.5.1
[06:54] <StevenK> lib/canonical/launchpad/icing/yui -> ../../../../build/js/yui-3.5.1
[06:54] <wgrant> Hmmmmmm
[06:59] <StevenK> 3.5.1 fails without that change
[07:00] <StevenK> Oddly
[07:00] <wgrant> Maybe it's the CSS?
[07:00] <wgrant> 'cause it fails for me *with* that change
[07:01]  * StevenK runs make clean && make
[07:02] <wgrant> clean_buildout is probably sufficient
[07:02] <StevenK> Eh
[07:03] <StevenK> Tests runnning
[07:03] <wgrant> I'm looking at upgrading us to a more modern ZTK
[07:04] <wgrant> Since ZTK 2 will be out soonish
[07:04] <wgrant> And we're way behind
[07:05] <StevenK> Failed tests:      0
[07:09] <wgrant> StevenK: Still fails here with 3.5.1: Expected: inline-block (string)\nActual: inline (string)
[07:09] <wgrant> And buildbot agrees with me
[07:20] <StevenK> Very strange
[07:52] <wgrant> StevenK: So, do you want to revert the inline-block change so we can get buildbot unbroken?
[07:56] <StevenK> wgrant: I did
[07:56] <StevenK> I just forgot to ping
[07:57] <wgrant> Ah, great
[07:58] <wgrant> StevenK: r=me, remember to [testfix] it
[07:58]  * wgrant is disentangling ancient zope stuff
[07:58] <StevenK> Aye, I have not forgotten
[07:59] <StevenK> wgrant: I think almost all of our Zope stack is ancient
[08:00] <wgrant> Well
[08:00] <wgrant> One bit I just removed has been unused since 2005
[08:00] <wgrant> We use so much deprecated stuff that I need to take an axe to a few things before I can upgrade to ZTK 1.1.5
[08:01] <StevenK> wgrant: testfix is r16556
[08:01] <wgrant> Once we're up to 1.1.5 future upgrades should be relatively painless
[08:01] <wgrant> Thanks.
[08:01]  * wgrant watches buildbot
[08:01] <StevenK> Which one is ZTK in versions.cfg?
[08:02] <wgrant> The main product of ZTK is the KGS
[08:02] <wgrant> http://download.zope.org/zopetoolkit/index/1.1.5/ztk-versions.cfg
[08:02] <wgrant> So ZTK 1.1.5 is just the aggregate of those version of the zope.* packages
[08:02] <StevenK> Ah
[08:03] <StevenK> zope.testing = 3.9.4-p17
[08:03] <wgrant> So it's most of our Zope stack, except that we have a broken implementation of zope.app.apidoc which is deprecated
[08:03] <StevenK> That's going to be fun
[08:03] <wgrant> And probably doesn't work
[08:03] <wgrant> So our Zope apidoc (not to be confused with the lazr.restful apidoc) will probably die tonight
[08:05] <StevenK> wgrant: And no one will miss it ever.
[08:05] <wgrant> I don't think it's worked in a couple of years
[08:05] <wgrant> Since our last Zope upgrade
[08:08] <wgrant> StevenK: One of the key changes in recent ZTK releases is that zope.app is basically dead
[08:08] <wgrant> it
[08:08] <wgrant> We still use bits of
[08:08] <wgrant> But in most cases they're just deprecated imports that have been moved elsewhere
[11:57] <lifeless> wgrant: StevenK: btw, new testrepository & subunit are out; you guys may want to consider upgrading at some point
[11:58] <wgrant> lifeless: Is that with the v2 protocol?
[11:59] <lifeless> yes
[12:00] <lifeless> the testr data store is still v1, but the backend -> coordinator is v2
[12:01] <lifeless> I'm flushing the last of the 'TestResult' API use within testr's internals
=== matsubara_ is now known as matsubara
=== deryck_ is now known as deryck[lunch]
=== deryck[lunch] is now known as deryck
=== almaisan-away is now known as al-maisan
=== al-maisan is now known as almaisan-away
=== gary_poster is now known as gary_poster|away
=== gary_poster|away is now known as gary_poster
=== czajkows1i is now known as czajkowski
=== elmo__ is now known as elmo
=== Ursinha-afk is now known as Ursinha