[00:57] afk - hospital visit [01:36] wallyworld, hi [01:41] rockstar: hello, just have to deal with a little domestic issue. can i ping you shortly? [01:41] wallyworld, sure. [01:41] thnks [01:41] wallyworld, although there is a Michigan game on tonight, so I might be a bit distracted... ;) [01:52] rockstar: skype? [01:53] rockstar: http://pastebin.ubuntu.com/517782/ [01:53] devel is not very happy right now [02:01] * thumper tries windmill tests locally [02:02] thumper: what was the failure message? [02:02] wallyworld: full errors here - https://lpbuildbot.canonical.com/builders/lucid_lp/builds/274/steps/shell_6/logs/summary [02:03] thumper, yeah, I think all windmill tests should be disabled. [02:04] wallyworld, sure. [02:04] rockstar: and replaced with what? [02:19] thumper, well, with jstestdriver, but for now, just not replace it. [02:19] thumper, basically, windmill tests aren't telling us what they're supposed to be telling us. [02:19] I mean, what good is a test that randomly decides it's broken? Might as well have no tests at all. === Ursinha is now known as Ursinha-afk [02:54] it's a mean thing to say, but the status of the windmill tests makes me glad not to be a fulltime lp developer any more :/ [03:13] mwhudson, yeah. :( [03:20] thumper, we're using jstestdriver in ubuntu one and landscape [03:20] beuno: and how's that working for you? [03:20] hopefully, that knowledge will go back with rockstar and he can migrate LP, if it makes sense [03:21] thumper, they are reliable tests, that I can tell you :) [03:21] * thumper wants reliable tests [03:21] also, there's a nice way of TDDing with javascript [03:21] which I have been meaning to send an email about for months [03:21] my new plan is to talk to rockstar, and let him be the messenger and take all the credit [03:21] beuno, :) [03:22] Yeah, the fact that _GMAIL_ gets tested with jstestdriver indicates to me that it's very robust. [03:22] they moved away from browsers clicking stuff [03:23] it didn't scale and they spent more time chasing false positives than actual problems [03:23] * thumper votes to slaughter windmill with prejudice [03:24] beuno: I don't want to have six months of windmill pain before we get something reliable [03:24] beuno: please send email :-) [03:25] thumper, I will have rockstar locked up in a third world country for a week, I'll make sure someone does so [03:25] beuno: what? Florida? [03:25] more country [03:25] buenos aires [03:26] thumper, yeah, this is where I'm stuck as well. We have no resources to maintain windmill, but we also have no resources to switch. [03:28] I would make it a personal vendetta if branch merge queues weren't already my personal vendetta. :) [03:34] lifeless: what are your thoughts on killing windmill and instating jstestdriver? [03:34] lifeless: we need to do something [03:34] lifeless: because what we have right now is a pile of smelly doggy do-do [03:34] thumper, I think it's just a matter of getting jstestdriver in sourcedeps, creating a layer, and then migrating the windmill tests. [03:34] no disrespect meant to dogs [03:37] thumper: I've no opinion [03:37] I hear terrible things about all the browser test drivers [03:38] lifeless: no opinion given that windmill is causing most of our test failures? [03:38] given the heavyweight nature of these things, I'd like us, should we do it, to do it full on [03:38] I can't imagine that you have no opinion, sorry [03:38] thumper: causing or showing [03:38] the windmill threads stuff is now sorted [03:38] see - hudson went green [03:39] lifeless: but buildbot is broke [03:39] devel or db-devel ? [03:40] devel [03:40] well, both actually [03:42] WindmillTestClientException: {u'suite_name': u'Inline bug page subscribers test', u'result': False, u'starttime': u'2010-9-22T1:26:51.618Z', u'params': {u'id': u'subscribers-links', u'timeout': u'20000'}, u'output': None, u'endtime': u'2010-9-22T1:27:11.722Z', u'method': u'waits.forElement'} [03:42] looks like a legitimate error to my naive eye [03:43] I ran the test locally from devel and it was fine [03:43] busy machine ? [03:43] so, no... I don't think it is legitimate [03:43] even a busy machine shouldn't take 20 seconds to do that [03:44] so, if we're getting spurious failures, we should fix them [03:44] *that* I have an opinion on [03:44] I simply don't know enough about the windmill design / implementation to have an opinion on *it* [03:45] lifeless, thumper, I think having mindshare in javascript testing with the rest of the web properties at Canonical is of value as well. [03:45] rockstar: agreed [03:46] my question is how to transition [03:46] lifeless, windmill will fail tests on random I/O issues. [03:46] should we just say to each team: do it, and do it now! ? [03:46] rockstar: it can be nice to share war stories, but unless we make lp -massively- simpler to work with, I doubt that there will be much direct cross pollination [03:46] rockstar: how random? [03:46] lifeless, the big issue is that the way to fix the spurious test failures is to fix it in windmill. [03:46] is that particularly hard? [03:47] Understand that I have -no- loyalty to windmill [03:47] it is if you don't know where the problem is [03:47] lifeless, it is in that no one has any understanding of windmill. [03:47] As I say, I simply don't know enough about its guts to comment [03:47] rockstar: do -we- have that understanding of jstestdriver? [03:47] * thumper forces a build just to see if they are transient [03:47] lifeless, well, maybe not anyone on Launchpad, but Landscape, lazr-js, and U1 all use it. [03:48] lifeless, I know enough to write tests in it, since lazr-js uses it, and I use lazr-js. [03:48] rockstar: by which I mean the LP team; most of the teams at canonical are pretty flat chat - we shouldn't *expect* that they can fix issues we may encounter [03:48] anyhow, as I say [03:48] I don't have an opinion other than: if we're going to do it, do it as a dedicated focus. [03:49] have two different test drivers around at once for firefox scares me [03:49] rockstar: one thing that would make a compelling difference to me is parallel testing [03:49] lifeless, actually, I think we just need to kill windmill. It'll remove test coverage, but like I said, if you can't rely on the tests, are they really tests? [03:49] rockstar: mmm [03:50] I can understand the feeling [03:50] but false positives are not false negatives [03:50] AIUI the issue we have is false positives, not false negatives. [03:50] Which means our tests fail spuriously, but they don't pass when they shouldn't. [03:53] lifeless, yes, but test failures drastically reduce our productivity and velocity. [03:53] they do [03:53] and we should fix that [03:53] risk assessment [03:53] lets say that the windmill tests cover 35% of our js code [03:54] and that 10% of js landings hit valid windmill failures [03:54] and that thats 5 landings a month [03:55] that would say that 65% of js could be broken without safeguard, and that of the 355 coverage we have we catch a bug every two months [03:55] and that a bug a month gets through to be caught later (e.g. by users) [03:55] I can't think of a single thing that would be unusable without javascript. [03:56] rockstar: when it breaks, people get unhappy [03:56] it can break in many ways [03:56] I don't know what those figures would be [03:57] but I can't really agree *or* disagree about the relative merits of the test coverage we have vs the velocity cost that it has without some close observation - or figures /like/ those. [03:57] I agree its a problem [03:57] and that we should fix it [03:57] and that that will help our velocity a lot - its why I've fixed the new threads issue with windmill [03:58] lifeless, did you fix it, or just not consider it a test failure? [03:59] rockstar: did you see my email about this? I detailed most fo it [03:59] one thing I didn't mention in my email was that WindmillLayer *already* had an *explicit* code path to disable the thread checks: it wasn't working. [04:17] lifeless: rockstar: i have tried to land branches 5 or 6 times this week and have had "failures" due to windmill [04:18] i have just now "solved" a different windmill issue which has consumed waaaay to many hours of my time [04:18] the sooner it goes bye bye, the better IMHO [04:20] we don't have 10000s of windmill tests. surely we can introduce jstestdriver, write new tests using that, and migrate windmill ones as drive bys or whatever as time permits [04:21] I smell a joint sprint [04:21] lazr-js: the revenge of the tests [04:21] also, bed [04:21] * beuno waves [04:22] wallyworld: yes, threads were fixed lastnight [04:22] beuno: /me smells a big, steaming pile of excrement and it's called windmill :-( [04:22] \o/ [04:23] lifeless: it's not just the threads issue. there other nasty things at work too [04:23] sure [04:24] if you add up my time, thumper's time, rockstar etc etc, and the future time we will surely waste and the scalability problems with the browser driven approach; the common tools/mindshare issue - surely a case can be made to swtich [04:26] sure [04:26] I'm not disputing that [04:27] I don't think anyone is saying "Windmill is great." I think we're just proposing different solutions. [04:30] i don't have much hair left to pull out :-) [04:37] james_w: https://code.edge.launchpad.net/~james-w/launchpad/more-matchers/+merge/32057 [04:37] james_w: can we land? no? [04:52] * StevenK wonders how to easily get failing tests out of a subunit stream [04:53] subunit-filter | subunit-ls [05:04] lifeless: i talked to james_w about some approved-but-not-landed branches a couple of weeks ago. he indicated there was no hold up except his lack of time. [05:04] thanks for following up. [05:09] it may be worth just doing the changes and landing them [05:11] StevenK: did that help? [05:12] lifeless: Hm, did which? [05:12] a16:53 * StevenK wonders how to easily get failing tests out of a subunit stream [05:12] 16:54 < lifeless> subunit-filter | subunit-ls [05:12] lifeless: Yes, it did, thanks. [05:12] k [05:33] jml: http://bzr.bz [05:45] * StevenK grumbles at failures in buildbot [05:46] StevenK: hows hudson [05:47] From my scrollback: [05:47] [02:56] < LPCIBot> Yippie, build fixed! [05:47] [02:56] < LPCIBot> Project devel build (142): FIXED in 4 hr 1 min: [05:47] https://hudson.wedontsleep.org/job/devel/142/ [05:48] But since devel is failing, stable isn't getting anything and therefore db-devel isn't [05:48] yeah [05:48] stuckage [06:59] \o/ i finally got a branch through ec2 without spurious windmill failures! [07:03] jml: cool [07:03] bah [07:03] wallyworld: cool [07:04] wallyworld: so regarding 'add js, remove windmill lazily' [07:04] wallyworld: I don't see this as a good strategy [07:04] wallyworld: because the problem isn't with *new* windmill tests, its primarily with windmill itself / existing tests. [07:05] wallyworld: I could see 'add jstestdriver, sprint (virtual or otherwise) on migrating' [07:05] lifeless: the problems this week for me were with a new test [07:05] wallyworld: but adding a new thing which has its own failure modes and leaving the old one around indefinitely just means we're paying two sets of overhead [07:06] i don't think we should have *both* windmill and jstestdrive - i just want one framework that is fit for purpose [07:06] right, me too [07:06] if that means we introduce jstestdriver and allow a bit of slack to migrate existing windmill tests, so be it :-) [07:07] mmm [07:07] that worries me [07:07] well, we can't let it become technical debt which is never repaid [07:07] previous attempts at that ... are still incomplete. [07:07] I certainly won't stop anyone doing this [07:07] Leaving technical debt around is the Launchpad Way™ :( [07:07] I just think its illadvised [07:08] yes, so we would need buy in from the teams that they see it as a very good thing so would be motivated to help get in and fix it [07:08] wgrant: oh no, not just the launchpad way. on my last job, there were 4 1/2 gui frameworks in use [07:08] wallyworld: That is... impressive. [07:08] and i do mean 1/2, cause one of them was only ever 1/2 finished [07:08] ... [07:09] what a noob i am - what does ... mean? you want more info? [07:10] wallyworld: its a verbal pause [07:10] Depressed silence, I guess. [07:10] lifeless: there's not *that* many windmill tests to migrate. it would only take each of us taking on one or two test classes each, maybe 4 hours work, and it would be done [07:10] wallyworld: if the story were 'windmill is ok, want new shiny', I'd be like 'ok rad' [07:10] wallyworld: great! organise it! [07:11] ok. i'll get some hard facts first. i'll talk to ohers about jstestdrive so i understand the scope a bit better [07:12] wallyworld: my specific concerns are: [07:12] - another half finished migration [07:12] - not resolving the problems [07:12] +1 [07:12] - and adding in a new set [07:12] these are all variations on a theme [07:13] yes, and at this stage, to my mind at least, we are in the "we have a problem, lets consider options" stage [07:13] where option 1 is still keep windmill but fix it [07:14] but given the obstacles to that and the other drivers for something better, imho we need to take the next step of scoping option 2 [07:14] once the cost is understood, we can make an informed decision [07:15] and we also should include as part of it all migrating one or two tests to dip our toes in the water and fully understand hpw it would play out [07:16] sure, but on a branch :) [07:17] huh? why would it be any other way? [07:17] wallyworld: I've seen some mad shit in my time [07:17] wallyworld: I didn't mean to insult [07:18] lifeless: no problem, i didn't take it as an insult. just swalllowing the bait :-) [07:55] hah [07:55] all those windmill failures were transient [07:55] due to our spurious test failure rules [07:55] we should disable windmill [07:56] s/isable/elete/ [07:56] I thought that was meant to be fixed :( [07:56] Like three times now. [07:56] In the last two days. [07:56] wgrant: It was mostly impacting hudson [07:56] * thumper shrugs [07:56] but I'm please buildbot is green [07:56] Same, I can land stuff now [07:57] Except PQM still thinks we are in testfix. Bleh [07:59] * StevenK kicks it [08:04] Or won't PQM come out of testfix until db-devel is fixed? [08:05] it should have come out by now [08:06] * StevenK tries to land again [08:07] Still fails with testfix [08:37] good morning [08:43] lifeless: I guess I'm right, since pqm is still in testfix [09:02] lifeless: StevenK: there's a short, indefinite period of time between submitting a testfix and coming out of testfix iirc [09:05] I recall something about it polling every 15 minutes. [09:05] But I could well be wrong. [09:07] dev.launchpad.net/Trunk/Glue iirc [09:07] "(Builbot's built-in "Force Build" button doesn't work for us for rather boring reasons). [09:07] " [09:07] That sounds very obsolete. [09:08] jml: The devel build was sucessful about an hour ago. That isn't enough to drag us out of testfix? [09:08] wgrant: it doesn't work for us [09:08] StevenK: probably [09:08] wgrant: we use a different, custom button [09:08] jml: Ahh, I see. [09:09] jml: So I should submit a branch as a testfix? [09:09] StevenK: I don't know. [09:09] Why does our PQM configuration have to be such a black hole? [09:13] Morning [09:13] StevenK: It's a rockstar conspiracy. [09:13] StevenK: To make us use Tarmac and merge queues. [09:17] mrevell: morning [09:19] StevenK: its in a branch, just need it pushed to lp (private) [09:20] StevenK: but this stuff is in BB anyway [09:21] lifeless: It still doesn't explain if buildbot needs both devel and db-devel built sucessfully before we are out of testfix or if we need to be pushed out of testfix manually [09:21] I was told that when bb starts a build it takes us out of testfix [09:21] that may have been a misstatement [09:22] so why aren't we building db-devel ? [09:22] Hell, I don't even know if devel => stable has happened [09:22] bzr pull [09:22] or missing [09:22] transparency! [09:22] or whatever, can tell you that [09:23] it's not even a river in Egypt [09:23] http://yoshinorimatsunobu.blogspot.com/2010/10/using-mysql-as-nosql-story-for.html [09:24] lifeless: that looks interesting. unfortunately, I have to go and have a driving lesson. [09:24] jml: what are you learning to drive? [09:24] lifeless: automobiles [09:24] lifeless: stable is missing 11 revs -- 11765 to 11775 [09:25] jml: Manual or automatic? [09:25] manual. [09:25] good bye. [09:25] 11775 passed lpbuildbot [09:25] so it should have pulled them [09:25] jml: Enjoy! [09:25] https://lpbuildbot.canonical.com/builders/lucid_lp/builds/275 [09:26] lifeless: Does prase run the poller? If so, could the lucid upgrade killed it? [09:27] I didn't think praé was lucid yet [09:27] It was upgraded yesterday [09:27] ok [09:27] * lifeless is suprised pqm kept working [09:28] StevenK: thanks for noticing this [09:28] Well, it hasn't clicked off testfix mode, so maybe it isn't working? [09:30] * lifeless boggles at launchpad-users === almaisan-away is now known as al-maisan [09:44] (lucid on prae did break lp buildbot polling) [09:44] appears fixed now [09:55] there's nothing like a stable dev environment [09:55] Even Hudson is happy now. Things have to be going better :) [10:13] grah [10:13] my ec2 test runs are not sending failure mails [10:13] or success mails [10:16] grah grah [10:16] * lifeless sends it to pqm [10:17] nothing like breaking it 16 hours before I fly [10:26] lifeless: i'm having the same issue, but only for one particular branch [10:37] lifeless: I need to book some time with you and discuss the scalability of PPAs. The increasing number of them is putting some strain on us. [10:37] are you @ UDS ? [10:37] yes [10:37] lets do it there? [10:37] sounds good [10:37] thanks [10:37] kk [10:38] process-death-row is taking 1h17m :/ [10:38] ! [10:38] What's it doing? [10:38] scanning a bazilion PPAs [10:39] That sounds like a stupid bug, not a real scalability issue. [10:39] possibly [10:39] but we also need to address the publishing cycle [10:39] * bigjools brb [10:39] Yes, but that's also stupid bugs. [10:41] I think we should make the code not suck before we decide we have a serious scalability issue. [10:47] For starters it could not iterate over EVERY PPA EVER. [10:48] And then issue a probably unoptimised query. [10:49] And then commit around 20000 times. [10:52] I bet you could remove the archive-driven approach and reduce it to two queries to identify all the publications that need work. [10:52] And that would take a few seconds. [10:53] Not 4500. [10:55] Or we could fix the data model slightly and make dateremoved work sanely and then trivially query the archives that require consideration. [11:20] +1 [11:20] but devil will be in the details === al-maisan is now known as almaisan-away [11:27] always [11:27] but the scalability issue remains nonetheless [11:28] lifeless: can the feature flag work cope with changes to existing pages that I don't want to release, but I do want to land? [11:28] They do. [11:33] I guess I need to read about FFs then [11:39] urg, this is going to be painful [11:39] It's not just a matter of guarding a couple of sections of the page? [11:40] StevenK: given that all your IDS stuff has landed, I am thinking that my changes to +addseries don't need to be behind a feature flag [11:40] wgrant: see https://dev.launchpad.net/LEP/DerivativeDistributions and you tell me :) [11:40] a lot of the existing stuff was cleaned up as well as new bits [11:40] but like I said, I think it can just land. [11:42] wgrant: actually I'd like your opinion on something [11:42] Hmmm. [11:43] Distroseries has a parent_series which we can use to work out if something is derived or not (if the parent_series's distro is different) [11:43] however this doesn't help for us making Ubuntu derived from Debian [11:43] The distroseries parent thing in that spec is screwed. [11:43] Will not work at all. [11:43] You need two parents. [11:43] such a delicate way of putting it [11:44] Heh, sorry. [11:44] the mockup was reviewed many times by many people [11:44] It certainly can't work for Debian->Ubuntu. And I don't see how it can work for OEM. [11:44] Unless OEM wants to restart from scratch every time. [11:44] explain? [11:45] An Ubuntu series' parent is the previous Ubuntu series. [11:45] But it needs derivation features from a Debian series. [11:45] For the OEM case, s/Ubuntu/OEM/, s/Debian/Ubuntu/ [11:46] yes [11:46] this we are aware of [11:46] Unless you want to fork, you need to inherit from within your own distro then use the new derivation UI from some other distro. [11:46] The proposed model does not support that. [11:47] which model? [11:48] I guess the spec may not actually mean parent when it says parent. [11:48] there are 2 types of parent, it's a little ambiguous [11:49] but the "parent" in the mockup refers to the derived parent [11:49] Ahh. [11:50] And that's separate from the current model's parent? [11:50] well, don't confuse what's in the model currently with what's on the mockup [11:50] we're using the mockup to drive the model [11:50] the model may have to change [11:50] it may not [11:51] I am pondering whether to add a "derived_from" column to distroseries [11:54] bigjools: I'm not sure how best to term them. [11:55] We do need a new column. That much is certain. [11:55] I think you are right. [11:55] You didn't say that to me last night when I said I think we do. [11:55] :-P [11:56] StevenK: I didn't say much at all did I? :) [11:56] * bigjools brb [12:00] derived_from seems fairly bad, but I can't think of anything better. [12:00] It's ambiguous :( [12:01] Morning, all. [12:03] wgrant: derived_from_series ? [12:03] howdy deryck [12:03] bigjools: But it's derived from both. [12:04] wgrant: both what? [12:04] bigjools: Natty is derived from both Maverick and Sid. [12:04] So calling one 'parent' and one 'derived_from_series' isn't awfully unconfusing. [12:05] wgrant: not really, it's derived from something way back, and its parent_series is maverick [12:06] Well, it's still derived from Maverick, even if that's not what Soyuz calls it. [12:06] wgrant: we are using derivation explicitly to mean "copied from a different distro" [12:07] natty is not derived from maverick, it's just a continuation of the series [12:07] consider them point releases if you like [12:07] In the Soyuz sense of the term, sure. [12:08] derived_from_series is fine, but something that doesn't use an ambiguous word would be better. [12:08] what's ambiguous for you? [12:10] bigjools: Now I have been informed of the limited Soyuz-specific definition of the word 'derivation', nothing. [12:10] But anyone else reading the schema is likely to be terribly confused. [12:10] wgrant: we have to make these definitions for that very reason [12:10] the whole project is called "Derived Distributions" for gawd sake :) [12:11] True. [12:11] as long as the meanings are well documented and not terribly confusing I think it's ok [12:11] I am thinking of some Registry dev coming along and wondering WTF is the difference between parent and derived_from_series. [12:12] But then again this may be the two hours of sleep talking, so you could just ignore me. [12:12] heh [12:12] The interface docstrings should make it very clear, I hope. [12:12] Point. [12:13] So, with that clarified, I have retracted my long-held view that the spec is insane. [12:13] yay. [12:19] heh [12:19] I was operating under the assumption that "parent" meant "parent". [12:19] it does mean, however, that the spec is not clear enough [12:20] I may clear up the ambiguity with s/parent/derived parent/ or something. [12:20] I think that would be good. [12:21] I am still a little worried about making this work for Ununtu though [12:21] I may take a more detailed look at that spec tomorrow. [12:21] Ubuntu, even [12:21] And work out how it would work for Ubuntu. [12:21] the Linaro workflow will be quite different [12:21] Howso? [12:21] I know the Ubuntu one, but not really the Linaro one. [12:21] Does the Linaro one exist yet? [12:21] I doubt they'll derive once, and keep releasing new series [12:22] They'll re-derive each series, then copy some stuff up? [12:22] we are also only making the differences stuff work through packagesets [12:22] I suspect so, but someone like james_w will help clarify that === matsubara-afk is now known as matsubara [12:56] that reminds me [12:56] I've been meaning to ask how visible packagesets are in the UI and how customizable they are [12:56] let me rephrase [12:57] if we wanted to start displaying aggregated bug information for Ubuntu & other distros, with packages grouped by an expert according to some heirarchy, what structures do we already have? [12:58] jml: packagesets are only exposed via the API [12:59] They're also exposed on +queue. [12:59] But that's it.l [12:59]