[00:02] lifeless: One could say the same for any other sourcecode change. [00:02] We could probably have make check that. [00:02] Do a no-op update-sourcecode run [00:04] lifeless: how do you get the query count to show in the top right corner of the page? [00:05] 'visible_render_time default 0 on' [00:05] wgrant: thanks [00:05] StevenK: Is this for the base version thing? [00:05] wgrant: that might be nice [00:06] wgrant: shipit is particularly weird though because it breaks on zcml, not on $specific tests. [00:06] lifeless: bzr-builder is also very good at breaking model imports. [00:06] ah [00:07] wgrant: anyhow, yes, these things all need communication :) [00:07] or we could stop using source code [00:07] Heh, good luck with that. [00:08] why do we use source code and not eggs for that stuff? [00:09] wgrant: Yes [00:09] wallyworld: eggs came along much later. [00:09] and we don't want to go in and eggify the source code stuff...? [00:10] Well, I don't really like eggs. [00:10] And everytime shipit is moved into an egg, it breaks and leaves this rotten smell ... [00:10] wgrant: what is your preference to eggs? [00:11] grah [00:11] wallyworld: Real packages are nice, because they aren't programming languages feeling that they need to reinvent the package manager :) [00:11] We have 3 ways to inject dependencies into our tree: eggs, sourcecode and binary packages from the PPA. [00:11] sample branches are not scanned [00:11] no wonder its approximately useless. [00:11] Also, real packages don't involve downloading unsigned code over HTTP. [00:12] wgrant: You mean that doesn't give you a warm fuzzy feeling? :-P [00:13] wgrant: yes, but python has eggs, java has jars etc - it a common paradigm. i can see the point of keeping such things separate from the overhead of a full blown package that is not universal across os's rpm vs apt etc [00:13] It is a common paradigm, but it also fucking stupid. [00:13] :( [00:13] wgrant: you forgot IMHO :-) [00:13] It isn't just his opinion. [00:14] *Anyone* who has done a bunch of distribution work shares it. [00:14] Except lifeless, because he's special. [00:14] :-( :-( make: *** [check] Error 1 [00:15] I tried running ec2 test --attached [00:15] everything seems to pass [00:15] then I get this ^^^ [00:15] I had several detached ec2 test runs, [00:15] i haven't done any distribution work so i don't yet appreciate the drawbacks of eggs etc. all i see are packaging overheads and competing implementations [00:15] thumper: Full output pls. [00:15] and they all fail [00:15] wallyworld: So, we have a wheel. It's round, it mostly turns without issues, but it has been reinvented by PyPI, CPAN, Gems, ... [00:16] wallyworld: you can't win, really [00:16] And then the reinventors of the wheels decide that they don't want anything to do with distributions. [00:16] eg. Ruby [00:16] wgrant: I've emailed it to you [00:16] "lol, why would people want distribution packages? end users should use gems" [00:16] It seems to be a bunch of NIH to me [00:16] wallyworld: random python upstreams of course would rather not learn new packaging tools and conventions for Debian, OS X, Windows, etc... [00:16] i'm hit, bleeding badly, can't hang on much longer, copping it from all directions..... [00:17] wallyworld: Well, you did just jump up and wave your shirt in the air and yelled "I'm over here, come get some!" [00:17] wallyworld: but at the same time Debian, Fedora, OS X, etc don't want dozens of non-cooperating packaging systems where one would do [00:17] thumper: That is a little bit special. [00:17] spiv: i guess that's one of my points - each os has it's own packaging mechanism, vs eggs for python which is universal [00:17] wallyworld: Universal... for Python. [00:18] wallyworld: it's the "for Python" bit that's the killer [00:18] StevenK: uhm, don't malign me please. [00:18] wallyworld: what happens when a Python package depends on a C library [00:18] And then never mind rpm vs apt vs emerge ... [00:18] wallyworld: etc :) [00:18] StevenK: I *loath* gems jars eggs etc [00:18] lifeless: I was poking fun, not trying to do so [00:18] StevenK: and I've been pretty vocal about that [00:18] I would love to just use debs for lp dependencies. [00:19] thumper: Ah, there it is. [00:19] lifeless: Hm. I had you in my mind as the other way -- so noted, sorry [00:19] spiv: fair point. if only each os would stop adopting it's own packaging implementation [00:19] thumper: There are failures in the stream. [00:19] but the deb python maintainers want *upstream* to decide on how to handle conflicting APIs for multiple versions of the same package on disk [00:19] StevenK: which is never ever ever ever ever going to happen [00:19] thumper: But the summary doesn't show them because of a bug I'm currently writing a test for. [00:19] wallyworld: if only each os was a derivative of Ubuntu :) [00:19] wgrant: why aren't they in the email? [00:19] because python folk wave the virtualenv magic flag [00:20] the result is that every python transition is lockstep [00:20] thumper: jml broke it when he added ec2 list. [00:20] (and I don't mean python VM, I mean individual packages [00:20] The fix is easy, but a test is slightly harder. [00:20] Package management is hard [00:20] and that results in us not being able to shift dependencies at all, which is why we're not using debs. [00:20] see for instance my threads on debian-python (both direct and through folk I've been poking about this) [00:21] StevenK: its easy if we accept that upstream have different drivers than us [00:21] Well, we're using a mix of debs, eggs and source trees [00:21] And random things that people thought it was a good idea to import into the tree. [00:21] wallyworld: Do you see why we don't discuss this often? [00:21] Heh. [00:22] wgrant: :-( [00:22] StevenK: yeah, i had no idea what a feeding frenzy it would create. wow :-) [00:22] StevenK: and the debs give us routine production fail, the source treess toolchain isn't being fully used and the eggs we have to cripple the toolchain because we like verified code [00:22] Baptism of Fire, part #58585 [00:22] <_mup_> Bug #58585: Installer is crashing (all disk activity stops) at scanning mirror message < https://launchpad.net/bugs/58585 > [00:22] StevenK: but you forgot, we *also* use: [00:22] - bundled code in hidden bits of the tree [00:23] * thumper sighs [00:23] - more bundled code in other better hidden bits of the tree. [00:23] lifeless: The debs only fail because there seems to be this magical step where packages jump from the PPA into CAT .... [00:24] StevenK: The main problem is the multiple versions issue. [00:24] StevenK: thats not the fail [00:24] StevenK: the fail is when version A and version B are incompatible with each other as far as LP is concerned. [00:24] StevenK: we then have to simultaneously upgrade the LP tree at the same time, or we have a fail [00:25] :-/ [00:25] which means a more complex rollout - take the entire appserver machine offline, (== 20% drop in capacity - when we are already overloaded), upgrade debs, deploy new LP, bring the servers back online [00:26] StevenK: what we need is what C/mono etc libraries have - sonames. [00:26] then we can install the new debs, lp ignores them, and then we upgrade lp to one that uses the new debs, bingo it works. [00:26] StevenK: in a java world things like e.g. osgi go a long way toward this [00:26] ... or write code that can work with both version of the code. But I understand that isn't always possible. [00:27] but in python 'import' is global, there is no built in concept of 'library version' - you cant say 'import bzrlib 2.2 as bzrlib' [00:27] and the plumbing for this goes deep [00:27] StevenK: its both not always possible and its /harder/ [00:28] osgi rules! [00:28] given the choice of two ways to achieve something, only particularly opinionated or previously burnt folk will choose the harder way. [00:28] wallyworld: for all its mind bending stuff, yes, yes it does. [00:29] so its a long term goal for me to beat enough sense into python to permit this [00:29] or JFDI a workaround in the distro using pkg_resources.require() [00:29] That could work. [00:29] but I suspect we'd end up auto rebuilding the entire python stack [00:29] What is Fedora doing about this sort of thing? [00:30] because the debian python group /don't get it/ [00:30] because they are thinking desktops and 'upgrade every 5 years' style server environments. [00:31] wgrant: I've not seen anything - reality is most folk deploy python stacks using eggs / the django thingy / virtualenv etc - for the previously discussed reasons. [00:31] *and* most really agile things I've seen are php or java, not python, so there isn't that much web world pressure on distros to fix this [00:31] (because python is 10 times slower than either php or java) [00:32] To write, run, deploy, or all three? [00:32] all three [00:32] deploy is about equal actually [00:33] php apps tend to be very lean [00:33] java apps aren't, but they have -wonderful- tooling around development [00:33] and for execution time, php and java eat python for breakfast and lunch [00:34] the only /serious/ python thing at *scale* I know of is youtube : and they don't run CPython, rather they run bleeding edge experimental binaries [00:34] We aren't serious? [00:34] StevenK: we're tiny [00:34] StevenK: all our problems are self inflicted [00:35] we've < 1/2T of data [00:35] wgrant, I think Fedora still haven't got so supporting multiple versions of the python runtime, so I don't think they've tackled this [00:35] lifeless: python has good development tooling too. you don't *have* to use emacs :-P [00:35] james_w: Hah. [00:35] a single (very high end) commodity PC server can run our entire backend load [00:35] And I'm not going to ask about other distros like Arch... given that they've switch /usr/bin/python to 3.x. [00:36] StevenK: we have the potential to be serious if we get our shit sorted and /fast/ [00:36] I will be delighted when we start getting user adoption scale problems [00:36] A year ago I would have said we were doomed to be slow forever. [00:36] rather than 'oh and we don't delete obsolete data' scaling issues. [00:36] But I think that may not be the case. [00:36] wgrant: I wouldn't have taken the job if I thought it was insoluable [00:36] bah [00:36] unsolvable [00:37] lifeless: Yes, but you are crazy :P [00:37] It doesn't dissolve in water either [00:37] Yet very effective. [00:37] * StevenK peers at subunit [00:39] subunit-filter's default operation is to strip sucessful tests. Except one turns up ... [00:42] I just love how the Australian Tax Office can give a recorded message saying that they are too busy to take the call ... for 3 weeks solid [00:44] StevenK: if the stream gets corrupted by things printing to stdout [00:44] lifeless: Such as test failures? [00:44] StevenK: you can see what looks like something being not-filtered, whenthe parser actually thinks 'this is junk' [00:44] StevenK: test failures shouldn't print to stdout, the raise exceptions internally [00:45] StevenK: you can see if this is the case with --no-passthrough [00:45] that will squelch what subunit thinks is chatter on stdout [00:45] I'm already using --no-passthrough [00:45] mm [00:45] then if you're seeing a success, [00:45] ah perhaps a skip ? [00:45] some things fold skip to success [00:45] and older subunit had a bug in that area [00:45] try --no-skpi [00:46] Ahh, no, it's these tests [00:46] There are two are cunningly named [00:46] woo, another stab at timeouts complete [00:46] lp:~lifeless/launchpad/bug-722956 [00:50] I need a review https://code.launchpad.net/~lifeless/launchpad/bug-722956/+merge/51481 [00:50] diff is generating still [00:52] lifeless: so.. curious.. how hard is it to setup a project as a distro w/ source packages and such? [00:53] lol [00:53] SpamapS: impossible [00:53] SpamapS: starting this week julians team is working on making it possible [00:53] SpamapS: a lot of ground work is in place [00:54] they were interrupted by the reorg [00:54] StevenK can tell you how much work remains [00:54] Lots [00:54] Tonnes. [00:54] o/~ I need a review o/~ to the sound of 'I need a hero' [00:54] Getting it working isn't that hard. Getting it effectively usable is another thing. [00:54] I'm working on low-level plumbing [00:54] lifeless: ahh.. thats good.. I think the principia ensemble project I started is going to need package tracking. [00:55] are they actually packages ? [00:55] or just vaguely similar [00:55] they're called formulas.. [00:55] should be called instruments [00:55] or scores [00:55] anyhow ;) [00:55] Heh [00:56] lifeless: agreed.. would have been much cooler. :) [00:56] though if it hadn't been formula.. I wouldn't have been able to come up with the name Principia Ensemble [00:56] SpamapS: if they are similar but not the same, then I wouldn't shove them in as packages - packages are tied up in the whole package building machinery [00:56] lifeless: yeah I don't need that.. I just need sub trackers [00:56] Or do you just want the bugtracking aspects of source packages? [00:56] SpamapS: think crisp and clean; we can bring up a dedicated interface pretty directly. [00:56] Right. [00:57] SpamapS: project groups do that, but the components are 'project' which is pretty big. [00:57] if we were to refine 'bug.target' we could do subtrackers on projects quite easily. [00:57] Yeah I'd expect there to be 300 - 500 formulas [00:57] thats the sound of 20 engineers screaming at me [00:58] Only 20? [00:58] StevenK: 5*5 in theory but we're short atm [00:58] plus I know at least 2 that agree [00:59] lifeless: We're not short any more :( [00:59] wgrant: we are, its just going to stay like that for a while [01:01] wgrant: are you actually working on https://bugs.launchpad.net/launchpad/+bug/575450 [01:02] <_mup_> Bug #575450: Archive:+copy-packages nearly unusable due to timeouts < https://launchpad.net/bugs/575450 > [01:04] lifeless: I haven't touched it for a couple of weeks, and don't have branches with any significant improvements. You may steal it if you wish. [01:04] It's not really difficult. [01:04] I was just trying to make some infrastructure improvements to make testing easier. [01:04] * lifeless gifts wgrant with a laserlike obsession with production performance [01:04] eg. ++oops++ doesn't work on form submissions or the webservice (as yet unsolved), query counts are difficult and non-deterministic (both solved) [01:05] wgrant: those are great things to do [01:05] wgrant: which query counts ? [01:05] wgrant: oh, you mean the session thing? [01:05] That, and your new matcher. [01:06] Although there is still that Storm thing where it will do a SELECT 1 to check if an object exists the second time. But I wonder if store.reset() will fix that. [01:07] lifeless: The most difficult bit of those timeouts it the DAS queries. [01:07] wallyworld, StevenK: mumble? [01:07] thumper: ok [01:07] lifeless: They're produced by createMissingBuilds' calls to getBuildByArch... and that is all a really big mess. [01:07] zomg [01:08] DistributionSourcePackage - 778 repeated lookups [01:08] Where? BugTask:+index? [01:08] bug 705713 [01:08] <_mup_> Bug #705713: Bug:EntryResource Timeout trying to set bug private - death by sql / late evalation < https://launchpad.net/bugs/705713 > [01:08] Heat? [01:09] not obviously [01:09] just 1.4K sql queries [01:10] It was a good thing, though. [01:10] bbiab, doctor visit. [01:10] wallyworld: https://dev.launchpad.net/PolicyAndProcess/MaintenanceRotation [01:12] ok so the consensus right now is that sub-trackers will just have to be implemented using tags at the moment. [01:12] ? [01:12] SpamapS: Unless you want to use a project group, which you probably don't. [01:13] wgrant: right I don't want to register 600 projects.. some of which will have 8 - 20 lines of code. [01:48] I think tags should work well [01:48] we probably have a few small bugs to sort out in making them scale and be usable enough [01:49] but that woul dbe significantly less than actual sub trackers. [01:49] less work I mean [01:53] wgrant: so what are you stabbing today? [01:53] lifeless: The last of checkwatches. [01:54] wgrant: would you like to get the loggerhead thing too? [01:54] lifeless: You mean jam's branch that I reviewed last week? [01:54] I think it just needs a merge to trunk + a merge to the lp deploy branch + a sourcecode rev [01:54] + a bug filed for capturing timeouts [01:54] yeah [01:55] launchpad_loggerhead is in the tree now. [01:55] 15K in one hit. It would be a record of some sort [01:55] It just needs to land. [01:55] oh true [01:55] Which relies on us being out of testfix. [01:55] Which relies on a LOSA [01:55] Right. [01:56] * StevenK writes a test, getting UploadFailed, and sighs at this layer crap [01:56] Is it unacceptable to cheat by forcing the build then landing stuff? [01:56] Haha [01:56] It will fail in the same way and never make it to stable [01:56] if your thing passed ec2 its fine [01:57] Right, it will just fail again, but at least stuff will get landed. [01:57] * wgrant does it. [01:57] wait [01:57] you could rollback the apt thing [01:57] Could. [01:58] file a bug with the checklist of missed systems [01:58] Ah, it's building again anyway. [01:58] Who landed that, anyway? [01:58] link it to a branch to reinstate it [01:58] jtv I think [01:58] StevenK: jtv. The RT says it's rolled out everywhere. [01:58] But apparently not buildbot. [01:58] (it did ask for it to be rolled out to buildbot) [02:08] lp.soyuz.scripts.tests.test_publishdistro.TestPublishDistro.testPublishDistroRun [02:08] soyuz-set-of-uploads_txt [02:08] ^ is that the apt thing ? [02:08] soyuz-upload_txt [02:08] Yes. [02:08] from ec2. sigh, I need a new ami don't I [02:09] Erm. AMI 508 should be enough. [02:09] Was this a really old branch? [02:09] wgrant: I have a different tree for submitting from [02:09] so ec2 vaguaries don't bit me - but when the upgrade is mandatory, it bites [02:12] thumper: wallyworld: now you're on maintenance - 5th top timeout : 26 / 18 RootObject:+daily-builds :) [02:12] lifeless: i'm working on that one right now [02:13] actually, just context switched to land some branches from alst week [02:14] we can't land anthing today [02:14] lifeless: i probably missed the discussion as to what's wrong with landing stuff? [02:18] buildbot's vm was not updated with the new apt, so when jtv landded the code to depend on it we went into testfix and cannot get out without a losa [02:18] or reverting his branch [02:18] And reverting his branch when the fix is so easy and it's not going to cause a QA chain of pain seems bad. [02:20] ok. thanks [02:27] lifeless: is it possible the compaction error is happening because of the low fd limit? [02:42] SpamapS: no [02:43] SpamapS: its a classdefnotfound error [02:43] SpamapS: got a lucid vm ? [02:45] SpamapS: I've pushed my test code to lp:~lifeless/oops-repository/trunk [02:46] SpamapS: just needs the cassandra ppa, the lp ppa [02:55] Hmm. [02:55] Codebrowse is down? [02:55] lifeless: ah ok.. right [02:56] SpamapS: I think its a missing jar that isn't wrapped into the war for whatever reason [02:56] SpamapS: but, branch my code, run make check. [02:56] (oh, read README and install deps) [02:56] 3 hours later.. ;) [02:57] wgrant: faaaark looks like it [02:57] and yet, noone has complained [02:57] spm: are you lurking? cause if you are, and aren't on leave, I'd rather bug you than wake elmo. [02:58] 3am London time :-( [02:58] pjdc: ? [03:01] lifeless: Is codebrowse downness critical now? I thought it had some ridiculously high threshold before it could be treated as such. [03:03] Hm. [03:03] There is no guava. [03:03] Everything else is pingable, so I presume the [03:03] machine itself is down. [03:04] guava replies to pings here? [03:04] In..deed. [03:04] But not from the DC. [03:06] Looks like it is up, and carob can rsync from it, but not ping it. [03:06] All the other prod machines are pingable :/ [03:06] wgrant: guava is in the dmz [03:07] for stupid reasons [03:07] Hmm. [03:07] guava CPU graphs are a bit sad. [03:07] https://lpstats.canonical.com/graphs/GuavaCPU/ [03:08] mwhudson: I suppose this isn't documented anywhere that non-IS can see? [03:09] wgrant: not that i know of [03:09] Heh. [03:09] i guess you can probably see the rt i filed for "please move guava out of the dmz" about two years ago [03:09] Anyway, looks like both codebrowses wandered off to chew on a CPU each at 01:20 [03:13] Ah, Francis rejected it. [03:13] Sad. [03:14] StevenK: OOPS-1884A1684 [03:16] wgrant: its gritical [03:16] wgrant: we have two redundant codehosting instances [03:17] why was the move rejected ? [03:17] wgrant: so yes, if its down, it needs to be fixed. This is bread and butter facilities for our users. [03:17] lifeless: See the bottom of https://rt.admin.canonical.com/Ticket/Display.html?id=33173 [03:18] lifeless: I recall that previously it has been said that it was not as critical as everything. I disagree, but that is how it was. [03:18] if we've killed the arch mirror [03:19] we can move within the main network [03:19] wgrant: the old definition of critical is what you're thinking of, I think [03:19] wgrant: that got overhauled late last year [03:20] I recall it also had the Soyuz threshold at 24 hours, or something stupid like that. [03:20] any launchpad service or feature becoming unusable [03:20] immediately [03:20] ... [03:20] We aim to have none of these, ever. [03:20] ...This includes beta features [03:34] wgrant: :-( [03:34] StevenK: ? [03:34] wgrant: The OOPS you linked me [03:34] Ah, yes. [03:35] I bet I just generated another oops [03:37] Hm. Only the no signer packages do that. [03:38] Why do I have this bad feeling there's a deleted recipe involved. [03:39] There is. [03:39] :-( [03:40] * StevenK tries to swap in the branch details. [03:43] wgrant: So a test for this would be create a recipe, link it to an SPR, call recipe.destroySelf() and then look at the view? [03:43] StevenK: I think so. [03:43] You may need to make sure that the SPRelease has no signer. [03:44] That's going to be fun to do in TAL. [03:44] Why? [03:44] (I suspect it's more your new thing than the signer, but best to be thorough) [03:46] I certainly am blaming my addition [03:51] wgrant: I have a test written, let's see ... [03:54] wgrant: Confirmed, the test fails in the same way as the OOPS. [03:56] * thumper is attacking a sub-100k bug number [03:57] made more fun by the blueprint and email aspect :) [03:57] Whee, 350 lines of traceback [04:04] wgrant: So, I can ignore printing the built by if the recipe is deleted, or I can say 'deleted recipe' -- what are your thoughts? [04:05] I'd say "deleted recipe" [04:05] ... or can I? How do you do if else in TAL? [04:06] Or can haz a link to TAL documentation that doesn't make my eyes bleed? [04:07] You'll have to use a tal:condition. [04:10] There's nothing like genshi's py:when. [04:10] tal:iban [04:10] lifeless: Two drums and a cymbal fall off a cliff. [04:10] boom boom tish ? [04:13] PTRuntimeError: ['Compilation failed', 'HTMLParser.HTMLParseError: EOF in middle of construct, at line 34, column 9'] [04:13] :-( [04:13] pretty simple review: https://code.launchpad.net/~thumper/launchpad/blueprint-email-handler-bug-83414/+merge/51488 [04:14] and un-contentious [04:15] Isn't it illegal to have a Blueprint branch which adds more lines than it deletes? [04:16] no [04:16] lifeless: thanks [04:16] do the simplest thing possible [04:16] velocity with solid code [04:19] wgrant: not when the addition is tests [04:19] doesn't matter what it is [04:19] its part of the same code base [04:19] same rules as for anywhere else apply [04:19] *if* IssueTracker goes ahead, it will get folded in [04:20] if it doesn't, we need it maintained, and that means having a level playing field for patches to it. [04:20] we *must not* make any part of the code less pleasant to work on than another, culturally, technically or otherwise. [04:22] :-([A[APTRuntimeError: ['Compilation failed', 'zope.tal.taldefs.TALError: Invalid variable name "recipe" in expression u\'not sprb/recipe\', at line 37, column 9'] [04:22] Can I not do that in TAL? [04:22] StevenK: what are you trying to do [04:22] StevenK: Not like that. [04:22]