[00:00] i don't think i did make sampledata [00:00] let's see what bzr says [00:00] poolie: did you merge db-devel perhaps? [00:01] i did, on the 19th [00:01] that will explain it. [00:01] perhaps i should rebase my changes onto devel [00:01] yes please [00:04] gary_poster: /expander [00:04] gary_poster: should that perhaps be /+expander ? [00:04] lifeless: yes, almost certainly [00:04] Please feel free to change on LEP (or not) [00:05] * gary_poster needs to run [00:05] ciao [00:05] bye [02:02] lifeless: gee, not even a whoop for lp:launchpad changing [02:03] I feel deflated :P [02:09] thumper: whoop [02:09] thumper: with leave on monday, I'm feeling a little flat out today [03:25] :-( [03:25] http 500 on https://launchpad.net/ubuntu/+bugs?field.tag=needs-packaging [03:25] second time lucky [03:46] * thumper afk [07:25] wgrant: Do you happen to be around? [07:30] StevenK: Hi. [07:30] wgrant: Hai [07:30] wgrant: Can you have a squiz at https://code.edge.launchpad.net/~stevenk/launchpad/db-add-distro-parent/+merge/41430? [07:31] 48 +Derived Distroseries [07:31] 49 +-------------------- [07:31] 50 + [07:31] 51 +Distroseries can be derived from other distroseries. This is completly [07:31] 52 +different from the derivation mentioned above. [07:31] Yes, I need to fix that bit [07:32] StevenK: What sort of review do you desire? [07:32] wgrant: Basically, "Do you think it's crack?" [07:33] StevenK: Has it been fully specced out? [07:33] If not, it is crack. [07:34] I haven't seen an analysis of all of the cases. [07:34] I think it's a good plan, because .parent is overloaded and this derivation work is making it be more so [07:35] Right, I agree. [07:35] But I'd like to see it fully thought through first... [07:35] We need to do something like this. [07:35] We do have a LEP :-) [07:57] wgrant: doctest twiddled, fwiw [08:22] jkakar: hi? [08:24] lifeless: Hi! [08:24] https://dev.launchpad.net/LEP/PersistenceLayer may interest you [08:26] lifeless: Cool, thanks, will check it out! [08:38] "How will we measure how well we have done? It Will Be Obvious." <-- Cute. === almaisan-away is now known as al-maisan [09:02] good morning [09:04] Hello [09:31] hi mrevell! [09:31] Hi there jtv, how are you? [09:31] fine, thanks—you? === al-maisan is now known as almaisan-away [09:53] hello [10:08] hi [10:08] I ran a bunch of tests on ec2 last night [10:08] they all passed [10:08] but the test run had a non-zero exit code [10:14] jml: sorry forgot to tell you about the test run yesterday, everything seemed ok. At least I got the usual amount of local failures I get. [10:15] but whether they are different failures on the same tests I don't know. I get 60 codehosting tests failing [10:36] bigjools: thanks. [10:36] I need to figure out my local failures at some point ... [10:39] jml: loads of my codehosting tests fail because they are trying to open /usr/lib/python2.6/dist-packages/bzr which does not exist, any ideas? [10:39] this is when it runs some bzr commands [10:39] hmm [10:39] yeah, I have an idea [10:39] but it's only half-formed [10:39] * jml pokes a bit [10:40] it's doing a "bzr push -d first branch-address" [10:40] bigjools: lp.codehosting.get_bzr_path is either buggy or your environment is buggy [10:40] bigjools: try running it in ./bin/ipy to see what you get [10:42] bigjools: also, try 'ls eggs/bzr-2.2.0-py2.6*/EGG-INFO/scripts' [10:42] "bzr" [10:43] when needs running in ipy, exactly? [10:44] what, even [10:48] bigjools: lp.codehosting.get_bzr_path() [10:50] jml: it returns the non-existent path [10:50] bigjools: yay, the bug is isolated [10:50] :) [10:50] bigjools: I guess step through that with pdb and see what's misbehaving [10:51] bigjools: basically, it looks like you are getting system bzr rather than the bzr in eggs [10:51] I am going to remove the egg and re-generate it, it worked with a similar bunch of test failures [10:51] bigjools: a fine plan [10:52] I am tempted to blow all the eggs away, it'll clean up all the old crap [10:56] bigjools: that could also work [10:57] we need a development story for that [10:57] yay, it works now [10:57] thanks jml [10:57] bigjools: np [10:58] bigjools: can I persuade you to run the lp.codehosting tests with that new version of Twisted? [10:59] jml: am doing so as we type :) [10:59] bigjools: thanks :) === matsubara-afk is now known as matsubara [11:32] what's the magic command to make my download-cache get, err, downloaded again? [11:33] bigjools: as in, to get a fresh one? [11:33] yes [11:33] bzr co --lightweight bzr+ssh://bazaar.launchpad.net/~launchpad/lp-source-dependencies/trunk/ download-cache [11:35] of course, it's a checkout [11:36] rather than say, a Debian archive, yes :) [11:36] well, we've got so many different ways of getting dependencies now, it's hard to remember what does what [11:36] definitely [11:36] I was just thinking that it would kind of make sense to distribute the eggs as a debian archive [11:37] ! [11:38] download-cache is a bit of a misnomer isn't it ... [11:38] bigjools: I believe the default buildout behaviour is to fetch code from pypi and run that [11:39] maxb: well, storing immutable tarballs in a bzr branch isn't exactly sensible either. [11:39] jml: It doesn't seem to work like that, at least not here. [11:39] jelmer: what doesn't work like what? [11:40] Sorry, that was a bit vague. I meant buildout doesn't seem to fetch code from pypi for me. I have to have it in the download cache. [11:40] that's because we have 'install-from-cache = true' in our buildout.cfg [11:41] ah, ok. [11:41] Means it's no longer effictively a cache though :-) [11:41] quite so. [11:42] it's a sensible name in the broader context of what buildout does, but not for our use. [11:43] jml: I think shipping the eggs in a deb is a good idea. it's far from ideal but better then what we have at the moment. [11:43] It also means it will be easier to e.g. move some of the eggs into their own proper package. [11:44] jml: where would you have the package install them though? In a Launchpad-specific directory that lp adds to its PYTHONPATH, or to the system python path? [11:44] that lp-source-dependencies is also carring loads of old versions, which is kinda annoying to download [11:44] carrying, even [11:45] it doesn't make sense that we create a heavyweight rather than a lightweight checkout [11:45] jelmer: mine's lightweight :) [11:45] so's mine [11:46] ahh right, I'm just reminded of what actual advantage eggs have over packages: rollback [11:46] you can do that easy with packages [11:46] and it's audited properly [11:54] jelmer: hey, did you get my email about testing packages etc? [11:55] jml: yep. Thanks for setting that up [11:55] my pleasure [11:55] as mentioned, the actual working daily builds are probably packaging nightmares [11:56] and most of the things aren't actually working [11:56] jelmer: so I'd appreciate it if you or others who actually know what you're doing could take a look and fixinate [11:57] jml: I'll see if I can contribute. Ideally we'd set these up to merge in their Debian packaging branch as well , that way we (and by we I mean lifeless, as he's the Debian maintainer) only have to do the work once. [11:57] jelmer: that'd be great. [11:57] oh [11:57] that reminds me [11:58] jelmer: the testr in that PPA has incremental output display, I think. [11:58] jml: I should mention, I still find daily builds a bit hard to work with in practice because it's hard to easily find out what packages are working on which are broken. [11:58] jml: Ah, neat. [11:58] jelmer: can you clarify? [11:59] * jml really wants daily builds to be _easy_ to work with in practice [12:00] although tbh, probably the best thing we can do is reduce the end-to-end latency [12:00] jml: I have a lot of daily builds, in particular related to bzr. It's very hard to get an idea of which ones are currently broken. [12:00] ahhh [12:01] yeah, I can see why that would be a thing [12:01] jelmer: have you filed a bug about it? [12:01] jml: The list of recipes doesn't show which ones have last produced a source packge successfully and it also doesn't show for which resulting source packages binary packages could be built. [12:02] nor indeed which have failing binary build steps [12:03] jml: I filed bug 669260 about this earlier. [12:03] <_mup_> Bug #669260: last build result in recipe overview page [12:04] jelmer: ta [12:15] bugs folk, is there a tag for https://dev.launchpad.net/LEP/BetterBugSubscriptionsAndNotifications ? [12:17] jml: so I see some test errors with the new Twisted [12:18] I'll re-run with the old one just to make sure [12:18] bigjools: thanks. could you also paste the errors? [12:19] yep, once I confirm === Ursinha-afk is now known as Ursinha [12:44] jml: http://pastebin.ubuntu.com/536278/ [12:44] w/o new Twisted they are passing [12:44] * bigjools -> lunch [12:44] bigjools: thanks === mrevell is now known as mrevell-lunch === Ursinha is now known as Ursinha-afk === mrevell-lunch is now known as mrevell === almaisan-away is now known as al-maisan [14:09] jelmer: and 'testr failing --list' too [14:47] flacoste: http://people.canonical.com/~jml/Tests-finished-each-minute.png ; http://people.canonical.com/~jml/Tests-finished-each-ten-minutes.png ; http://people.canonical.com/~jml/Tests-finished-each-hour.png [14:47] mrevell: ^ [14:47] * mrevell l;ooks [14:48] flacoste: http://paste.ubuntu.com/536318/ [15:54] jml: so do you remember that what seemed to a pointless xmlrpc "echo" command that the buildd-manager issued at the start of every build? === salgado is now known as salgado-lunch === matsubara is now known as matsubara-lunch [16:20] bigjools: vaguely [16:20] bigjools: does it have something to do with our mystery connect() timeout? [16:20] jml: I think I found why it was there [16:20] allenap: hi [16:20] you guessed.....wisely [16:20] bigjools: I need to know more. [16:21] the first packet to the Xen slaves is sometimes discarded [16:21] I have a test script that demonstrates this [16:22] bigjools: may I see the script? [16:22] allenap: the build just broke with that error we saw a while ago, eh? [16:22] jml: Yep :) [16:22] jml: http://pastebin.ubuntu.com/536317/ which calls out to http://pastebin.ubuntu.com/536317/ [16:23] err that';s the same [16:23] jml: Shall we try kicking it off again, and I'll have a go at the fix you suggested in the "ec2 land submission problems" thread? [16:23] allenap: just pulling up the thread... I seem to remember saying something about it [16:23] jml: well anyway there's a top-level script that resets the slave and then immediately calls that one [16:24] bigjools: is the xmlrpc server running inside an instance? [16:24] bigjools: a xen instance, that is [16:24] jml: the reset script does not exit until it's running [16:24] allenap: I would suggest applying the fix before retrying [16:25] jml: Okay, I'll have a bash at that now. [16:25] allenap: I mean, it's not that hard, and it might make our branches land ~3.5 hrs earlier :) [16:26] jml: Oh yes :) I'll email the list too. [16:26] bigjools: sorry, I don't really know what the reset script does or why exiting matters [16:27] bigjools: also, in what way is that paste different from the script I put on the bug? (is it just that it actually works & reports a little better?) [16:27] allenap: thanks! [16:27] jml: it restarts the Xen guest and then waits until the slave manager responds with a BuilderStatus.IDLE. At that point we know for sure it's up and working. Afterwards we poke it with that script which sometimes fails with the first connection. [16:28] jml: it logs the first packet failures [16:28] bigjools: were we only seeing the error against Xen guest slaves? [16:28] yep [16:28] nice [16:29] bigjools: so I guess putting the echo (or even a TCP connect & disconnect) back in, with a nice big fat comment of course, would fix the issue [16:29] jml: lp:~julian-edwards/launchpad/timeouts-feature-bug-681426 [16:29] bigjools: oh, is there a bug for the issue with xen discarding packets? [16:29] :) [16:30] surely someone else must have noticed this... [16:30] https://bugs.edge.launchpad.net/launchpad-buildd/+bug/586359 [16:30] <_mup_> Bug #586359: Virtual builders are sometimes very slow to accept connections [16:30] jml: I saw some PDF describing packets dropped from the DMA ring when system load is too high [16:31] bigjools: I meant a bug against xen -- it's not actually a canonical.buildd issue is it? [16:31] zoinks [16:31] we're not sure exactly where the bug is [16:31] we need to prove that first [16:31] but it looks very suspicious [16:31] so 586359 stays open even when the builder has the workaround? [16:32] yeah, I need to add a soyuz task [16:32] *nod* [16:32] bigjools: well, you have my sincerest preliminary congratulations :) [16:33] jml: team effort, I can't take all the kudos [16:33] but thanks :) === al-maisan is now known as almaisan-away === matsubara-lunch is now known as matsubara [16:59] jml: Could you sanity check my branch? https://code.launchpad.net/~allenap/launchpad/zope-test-in-subprocess-unicode-issue/+merge/41886 [17:02] allenap: I have to look up what codecs.getreader does now :) [17:04] allenap: have you tried running a subclass of ZopeTestInSubprocess? [17:06] morning [17:06] morning lifeless [17:06] jml: I've run all the tests that rely on it and they ran fine. [17:06] allenap: cool. [17:06] lifeless: good morning. === beuno is now known as beuno-lunch [17:07] jml: Thanks for that. Shall I just lp-land --testfix it now? [17:07] allenap: deffo [17:07] jml: care to look at https://code.launchpad.net/~julian-edwards/launchpad/timeouts-feature-bug-681426/+merge/41881 please? [17:07] it's short :) === salgado-lunch is now known as salgado [17:12] bigjools: you should probably link to the bug report in the comment. [17:13] ok [17:13] other than that, good to go [17:13] cheers [17:16] If I finally put this one to bed I am going to be a very happy man [17:16] jml: could you eyeball LEP?PersistenceLayer? Thanks. [17:17] lifeless: fo shizzle [17:24] allenap: btw, do you know of anyone tackling the other failure? the list was silent. [17:27] I've got a bunch of specs to read. Heading afk to do so & eat. Back later to catch the antipods. [17:32] jml: Oh, the db-devel one? No, no idea. I didn't even look at it myself :-/ [17:40] lifeless: FYI I think I figured out the timeouts [17:40] so I'm going to skip that feature-controlled builder fail bug [17:41] bigjools: It might be a good idea to have a realtime knob for number of failures regardless. [17:41] bigjools: but its up to you :) [17:41] lifeless: I don't think that's at all necessary [17:41] IMNSHO :) [17:41] jml: If you mean danilo's, then I also don't know. [17:42] bigjools: the next time you land a branch changing or disabling that figure.... think of me :) [17:42] well if this branch doesn't work then I'll do it [17:42] 6 months, 9, I'm sure you'll remember ;) [17:42] lifeless: how could I not always think of you [17:42] awww :) [17:43] so, time to go. heavy eyes from staying up late to watch the cricket [17:43] night al [17:43] l [17:44] gnight [17:45] allenap: ping === beuno-lunch is now known as beuno [18:24] flacoste: hi [18:27] lifeless, do we have only the 'devel' and '1.0' versions of the webservice or are there others? do you know where's that configured? [18:27] salgado: we have beta, 1.0, devel [18:27] in the [18:27] exported() clal [18:27] call [18:28] lifeless, I was asking about the config which has the list of versions [18:28] beta was the first devel, then when we did 1.0 we deleted stuff we didn't like in beta, and devel is like the next beta. [18:28] salgado: oh, the master list? [18:28] uhm [18:28] lib/canonical/launchpad/rest/configuration.py [18:28] I think [18:28] looks like [18:29] yeah, that's itactive_versions = ["beta", "1.0", "devel"] [18:29] lifeless, so, what is beta? should the blueprints API be available there? [18:30] no [18:30] see version_descriptions [18:30] in that file [18:30] oh, I see [18:30] new stuff should generally just go into devel. [18:31] if we're -totally- confident in its stability, putting it into 1.0 is fine, putting it into beta is pretty pointless given its substantial skew vs 1.0 [18:32] hi lifeless [18:32] flacoste: shall we catchup/handoff ? [18:33] lifeless: sure [18:33] lifeless: i can do it later also [18:33] lifeless: really up to you [18:35] now is good [18:36] lifeless: skype? [19:06] allenap: no worries [19:17] flacoste: https://bugs.launchpad.net/launchpad-foundations/+bug/651766 [19:17] <_mup_> Bug #651766: please record concurrent query count on servers [19:19] :( [19:21] looks like db_lp is still broken [19:21] and that something similar is now also affecting lp [19:22] lifeless: ping [19:22] otp [19:23] ok. [19:23] won't be long [20:12] morning [20:12] jml: I have a quick question for you [20:12] thumper: sure. I'm otp atm. [20:13] jml: I remember in the years past you had looked at extracting the codehosting server into a separate project, did that happen? [20:14] thumper: not really. I made txsshserver, but it's kind of dead. lp.services.sshserver is much closer to the spirit of that. [20:14] thumper: my goal was always to extract the generic ssh stuff we did, rather than a separate codehosting service [20:14] ah, ok === salgado is now known as salgado-afk [20:28] thumper: did you want to have a catch up call? [20:28] lifeless: pong [20:28] jml: sure, I'm just going through emails [20:28] now? [20:28] mumble? [20:28] thumper: mumble's great. one sec. [20:29] thumper: join me in the strategerie [20:29] allenap: I humbly suggest that the bug you qa-ok'd which timed out on staging and has additional repeated queries, isn't qa-ok :) [20:34] lifeless: Okay, fair enough :) There's nothing built on it so I'll revert it. [20:34] allenap: if the additional queries *do not* happen on users outside the saved-search user group [20:34] allenap: then I think it is qa-ok [20:34] allenap: but I couldn't tell if that was the case, thus pinging you [20:35] wow, one of the top landing page on launchpad is https://launchpad.net/~jd-team/+archive/jdownloader [20:35] even above /ubuntu ! [20:36] lifeless: There are no saved searches yet, and it's not possible to create them, so this will happen to everyone. I'll revert it. [20:36] allenap: thanks. [20:36] allenap: you need to mark the bug qa-bad and add a tag bad-commit-12345 (the revno in stable thats bad) [20:36] allenap: when you revert, you need [rollback=12345] in your commit message (or use the --rollback option to lp-land) [20:37] lifeless: Do I need to run the revert via ec2, or just land it? [20:37] I generally just land [20:38] if its a straight revert [20:38] if its days old or something ec2 can be a good sanity check ;) [20:38] lifeless: In this case I think it'll be safe. I'll run some sanity checks locally. [20:39] I got weird behavior on api ?ws.op=getBranches; There is branch that's fully merged and it sometimes appear on the list and sometimes not which results in useless messages sent by my irc bot. [20:41] ccxCZ: for how long has it been fully merged? [20:50] ~40 minutes now [20:53] ccxCZ: hmm, that's weird [20:54] maybe it's gone now, let me check [20:54] but it lasted for 20 minutes at least [20:55] should the branch appear there or not? [20:58] seems it settled now, but fluctuations like this are seriously annoying === matsubara is now known as matsubara-afk [21:08] is your script logged in ? [21:10] no, I just do plain http request on the uri [21:11] then you're getting cached responses [21:11] from the squid cluster that anonymous requests go to. [21:11] If you want accurate data, you need to sign in and use the auth cookie [21:12] abentley: http://paste.ubuntu.com/536445/ [21:12] the reason its fluctuating is that we have multiple squids and one had an older version of the answer, the other a newer version [21:12] i see [21:13] is there way to stay at one proxy without having an account? [21:13] no [21:14] primarily because its a load balancing cluster; it would defeat the point :) [21:15] poolie: ping [21:18] and is there a way to tell apart older data from newer for example by some http header? [21:20] allenap: also [21:20] allenap: you might want to revert your subunit change [21:21] Date header seems usable, how often do responses stay in squid? [21:21] lifeless: why would he want to do that? [21:21] because subunit's parser wants bytes [21:21] no it doesn't [21:21] it *ought* to want bytes [21:22] it does want bytes [21:22] it embeds binary data literally in-line in its output [21:22] you'll get unicodedecode errors trying to read a subunit stream as utf8 [21:22] if there is nonutf8 data in any of the attachments. [21:22] TypeError: _StringException expects unicode, got "lost connection during failure report of test 'lp.services.job.tests.test_runner.TestTwistedJobRunner.test_timeout'" [21:23] yes, there is a problem there. [21:23] passing a unicode file handle in is not a solution. [21:24] or at least, not a very good one [21:24] a solution would work. [21:25] you think the failures we're seeing on the bot now are related to allenap's fix? [21:26] no [21:26] I think that allenaps fix will break if/when any binary data attachments that are not utf8 get into the stream [21:27] lifeless: Okay, I'll revert that one too :) Once we're out of testfix anyway. [21:27] jml: this is the current subunit code: [21:27] def _lostConnectionInTest(self, state_string): [21:27] error_string = u"lost connection during %stest '%s'" % ( [21:27] state_string, self.current_test_description) [21:28] error_string *is* unicode. [21:28] upgrading the subunit used is a solution. [21:29] ahh, *that* will be why I didn't see the bug when I looked a week ago [21:29] because I was looking at a version that already had the fix [21:30] yes [21:30] lifeless: Via the Launchpad PPA [21:30] ? [21:30] allenap: however we do it, we should do it. [21:30] (I don't know ;P) [21:30] Okay. I'll file a bug for that. [21:30] thanks ! [21:33] we probably also want one that subunit *accepted* a unicode stream at all [21:34] allenap: I'm sorry that you went to the effort to fix it and had this happen :( [21:37] allenap, lifeless: if a newer subunit isn't availabe in Ubuntu, someone needs to upload it to the ppa [21:38] and then upgrade the launchpad-dependencies to depends on that newer version [21:38] and then file a RT to have it upgraded in the data centre (PQM +buildbot images) [21:38] upgrading the ec2 images is also probably a good idea [21:38] lifeless: That's alright, it was a pretty small change, and it put it where you could comment on it. [21:39] "Xen guests sometimes ignore the first network packet", wow i didn't expect the builder problems to be at that level [21:40] mwhudson, where did you read that [21:40] mwhudson, ? [21:41] more interesting to me is why we're seeing interrupted streams; I suspect something is printing to stdout [21:41] *or* we're mixing stderr into stdout. [21:41] jml: ^ [21:41] mwhudson: That's not it. [21:41] * jml blinks [21:41] mwhudson: Non-virt builders are also affected. [21:42] lifeless: can you phrase that as a question? [21:42] cody-somerville: some soyuz bug [21:42] wgrant: ah [21:42] LP #586359 [21:42] <_mup_> Bug #586359: Virtual builders are sometimes very slow to accept connections [21:42] wgrant: bigjools said otherwise when I asked him about it [21:42]