[00:58] lifeless: ^ Did you get distracted? [01:00] yes [01:07] StevenK: uhm [01:07] auditorfixture appears to be auditor ? [01:08] How the heck did I screw *that* up [01:10] NFI [01:10] but [01:10] you did :) [01:10] Yeah, I'm just about to push --overwrite [01:14] lifeless: It should look better now? [01:20] p[-;;;;;;;;;;;;;;;;;;;;;xccccccccccccccccccbvvvvvvvvvvvvvvvvvvvvv [01:20] uhm, cat. [01:20] Hah [01:21] does runserver support dynamic port allocation ? [01:23] lifeless: From what I saw, yes. [01:23] StevenK: you did something similar to txlongpollfixture, I presume; that did what I did for rabbit, which doesn't support dynamic port allocation [01:24] a better thing, if its doable is to just run runserver and watch what port it chooses [01:24] StevenK: that said, if this works, great. [01:24] Which it doesn't. [01:25] But I've hit the limit of Django-ness -- I dislike the idea of forking off python to run it [01:25] StevenK: my main concern looking at is is that we'll have the same bugs that rabbitfixture has had w.r.t. race conditions on startup (such as the timeout logic); we should perhaps consolidate the common code into fixtures [01:25] StevenK: running django in a subprocess is entirely appropriate [01:26] StevenK: we /do not want/ to try and import django into our python environment - for multiple reasons, including django having a honking great set of global variables that will collide [01:26] https://docs.djangoproject.com/en/dev/ref/django-admin/#runserver-port-or-address-port suggests it does not do dynamic ports. [01:27] StevenK: ok, so it doesn't work. What happens, specifically ? [01:29] * StevenK rips out the devnull, so he can see what is going on [01:29] StevenK: oh, I see the devnull. Don't do that :) [01:30] Hm, smoking gun [01:30] StevenK: you can capture the stdout to e.g. the logfile [01:30] StevenK: and you won't lose information that way [01:30] devnull is for stdin [01:30] However, for this I wanted the output to the terminal [01:31] python manage.py is all well and good -- when you're in auditor's directory [01:31] bleep, I misread. [01:31] yes. [01:32] So I wonder if auditor is hiding somewhere I can call it ... [01:32] why did you want terminal? To see what was up ? [01:32] The failure message should have given you the log file ocontents. [01:32] lifeless: Nope, I was just getting the raise Exception("Timeout .... [01:33] StevenK: so you have a fixture object [01:33] that.getDetails() should give you a path in, for future reference. [01:33] e.g. f.getDetails().items()[0].itertext() [01:33] lifeless: In _start(), that's going to be self, isn't it? [01:33] StevenK: yes, but _start doesn't need to change. [01:34] StevenK: you already attach the logfile; the caller can choose to look [01:34] StevenK: TestCase.useFixture attaches the details before it sets up, IIRC, so it should be doing this. [01:34] lifeless: I think _start() does need to change, since the error is "python: can't open file 'manage.py': [Errno 2] No such file or directory" :-) [01:34] StevenK: doesn't need to change w.r.t. reporting :P [01:35] StevenK: you need to introspect to get the path to the auditor egg, txlongpoll should have example code. [01:35] txlongpoll or txlongpollfixture? [01:35] the latter [02:00] lifeless: Right, I see how txlongpollfixture does it. But auditorfixture doesn't have parts/auditor :-( [02:01] lifeless: And I have a [auditor] section in buildout.cfg [02:08] * wgrant cries [02:08] for team in administers_and_in: [02:08] if (bug_supervisor is not None and [02:08] not team.inTeam(bug_supervisor)): [02:08] continue [02:08] inTeam is not preloaded [02:09] StevenK: uhm, your auditor section doesn't define an entry point [02:09] lifeless: It needs to? My buildout-foo is very weak [02:10] StevenK: I *think* so, it generates the right interpreter etc [02:10] I just realised, running 'python manage.py' is wrong because of that, it should be a buildout created python executable [02:11] which the entry points are used to do [02:12] lifeless: I still need a path to manage.py, I don't think it matters which interpreter I use. [02:12] txlongpollfixture somehow ends up with parts/txlongpoll [02:14] uhm [02:14] not sure where to pull on this one [02:15] you don't run the manage.py from auditor directly [02:15] python version mismatch etc stuff [02:15] Not even the unpacked egg? [02:15] indeed [02:15] txlongpollfixture doesn't either. [02:18] Ah. So what should I do? [02:18] so the basic idea [02:19] is you have a function which can be an entry point [02:19] e.g. def foo(argv=None): [02:19] if argv is None: argv = sys.argv [02:19] ... [02:19] that is probably inside manage.py already [02:20] you point at the *python path* to that function, which will be inside auditd (and you'll probably know far more about django glue than you want to by the time you're done) [02:20] buildout generates a local executable for you in bin/ when you set scripts= [02:20] see entry-points = twistd-for-txlongpoll=twisted.scripts.twistd:run [02:20] scripts = twistd-for-txlongpoll [02:20] for instance, in LP's buildout.cfg [02:20] so you'll have something like [02:21] entry-points = auditord-manager=auditord.manager:run [02:21] scripts = auditord-manager [02:21] then, the fixture runs bin/auditord-manager [02:22] probably adjusted to be os.path.join(os.getcwd(), 'bin/auditord-manager') [02:22] because of the cwd param to Popen [02:23] StevenK: does that make sense? [02:23] lifeless: Barely [02:24] lifeless: So I need a auditord module now? [02:24] you have one [02:24] the django app [02:25] whats the auditord project name ? [02:25] auditor [02:28] auditlog is your module name [02:29] you need to move / symlink or something manage.py within auditlog from the looks of things [02:29] and ugh yeah the one thing I really don't like about django - settings [02:30] Um [02:30] That's the one thing? [02:30] Haha [02:30] It's one of the worst things, but lots of the rest is also hideous :) [02:30] lifeless: And apparently a fixture would be *easy* [02:30] ... and other lies I've been told. :-) [02:32] lifeless: I'm not sure what module you want me to jump into? [02:32] auditor.manage seems pointless [02:34] lifeless: Shouldn't manage.py be replaced by a buildout-generated bin/manage? [02:35] wgrant: yes, thats what I'm describing [02:36] StevenK: you have a package 'auditlog' [02:36] StevenK: We have a project 'auditord', with a public name of 'auditor' and package 'auditlog' - I think you should choose one and go with it [02:37] lifeless: I can't! [02:37] StevenK: why not ? [02:37] You can't call the app 'auditor' if the project is. [02:37] .expand() [02:37] I'm pretty sure you can. [02:38] lifeless: I never picked the name auditord, that was you [02:38] It just gets a bit odd, due to implicit relative imports. [02:38] ok, so, auditor, fine. [02:38] What happens if you use the name auditor for the package ? [02:39] steven@undermined:~% django-admin startproject foobar [02:39] steven@undermined:~% cd foobar [02:39] steven@undermined:~/foobar% python manage.py startapp foobar [02:39] Error: You cannot create an app with the same name ('foobar') as your project. [02:39] Try it anyway [02:40] It'll probably work [02:40] I just did? [02:40] You tried using Django's stupid tool to do it. [02:40] And? [02:41] I'd been avoiding Django just fine until you two railroaded me into auditor. [02:42] https://code.djangoproject.com/ticket/1908 [02:43] Well [02:43] I never suggested Django is a good fit. [02:43] It's really not. [02:43] 5 months ago [02:43] Triage Stage changed from Accepted to Fixed on a branch [02:43] The bug has been fixed in trunk, apps with same name as project name can be created now [02:44] wgrant: I think that was lifeless [02:44] StevenK: what was me ? [02:44] Suggesting Django [02:45] its a preferred tech [02:45] I've said explicitly that I think its sufficient for the small targeted things we need [02:45] that it has some very good values around lean-ness, clean& understandable code, and performance [02:46] Um [02:46] and that it may be too heavyweight for some things (like gpg-httpd) [02:46] The problem is not that it's insufficient. [02:46] And then you try and write a fixture and go insane [02:46] StevenK: you're breaking new ground [02:46] It's that all of the awkward stuff it provides is entirely useless for this sort of service. [02:46] For auditordloger, the useful stuff it provides over raw WSGI is an ROM [02:46] ORM [02:47] * lifeless notes that this is a distraction [02:47] Clearly you want the app to be called auditor too, which means I need to look at packaging Django 1.4 [02:47] StevenK: auditlog is a package name, that is the package you need to make a manager.py exist in, it's probably easiest to have it be a new file with contents derived from the django made one. [02:48] StevenK: I do, but even that is a distraction [02:48] StevenK: and, its one that can be solved later. Because nothing in LP will know about *either* auditor or auditlog. [02:48] lifeless: Why does it need to exist under auditlog? [02:48] StevenK: so that buildout can import it. [02:48] But buildout can import from auditor too? [02:48] auditor/__init__.py exists, so it import auditor should work [02:49] wait, what [02:49] OHHH [02:49] crazy f*d up layout. [02:49] ok, so yes, auditor.manage:run [02:49] and edit manage.py to have a run() method [02:49] (and the usual if __name__ == '__main__': boilerplate [02:50] manage.py already has __main__ boilerplate [02:50] it does, it needs altering [02:52] e.g. http://paste.ubuntu.com/998398/ [02:54] StevenK: btw, we probably want to change auditor from being a django project to being a django app (and then have a project which is for testing created on the fly, and one for prod deployment) [02:54] StevenK: but again, Future Work. [02:55] StevenK: I will make one note on the naming of things; the reason I propose auditord is so that auditor is available for a client library, if you were to decide you want one. [02:55] StevenK: as it is, you'll have to use auditorclient or something instead. [02:56] Client library? Err? [02:56] It returns JSON, how can that possibly require a client library? [02:57] you never know your own future :) [02:58] I'm not asking you to rename it, I'm explaining why I'm sticky on the d's - its an easy default and makes it easier for me to think about names. [02:58] I can see a gun and my temple in my future [02:58] I don't think quake3 supports that. [02:58] Who said anything about Quake 3? [02:59] lifeless: bin/auditor-manage can not import auditor.manage [03:00] StevenK: I did, because why not?. [03:01] StevenK: so, thats python path debugging time [03:02] sys.path = ['/home/steven/auditorfixture/parts/auditor', '/home/steven/auditorfixture/bin', '/usr/lib/python2.7', '/usr/lib/python2.7/plat-linux2', '/usr/lib/python2.7/lib-tk', '/usr/lib/python2.7/lib-old', '/usr/lib/python2.7/lib-dynload', '/home/steven/auditorfixture/eggs/auditor-0.0.1-py2.7.egg', '/home/steven/auditorfixture/eggs/Django-1.4-py2.7.egg'] [03:02] and is manage.py present within /home/steven/auditorfixture/eggs/auditor-0.0.1-py2.7.egg ? [03:03] Sigh, no. [03:04] I guess that's due to find_packages() in auditor's setup.py [03:12] StevenK: yes, I suspect so [03:13] StevenK: note that its not finding the top level __init__.py at all [03:13] StevenK: so your egg only contains auditlog. [03:13] Yes [03:18] http://paste.ubuntu.com/998418/ [04:01] lifeless: Oh yeah [04:01] lifeless: I discovered on Friday evening that my optimisation of bugsummary was bad. [04:02] lifeless: I'd accidentally typoed so it flushed the journal into the journal, so it was writing to a temp table, not a real one. [04:02] So it's only about 2.5x faster than before. [04:03] s/flushed the jounral into the journal/flushed the temp journal into the temp journal/ [04:03] And that's with journal FKs removed. [04:04] wgrant: still, 2.5x is fast enough to run two in parallel [04:05] lifeless: I have a cunning plan to avoid having to do that. [04:05] A couple of unresolved issues, but it should work. [04:08] StevenK: that pastebin was to you, should work. [04:10] StevenK, wallyworld_: https://code.launchpad.net/~wgrant/launchpad/bugsummary-v2-app-no-fixed-upstream/+merge/106557 [04:11] * wallyworld_ looks [04:17] wgrant: looks ok to me. lots of code deletion :-) [04:17] wallyworld_: Yup. Thanks. [04:47] https://bugs.launchpad.net/lava-deployment-tool/+bug/1000819/@@+huge-vocabulary timeouts are known? [04:47] <_mup_> Bug #1000819: uwsgi 1.0.3 no longer available < https://launchpad.net/bugs/1000819 > [04:48] https://lp-oops.canonical.com/oops.py/?oopsid=OOPS-f6d562060ee52035ead5254827fee6ab\ [04:52] mwhudson: Yeah, branch search is criminally bad [04:53] :( [04:54] is it just select from branch where unique_name like '%${TERM}%' or something? [04:54] The query is awful awful awful [04:54] But it shouldn't be as slow as it is. [04:55] * mwhudson looks at the code [04:55] It's in the OOPS [04:55] Top statement. [04:55] i think my bad version is a lot better than this [04:55] Oh [04:55] wut [04:56] SELECT Branch.id [04:56] FROM Branch, [04:56] Product [04:56] WHERE Branch.product = Product.id [04:56] AND Product.name LIKE u$STRING) [04:56] I'm pretty sure that comes under "who could possibly think that was useful or a good idea" [04:56] I didn't realise it did that too [04:56] OR Branch.url = u$STRING) is my favourite useless bit i think [04:56] Why? [04:57] branch.url is NULL for !mirror branches [04:57] Oh, so it is. [04:57] and noone case about mirror branches any more [04:57] Not unique_name, right [04:57] Yeah [04:57] Product.name LIKE u$STRING is pretty special yes [04:57] The theory behind it seems to be "find ALL the things" [04:58] And then leave the user to page through the 9s batches. [04:58] itym "try to find all the things, but timeout instead' [04:58] Apparently, find_packages() looks for __init__.py in directories. I think it lies. :-( [04:58] It works sometimes! [04:58] mwhudson: Part of the problem is bugtasks. [04:58] mwhudson: Because there's no logical context to search in. [04:58] oh right [04:58] well [04:59] So you have to search everywhere, probably. [04:59] you could look in the context for all the extant bugtasks i guess? [04:59] Yeah [04:59] but that gets a bit complicated [04:59] We introduced that concept with disclosure. [04:59] It's possible. [04:59] cool [04:59] Then I'd do name LIKE '%foo%' or unique_name = 'foo', I think... [04:59] seriously though unique_name like $string would be as good as this [04:59] Not sure it needs much more than that. [04:59] StevenK: it does, but it starts with listdir() [04:59] StevenK: did you apply my patch ? [04:59] unique_name LIKE '%foo%' is a bit bad, as it matches the product name. [05:00] But maybe. [05:00] (of course, unique_name is only 4 years old or whatever, so much newer than this code) [05:00] lifeless: I was out for lunch with a friend, so if it was a patch to setup.py, I'm sorry but I missed it [05:00] wgrant: not saying mine is good, just better than existing [05:00] 15:18 < lifeless> http://paste.ubuntu.com/998418/ [05:00] mwhudson: With 9.1 we can even use trigram indices to make LIKE '%foo%' very fast. [05:00] (its now 17:00 and EOD) [05:00] bug 372591 [05:00] <_mup_> Bug #372591: Branch vocabulary and search needs thought and cleaning up

< https://launchpad.net/bugs/372591 > [05:00] mwhudson: That title has a few too many euphemisms. [05:01] it's a dup of https://bugs.launchpad.net/launchpad/+bug/793830 anyway [05:01] <_mup_> Bug #793830: Branch:+register-merge time out due to substring matching many tables < https://launchpad.net/bugs/793830 > [05:01] Yeah [05:01] That's the one that wallyworld was hitting a lot last week [05:02] lifeless: Right, I did miss it -- I scanned backlog until I hit wgrant begging for a review and stopped. [05:02] Oh, the prereq or target branch you specified doesn't match a unique_name exactly? Let me just do a 10s search for you. [05:02] heh [05:02] that makes the pain less avoidable i guess [05:03] (my oops was from the suggestions in the 'link a branch' overlay) [05:09] lifeless: error: can't copy 'uditor.egg-info/PKG-INFO': doesn't exist or not a regular file [05:09] EWTF [05:09] mwhudson: I don't know about you, but my distribution of searches is roughly this: 95% of the time it's one of the last 5 branches I touched; 4% I want to search by substring of name in the current project; 1% I want to search by full unique_name (optionally including lp:) [05:12] wgrant: certainly "last branch i touched" is a good guess a lot of the time [05:13] and in this case it was a mixture of the first two points [05:13] Yeah, but ranking like that is hard. [05:13] So we tend to try to avoid combining them like that. [05:14] solr-as-a-service! [05:14] * mwhudson goes back to real work [05:15] mwhudson: We know where you live! [05:38] * StevenK stabs PlainPackageCopyJobTests.createCopyJob() for failing [06:08] mwhudson: patches solicited [06:08] mwhudson: if you get tired of doing easy things [06:19] wgrant: Should I need to do anything after running acceptFromQueue() on the PU and then re-running the PCJ? [06:20] wgrant: Since IArchive.getPublishedSources(name=) is returning [] :-( [06:23] StevenK: Sounds like the PCJ is not running. [06:29] Oh, it does run, it just raises CannotCopy [06:29] Which it doesn't tell me [06:30] unique-from-test-packagecopyjob-py-line361-100000 1.0-1 in breezy-autotest (The signer of this package has no upload rights to this distribution's primary archive. Did you mean to upload to a PPA? [06:30] Right [06:52] StevenK: how are you going ? === jam1 is now known as jam [07:13] lifeless: I gave up in disgust [07:14] Trying to get my SPPH linked to PUs working with PCJs [07:18] why are you doing that? [07:19] bigjools: Because we'd like to know who accepted something, and that information is directly realted to the PackageUpload, and PCJs create PackageUploads if needed. [07:20] sounds like a hack [07:20] How? [07:20] did you consider any other solutions? [07:21] Like adding yet another column onto SPPH that shows who accepted it? Now that is a hack. [07:21] like an audit trail table [07:22] lifeless nacked it [07:22] Which is what auditor is for [07:22] linking to PU is horrible [07:22] But the acceptance is for the *PackageUpload* there is no publication at that point [07:22] not all SPPHes result from a PU [07:22] Of course [07:22] so, yet another nullable FK [07:23] Like sponsor? :-P [07:23] yes :/ [07:23] Nullable FKs are only a problem in theoretical relational algebra. [07:23] I'm just interested in how you arrived at this solution [07:24] if it's the least horrible then fair enough [07:24] bigjools: It was the solution that made the most sense -- with auditor being used and keeping track of a PU as it moves through the queue, we needed a way to easily look up the PU for an SPPH [07:25] I think it's least bad for now. [07:25] the whole model is a mess :/ [07:25] Eventually we should replace PU with something that tracks each operation on the archive. [07:25] Copies and uploads. [07:25] And overrides. [07:25] And deletions. [07:25] like an audit table ... [07:25] Audit tables have been vetoed. [07:26] Audit services have not. [07:26] So StevenK is writing an audit service. [07:26] what SPPH should have been [07:26] SPPH only tracks sources. [07:26] not audit audit ... just a history [07:26] That'll work OK if we can convince Ubuntu to turn into Gentoo. [07:26] But I think cjwatson might not approve. [07:26] lol [07:27] the lack of any recording on who did what action was a terrible omission from the start [07:27] but what is done is done [07:28] As long as we learn from our mistakes. [07:28] Which we haven't classically done, but we're hopefully getting better. [07:28] we mostly do [07:29] The audit service will hopefully mean we can do it easily. [07:29] For everything. [07:29] For every action. [07:30] well I see auditing as two things [07:30] hysterical data for later analysis [07:31] and stuff that affects immediate decisions [07:31] I can't see how a service helps with the latter [07:41] wgrant: bigjools: is there a link from bug(task) -> packageupload for fixed bugs? [07:41] bigjools: the service lets you query the actions on an object, so it should handle the latter nicely. [07:42] wgrant: https://code.launchpad.net/~stevenk/launchpad/populate-spph-pu/+merge/105601 now knows about PCJs. You can review it now if you wish, or tomorrow when you're OCR. :-P [07:42] lifeless: Fixing bugs does not use the PU, but the changesfile [07:42] lifeless: There isn't. You'd have to look for the comments [07:42] StevenK: so if ev wants to get from bug -> BPPH, he goes bug -> changesfile -> SPPH -> BPPH or some insanity ? [07:42] StevenK: I'll do it tomorrow [07:43] lifeless: the upload code scans the changelog [07:43] bigjools: yah, is the other way round we want [07:43] I know :) just telling you how shit it is right now [07:43] bigjools: so, schema work needed? Thats ok, just want to be sure. [07:43] it doesn't store the refs [07:43] just pokes the bugs [07:43] lifeless: Absolutely schema work needed [07:44] lifeless: Anyway, you want the SPPH, not the BPPH [07:44] StevenK: no, we want the BPPH [07:44] lifeless: bear in mind that ultimately it'd be really nice to serviceify soyuz [07:44] users don't download sources to fix bugs [07:44] lifeless: The bug is closed when the *source* is uploaded, not the binaries [07:44] bigjools: orly :P :P :P [07:44] StevenK: I know [07:44] lifeless: bugs are not filed on binaries [07:44] I know that too [07:45] expand your minds [07:45] I've been dealing with buildout and setuptools today. Maybe my mind will uncurl from the fetal position tomorrow morning. [07:45] linking at schema level adds more hurdles to serviceifying [07:46] bigjools: it changes the work needed, thats true. [07:46] lifeless: what is it that you're trying to achieve? [07:47] coz you're breaking your own rule here :) [07:49] otp right now, happy to expand later [07:49] what rule am I breaking ? [07:50] lifeless: report the problem not a proposed solution [07:50] no prob anyway, I am about to scoot off for dinner [07:51] The problem is how to map from bugs (apport ones specifically) to the binary packages that fix problems for users [07:51] ev and I are discussing how to do that, and trying to identify big stroke work needed to do that [07:53] Putting that in LP seems like entirely the wrong thing to do. [07:53] Isn't that a daisy thing? [07:55] lifeless: This info is in the bug comments [07:55] yah, we established that [07:55] sorry, really otp [07:55] so where's the problem? [07:56] Like [07:56] parsing bug comments in a service [07:56] ah "parsing", you didn't say that ;) [07:56] Why not have an external service do the same thing. [07:56] we're discussing that as an option [07:56] Keep a list of (bug#, package, version) [07:56] good morning [07:56] Done [07:56] Problem solved [07:56] No LP crap :) [07:56] wgrant: needs to cross reference tasks === adeuring changed the topic of #launchpad-dev to: http://dev.launchpad.net/ | On call reviewer: - | Firefighting: - | Critical bugs: 3.47*10^2 [07:56] wgrant: so no, its not that simple. [07:56] lifeless: package is there [07:56]