[12:13] sabdfl: Hm...I know we spent a while discussing this, so it might seem like a crazy idea, but I almost wonder if it's cheaper (less time, less cost) to simply forget about merging assignments into one table, leave them as is, and do bare SQL to speed up the bug listing when needed. That's really the one and only problematic screen by having them in two separate tables. [12:13] there is another big advantage to the consolidation [12:13] we can potentially allow people to refer directly to the id of the BugAssignment table [12:13] upload package foo1.1.1-1 fixes assignment#3134532 [12:13] sabdfl: we can do that now [12:13] no we can't [12:13] we'd just combine bug id + ass id [12:13] yuck [12:13] 1-12 [12:13] 1-13 [12:13] no? [12:13] 3536563-345365633632 [12:13] ? [12:13] heh [12:13] will the second number refer to a packageassignment or productassignment? [12:13] what if the numbers don't match? [12:13] sabdfl: er, yeah, hmph [12:13] i.e. [12:14] ok [12:14] ok, i'll keep paring down this pg_dump output [12:14] so there is the idea of the "bug number" and the "work number" [12:14] yeah [12:14] it's not ideal, because people *will* get confused between them [12:14] "but i closed bug #23234234" [12:14] "no you didn't you closed the *assignment* #..." [12:15] but malone has that extra layer, not much we can do about it [12:15] *potentially* we could make the assignment #'s much more visible, so that they are the only numbers people think of [12:15] yeah. but once we have the cross-distro patching, they'll love us. [12:15] yeah [12:15] a number then correclates to a specific piece of work for a specific bug [12:16] and if you really want to refer to a bug, you can give it a name and use that [12:17] i have a working patch for products-without-projects [12:17] daf: good work on the page-template-title test idea [12:17] sabdfl: thanks! [12:17] good thnking: spot the systemic problem, add a verbose check for it, initially in bleat mode [12:18] thank YOU! [12:18] :) [12:21] carlos: i'm seeing the POExportTestCase failure here === mdz [~mdz@69-167-148-207.vnnyca.adelphia.net] has joined #launchpad [12:51] sabdfl: as I said, ./test_on_merge.py canonical pass [12:51] there should be something wrong with make check === spiv [~andrew@adsl-66-203.swiftdsl.com.au] has joined #launchpad [01:10] Merge to rocketfuel@canonical.com/launchpad--devel--0: add SQL patch to merge *BugAssignment tables into one BugAssignment table (patch-924) [01:10] g'night === elmo_away is now known as elmo === stub [~stub@dsl-246.248.240.220.dsl.comindico.com.au] has joined #launchpad [01:48] spiv: ping [01:48] spiv: here is better [01:48] :-D === daf workraves [01:55] carlos: pong [01:55] spiv: we are having problems with initzopeless and cached values [01:55] we have a daemon script that runs outside launchpad [01:56] that process a queue of po files to import [01:56] and when we add new files from launchpad to the queue, the daemon does not see them anymore [01:56] /s/anymore/ever/ [01:56] SteveA suggested to use: sqlos.connection.connCache.clear() [01:56] after the commit [01:57] but it does not changes anything [01:57] our importer calls calls initZopeless when it's instantiated [01:57] and commits transactions as it goes [01:57] carlos: correct? [01:57] daf: yes [01:58] and the problem is fixed if it calls initZopeless for each transaction [01:58] but of course, that's not really a good solution (we don't think) [01:58] daf: No, I don't think so either :) [01:58] am I right in thinking that initZopeless returns something called a transaction manager? [01:59] carlos: That specification of how to reproduce sounds remarkably close to something that could be turned into a test case... ;) [01:59] daf: I think so, and we use it for the commits [01:59] daf: Yep. [01:59] spiv: excuse me? [02:00] carlos: What if you do a rollback immediately after a commit? [02:00] carlos: spiv wants you to write a test case for him :) [02:00] (As a dodgy way to try resync the connection to the database) [02:00] spiv: the connection is working [02:00] spiv: the postgres log shows new transactions [02:01] the start and end with the commit [02:01] and I get data instead of an error [02:01] are these transactions subtransactions? [02:01] Oh, I misread the description. [02:01] daf: don't think so [02:01] ok [02:01] daf: all them have a start and an end [02:01] before the other starts [02:02] sure I could write a test case [02:02] spiv: where should It be stored? [02:02] carlos: The test case? [02:02] yes [02:03] If it depends on rosetta bits, initially I'd put it in rosetta, and aim to reduce it to something more minimal and move it to canonical.database later (when we know exactly what's going on). [02:03] (And obviously give it a comment saying so) [02:03] ok === carlos opens a bug report [02:04] spiv: any suggestion to fix it? :-D [02:04] in theory, the same problem will happen with any table, not just Rosetta-specific ones [02:05] One thing I don't understand is that the Postgres log shows some queries [02:05] so it's not caching everything [02:07] weird [02:07] some queries, but not all? [02:09] carlos: Not yet. :/ === kiko_bz is now known as kiko [02:10] carlos: I think you are seeing transaction isolation in action. [02:10] Ok, there's two possibilities: [02:10] 1. sqlobject is caching objects [02:10] 2. the transaction is isolated, as stub just said. [02:10] carlos: If you close and reopen your connecction, you should see the changes launchpad has made. [02:10] I've been assuming 2 was likely, so I think we've been talking a cross-purposes slightly. [02:11] carlos: If you want to avoid this (there is no real overhead doing it, as psycopg caches connections I believe), you can explicitly set the isolation level. [02:11] Don't know if it's related... [02:11] 2004-12-02 01:45:23 [19630] LOG: statement: BEGIN; SET TRANSACTION ISOLATION LEVEL SERIALIZABLE [02:12] carlos: just to confirm, you've seen launchpad commit its transaction? [02:12] spiv: yes [02:12] carlos: hmm, that looks suspicious to me :) [02:12] That is psycopg starting the transaction and setting the default isolation level, so it won't see any changes made to postgres until it is committed or rolledback [02:12] and launchpad sees the changes from the script without restarting [02:13] stub: that's ok [02:13] but it does not sees the changes after the commit [02:13] stub: oh, so it shouldn't be causing this problem, then [02:13] So, hypothesis: [02:14] statement: END [02:14] it will if I open a psycopg connection, I won't see database changes until *my connection* has been committed or rolled back. [02:14] Actually, hmm. [02:14] stub: I know transactions are committed/rolled back -- I didn't know that connections are [02:15] connections get implicit transactions, as per the DB-API [02:16] oh, I see [02:16] but we cache connections, don't we? [02:16] (The implcit transaction is probably done when you execute the first bit of SQL on it, or when you create a curor. It isn't defined) [02:17] so if you reuse an old connection, it might be in the middle of an old transaction that is hiding new data === spiv hates the implicit-transactions aspect of DB-API [02:17] daf: psycopg caches connections, but handles this stuff for you so you don't have to worry. If you have higher level caches, you end up with trouble (which is why we nuke the SQLOS connection cache every request) === spiv sometimes even finds himself wishing he could use MS's ADODB API! [02:18] spiv: gosh -- it's that bad? :) [02:19] from the logs, seems like all queries are executed into postgres, so it should get fresh data... [02:19] spiv: nobody has ever successfully provided a rational as to why it is bad to the DB-SIG, and it won't change unless somebody bothers to. I know I have no problem with it and think it is fine. [02:20] stub: No-one has ever explained to me why it's good, either ;) [02:21] spiv: Because requiring con.begin_transaction() would be a dead chicken [02:21] implicitly-created transactions are convenient [02:21] spiv: And not all database backends support the notion [02:21] Unless you didn't want to run the statement in a transaction... [02:22] spiv: Which is strongly discouraged by everyone except MySQL users [02:23] Heh. At the time I was an MS SQL Server user :P [02:23] (Well.. actually it causes me some grief since I sometimes run wierd postgresql commands like 'create database' that can't be executed inside a transaction....) [02:23] New Malone bug #113: "Write a test case for the cache problem with initzopeless", submitted by Carlos Perell Marn [02:23] https://dogfood.ubuntu.com/malone/bugs/113 [02:25] Kiko: we were in the middle of a conversation when my ADSL dropped out. [02:25] stub, oh. I thought you reckoned it wasn't a pleasant and just pulled the plug! [02:25] kiko: You were bitchi^h^h^h^h^h^hdiscussing plpythonu usage [02:25] I assume too much, it appears. [02:26] well. [02:26] here's the issue [02:26] daf: malone does not likes you, the person search does not see you... [02:26] is the plpythonu cr^hode going as far as requiring zope and launchpad? [02:26] if so, I need to budget some hours for server work [02:27] stub: so, do you think that creating a new connection with initZopeLess is a good solution? [02:27] No. Anything I import will be dead simple, only requiring stuff like re [02:27] stub, it would be nice if you could skip True/False for the time being, then. or give us a workaround [02:28] I know that sounds horrible. [02:28] carlos: Sure. Connections are cheap for us. [02:28] spiv: test case filed in malone, will do it tomorrow [02:28] ok [02:28] daf: do you agree? [02:28] carlos: Thank you :) [02:28] could I commit that change? [02:28] but level with me -- I run a debian woody server. we have only one server with disks. 15ish people use it every day. I don't want to break people's work. [02:28] carlos: agree? what? [02:28] kiko: No problem. I assume you need python2.1 compatibility for any plpythonu code until further notice? [02:28] right. [02:29] daf: to use your workaround calling initZopeLess with every transaction [02:29] I can upgrade the server, but it's an effort that if avoided makes everybody happy [02:29] if you can do that, stub, well, I'm adding to that eternal debt i'm running with you [02:30] now [02:30] stub, there's yet *another* matter [02:30] it is now impossible to initialize a postgresql server remotely [02:31] it used to be possible using PG_HOST or -h [02:31] but now because you require some sql-fu for the fulltext indexing, I believe, it's no worky [02:31] carlos: It is because the system searches on 'words', and daf's name is dafydd in the database. We would need customize our stemmer to make the full text stuff match 'dafydd' if you query for 'daf'. [02:31] but it worked some days ago [02:31] but who would want to do that?! [02:32] and dafydd is not working [02:32] in the popup window [02:32] kiko: ok. That was me being lazy - I had assumed developers were running with their databases on localhost. [02:32] the form accepts it directly [02:32] but the search popup fails to search "daf" or "dafydd" [02:32] stub, I realize; we have an "office" setup here, which is ironically unusual here ;) [02:32] carlos: oh - the form also does a substring search on person.name, so that should work... [02:33] Kiko: I used to work in an office where we developed on our laptops with not a server in sight ;) [02:33] that's not real work! [02:33] you need a coffee machine and a server room [02:34] kiko: We had the coffee machine, and lack of server let us justify kick arse laptops :-) [02:34] carlos: oh, sure [02:34] daf: thanks ;-) [02:35] stub, I really appreciate any help you can provide to make my life easier. [02:35] maintaining this whole crapola is a serious effort! [02:40] good night [02:40] Malone bug #77 fixed for product The Rosetta Translation Portal: Implement a potemplate form handler [02:40] https://dogfood.ubuntu.com/malone/bugs/77 [02:49] Merge to rocketfuel@canonical.com/launchpad--devel--0: Fixed the cache problem with the import daemon as daf and stub suggested (patch-925) [02:53] time to run off [02:54] Merge to rocketfuel@canonical.com/launchpad--devel--0: Python 2.1 compatibility in plpythonu for Team Brazil's antique system (patch-926) === spiv [~andrew@adsl-66-203.swiftdsl.com.au] has joined #launchpad === ChanServ [ChanServ@services.] has joined #launchpad === warthylog [~warthylog@port49.ds1-van.adsl.cybercity.dk] has joined #launchpad === Topic for #launchpad: lunchpad: home of the sandwich artists | fogo na bomba | "qorking along happily, with SteveA egging us on" === Topic (#launchpad): set by sabdfl at Tue Oct 19 11:05:38 2004 === lifeless_ is now known as lifeless === ddaa [~ddaa@host217-37-231-28.in-addr.btopenworld.com] has joined #launchpad [10:17] Merge to thelove@canonical.com/bazaar--devo--1.1: relax source defaults and add tags and cscope to precious - closing #3112 (patch-7) [10:17] bob2: lifeless: so, what is going to be the name the new project to put all the little products in? [10:18] spiv: ping === lulu [~lu@host217-37-231-28.in-addr.btopenworld.com] has joined #launchpad === Kinnison [~dsilvers@host81-153-126-219.range81-153.btcentralplus.com] has joined #launchpad [10:34] Morning [10:41] stub: pong [10:42] psycopg.ProgrammingError: ERROR: could not find tsearch config by locale [10:42] :-/ [10:43] daf: aha, same problem as Kinnison had [10:43] did Kinnison manage to fix it? [10:43] you're obviously using a deviant locale on your machine... maybe CY [10:43] I tried changing that === Kinnison did === Kinnison was magical [10:43] stub and I fixed it [10:43] Merge to thelove@canonical.com/bazaar--devo--1.1: Removing extraneous file-diffs command in help (patch-8) [10:43] but it did involve reinitialising the database from scratch [10:43] perhaps I need to explicitly set the locale to C in the configuration file [10:44] oh [10:44] daf: It's not that easy :-) [10:44] grumble [10:44] wow -- I'm not wearing my glasses and I read that as "ritualising the database" [10:44] dead chickens anyone? [10:44] daf: Do you have data you can't bear to lose or can you reinit your entire pg database? === Kinnison wraps the rubber chicken around stevea's head [10:45] actually, no, resetting the DB is ok right now [10:45] okay, is stub around, or shall I walk you through i? [10:45] s/i.$/it?/ [10:45] Since I need to do it on my laptop too [10:45] dude, don't rubber chicken me. I am "the management". [10:46] SteveA: Not even 'da management' can save you from the rubber chicken === SteveA holds the chicken in the air and waits for the deckchair to arrive === Kinnison shoves it up stevea's nose [10:47] better stop there. a bit cold to bury all my clothes. [10:47] Heh [10:48] stub: ping? === SteveA is now known as Keith [10:48] hi, I'm keith === Keith is now known as SteveA [10:48] stop pretending [10:49] spiv: tell me about how the shipit demo is going [10:56] SteveA: I sent a mail to the admins a while ago about getting apache to point to it, otherwise it should be ready. [10:57] morning all [10:57] stub: have a patch for ya [10:57] Kinnison: what exactly did you do to fix your DB? [10:58] Yo [10:58] daf: can you wait until I've finished this upgrade? Install postgresql-contrib for now if you haven't already and set up the search path as per stub's mail [10:59] Kinnison: done that [10:59] anybody else seeing a test failure on POExportTestCase? [10:59] daf: right. given that stub appears to not be around, here goes [10:59] daf: stop postgresql [11:00] ok [11:00] spiv: What do I pass to the authservers createUser method as the loginID? [11:00] daf: cd into /var/lib/postgres [11:00] daf: mv data data.before-kinni-fucked-it-up [11:00] spiv: can you cc me into the mail to admins next time? that allows me to see that things have progressed, but are waiting for the admins to act, without needing to remember to ask you about it. === daf senses magic [11:00] SteveA: Oops, yes, good idea. [11:01] Kinnison: ok [11:01] sudo -u postgres initdb --locale=C --debian-conffile --encoding=UNICODE [11:01] hrm: "sudo: initdb: command not found" [11:02] sudo su - postgres [11:02] stub: an email address. I should rename that parameter I guess... [11:02] initdb --l...... [11:02] You must identify the directory where the data for this database system [11:02] will reside. Do this with either the invocation option -D or the [11:02] environment variable PGDATA. [11:03] add -D /var/lib/postgres/data [11:03] to the cmdline [11:03] right === Kinnison is about to try [11:03] /usr/lib/postgresql/bin/initdb: line 648: /etc/postgresql/6328: Permission denied [11:03] :_/ [11:03] s/_/-/ [11:04] erm, blergh [11:04] hold on, my upgrade has almost finished then I'll try === daf can wait :) [11:06] daf: Your -D option didn't take effect - it is trying to create the database in /etc/postgresql [11:06] oh [11:06] curious [11:06] it's having some effect, because it's no longer complaining about the missing option [11:06] Oh... hmm.. [11:08] Do it again without the --debian-conffile option [11:08] then cd into /var/lib/postgres/data [11:09] ln -sf /etc/postgresql/postgresql.conf [11:09] then back as root, run /etc/init.d/postgresql start [11:09] and then try a 'make' in database/schema [11:11] daf: Oh yeah, su back to postgres and create yourself first [11:11] yeah, just tripped over that :) [11:12] :) === Kinnison wonders if 'make check' on batteries is such a good plan === Kinnison hunts around for a power socket [11:12] ok, make -C database/schema: [11:12] psycopg.ProgrammingError: ERROR: relation "pg_ts_cfg" does not exist [11:16] Did you make the symlink as I said? Did you update postgresql.conf as per stub's mail like I asked? [11:16] the symlink is there [11:16] I modified postgresql.conf when I installed postgresql-contrib [11:17] I just didn't realise at the time that I'd have to totally nuke all my postgres data [11:17] you wouldn't have if your system locale when postgres was installed had been 'C' [11:18] I didn't know that it would be a problem when I installed it last year :-/ [11:18] I'd assumed all the Ubuntu and Debian installs would have had the C locale === Kinnison grins [11:18] stub: poor assumption :-) [11:19] perhaps the priority of that debconf question is lower now [11:19] daf: It seems to come up regularly on the mailing lists [11:19] daf: psql -d launchpad_dev -c show\ search_path [11:19] search_path [11:19] -------------- [11:19] $user,public [11:20] The search path hasn't been updated [11:20] but my postgresql.conf says [11:20] search_path = '$user,public,ts2' [11:20] and /var/lib/postgres/data/postgresql.conf is a symlink to /etc/postgresql/postgresql.conf ? [11:20] yes [11:21] restart postgres once more [11:21] done, search path is still the same === Kinnison boggles === Kinnison did exactly as we just went through and it worked for him [11:22] Add 'this is a syntax error' to your postgresql.conf [11:22] And bounce [11:22] (The postgresql server, not you) [11:22] oop! [11:22] s! [11:24] daf: ? [11:25] symlinked into /var/lib/postgres rather than /var/lib/postgres/data [11:25] things seem fine now [11:25] So simple even a programmer can do it === Kinnison grins stub [11:26] stub: See, I even remembered the process :0) [11:26] thanks! [11:30] so nobody else is seeing the test failure on POExportTestCase [11:31] sabdfl: I'm just updating to check === Kinnison waits for baz to merge 69 new patchlogs [11:34] Merge to thelove@canonical.com/bazaar--devo--1.1: use the current project tree archive by default in archive-mirror fixes bug #4115 (patch-9) === kiko [~kiko@200-206-134-238.async.com.br] has joined #launchpad [11:40] stub, you are a riot [11:41] as fast as baz can be, goddamn it takes ages to star-merge a week's changes [11:41] want a t-shirt? === stub got a t-shirt last time === stub likes his t-shirt [11:44] the new t-shirt should say MAKE BAZ FAST [11:44] but I refrained from political issues this time around [11:44] to be fair on baz, it is being squeeeeeeezed through a tiny tiny pipe right now thanks to Kamion chewwing all his available bandwidth [11:44] can't he ease up on the pr0n just for your star-merge? [11:45] What do you think he is star merging? [11:47] Preparing the artwork for the HumanNG theme ;) [11:49] Merge to rocketfuel@canonical.com/launchpad--devel--0: user creation and authentication tests (patch-928) [11:51] um, i think you mean "testing the artwork for the HumanNG theme" [11:53] how would one test that artwork 8) [11:54] Kinnison: try star-merge -t for three-way merges, MUCH faster on my machine [11:54] sabdfl: -t ? [11:54] three way merge [11:54] I was doing a fairly clean merge anyway I think, but I'll bear that in mind [11:54] sabdfl: what was taking the time was downloading 69 changesets over a choked link [11:55] ah [11:55] It only took ca. 30 seconds to star-merge once it had the patches [11:58] Merge to thelove@canonical.com/bazaar--devo--1.1: Quieter deltas (patch-10) [11:58] Librarian meeting time? [11:58] it is almost time for us to talk about the librarian [11:59] and I am here [11:59] but first, SteveA, debonzi and I have a question [11:59] can I steal a few minutes of your time === Kinnison goes to make coffee first then [11:59] say yes [12:00] okay [12:00] Merge to thelove@canonical.com/bazaar--devo--1.1: do not allow newlines in archive locations passed to make-archive and register-archive (bug #3590) (patch-11) [12:01] hrmph [12:01] why do I see: [12:01] /home/daf/src/canonical/dists/launchpad/lib/canonical/database/sqlbase.py:90: UserWarning: Something tried to set a _connection. Ignored. [12:01] ? [12:01] (this is when using scripts which use initZopeless) [12:01] who changed the layout of the project / product search page? [12:02] daf: maybe you used initZopeless twice [12:02] spiv: ? [12:02] i've seen that too [12:02] SteveA: don't think so [12:02] daf: can you reproduce it? [12:02] it seems to happen every time I run such a script [12:02] that's great. email me about it and I'll find out what is going on. [12:03] kiko: what's the question? [12:03] SteveA: what additional information can I give you? [12:03] tell me exactly how to do what you did [12:03] SteveA, the question is in what cases are classmethods an acceptable way to obtain a collection of objects? [12:03] daf: It's a warning because I wasn't sure if it would happen, and if it did, I wasn't sure if it would necessarily be a real problem or not. Probably making it a warning is too conservative, because it looks like it happens harmlessly inside SQLObject. [12:04] SteveA, I use the rule of thumb, if there's no container, a classmethod on the class itself is acceptable [12:04] I might be able to detect a harmless occurance, actually, in which case I could suppress the spurious warnings. Hmm... [12:04] spiv: ah [12:04] "LP_DBNAME=launchpad_dev python lib/canonical/rosetta/scripts/show-projects.py" is enough to replicate it here [12:04] kiko: class methods are a pain in the arse for security. I'd rather avoid them where possible. [12:05] SteveA, fair enough. How do I get a collection of sourcepackages? [12:05] Yeah, virtually any script with initZopeless is. I figured it was better to have the warning than not, at least initially :) [12:05] they make for confusing code, because you can get to them from an instance or from the class. [12:05] kiko: write a SourcePackageSet class, I suggest. [12:06] Kinnison: please say when you have your coffee [12:06] and how do I get an instance of SourcePackageSet? [12:06] SteveA: the machine is gurgling enthusiastically [12:06] SteveA, if you can answer that then I think we are gold [12:07] kiko: getUtility(ISourcePackageSet). register it as a utility. [12:08] or, from a script, directly import it and use it, seeing as scripts don't register utilities yet [12:08] use it how? sps = SourcePackageSet() [12:08] sps.getPackages("XXX") [12:08] this way, collections of database things are a lot like other database things [12:08] or sps = SourcePackageSet("XXX") [12:08] we have the implementation class, and the interface [12:08] sps.getPackages() [12:09] sps = getUtility(ISourcePackageSet) [12:09] sps.getPackage() [12:09] sps.getPackagesMatching('foo') [12:09] or whatever you need [12:09] SteveA: I now have coffee as black as night, as strong as a hero and as kick-arse-wakeup-goddamnit as a bunch of over-excited geeks at a free-hardware convention [12:10] so, you're ready to talk librarian. [12:10] let's talk about the librarian. [12:10] yesterday, we talked about thel librarian-wrapper thing [12:11] for making a bunch of files appear as a package layout for serving by apache [12:11] it is not clear to everyone why such a wrapper is needed [12:11] also, we talked about how the librarian is to be deployed [12:11] you've conflated two concepts right there [12:11] The wrapper is nothing to do with laying out packages [12:11] :-) [12:11] whether it serves the outside-world directly, or whether it is served via apache [12:12] ok, that's my inaccurate introduction done ;-) [12:12] go ahead, Kinnison [12:12] Okay, so cprov has been working on something called the librarian-wrapper which is to be used by lucille [12:12] It's purpose is to provide a way to optimise disk-usage and network consumption for archives [12:12] Basically it's a local-cache of files uploaded to the librarian. Which can be blown away if needed [12:13] The idea is that it should speed up publishing and provide us with a good basis for using hardlinks to optimise disk consumption when we have piles of derivative distributions [12:13] Why isn't it called librarian-cache then? [12:13] Because the cache is only one bit of its functionality as far as lucille is concerned [12:14] I don't understand "local-cache". To understand this, I need to understand how this stuff is deployed. [12:14] I think that particular explanation will be best performed in a room with everyone present and a whiteboard [12:14] Merge to thelove@canonical.com/bazaar--devo--1.1: Quieter deltas (patch-12) [12:14] it's very hard to draw diagrams and arrows on IRC [12:15]