[12:01] SteveA: what on earth do you want me to do about this? [12:01] sabdfl: send me an email if you want ... I'll be away for 40 min .. nicole is running on zhongshan [12:01] cprov: so nicole is creating productreleases? cool [12:01] i'll keep going with my code, then we'll merge later [12:01] I'm not lifeless and I don't have the time, energy or skills to debug it - I can kill it and move your merge request out of the way if you want? [12:01] elmo: no idea. can you kill 11951 or 11878 ? [12:02] sure, I can kill 11951 [12:02] Process 11951 attached - interrupt to quit [12:02] write(1, "ALTER TABLE\n", 12 [12:02] fyi, that's the strace [12:03] this kind of thing happens when some other process is accessing that database [12:03] I wish there was a way of telling postgres to fail rather than block in these situations [12:03] killed [12:04] can you kill 11877 too? [12:05] now make can't notice it's children dieing? :P [12:07] oh, for christ's sake, every child up is hanging on writes to 1 [12:08] thanks for playing the grim reaper [12:08] I need to leave the office now. I'll talk to people about this tomorrow, try to get something sane sorted out. === kiko is now known as kiko-afk === Kinnison_ [~dsilvers@haddenham.pepperfish.net] has joined #launchpad === Kinnison_ is now known as Kinnison [12:39] Kinnison: turns out that a local test-pserver is actually easy to do [12:39] a 2 line bash script made executable, and nc, and voila [12:41] lifeless: cute [12:49] jblack: did you or someone end up helping limi out ? === BradB is now known as BradB|away [01:01] how can I get my username from python? [01:02] is os.getEnv('USER') reliable enough ? [01:02] getent stuff? [01:02] os.getlogin() ? [01:02] elmo: bingo, thanks [01:03] yay, one automagic pserver passwd file for the test suite [01:09] sabdfl: yes, now nicole is running and creating productreleases. [01:10] great work cprov [01:10] sabdfl: btw, I'm just running through main and restricted packages (1009) rather than universe, I think it's enough for test proposes [01:11] flippin 'eck, it's 00:11 again === Kinnison decides to go to bed so he can actually get up in the morning [01:11] c'y'all tomorrow === kiko-afk is now known as kiko [01:32] more pythno foo - how do I wait say a half second ? [01:38] lifeless: Sorry. Just got home. The neighbor was really sick, so I had to take care of her and her *four* kids all day. [01:38] ouch [01:38] no need to be sorry, I just needed to know if Limi was helped or not, theres no mail showing that he was. [01:38] Not by me he hasn't. Didn't know until now he had a problem. === cprov [~cprov@200.158.100.251] has left #launchpad ["Leaving"] [01:53] why dbschema.PackagePublishingStatus.PROPOSED does not exists anymore? [01:54] should I replace it for something else? [02:00] debonzi: was changed as part of Kinnison's work [02:01] best leave it as it is for the moment, and discuss with him tomorrow [02:02] sabdfl, nice, Ill comment the template to do not break the page for while. Thanks [02:03] ok [02:31] lifeless: how can i see the diff from a previous revision? === kiko is now known as kiko-afk === debonzi goes for dinner === stub [~stub@dsl-246.248.240.220.dsl.comindico.com.au] has joined #launchpad [03:01] sabdfl: in your tree? [03:01] tla changes --diffs patch-34 [03:02] lifeless: thanks [03:02] np [03:04] lifeless: [03:04] changes: illegal revision name: patch-253 [03:04] sabdfl: oh bah. [03:04] use this idiom [03:04] tla changes --diffs $(tla tree-version)--patch-253 [03:05] which is a nonsense, I'll file a bug for that. === kiko-afk is now known as kiko [03:25] anyone seen spiv around? === kiko is now known as kiko-afk === kiko-afk is now known as kiko === kiko_ [~kiko@200-206-134-238.async.com.br] has joined #launchpad === kiko_ is now known as kiko [10:29] morning all [11:14] hey === Kinnison [~dsilvers@host81-153-126-219.range81-153.btcentralplus.com] has joined #launchpad === limi [~limi@212.80-202-72.nextgentel.com] has joined #launchpad === ddaa [~ddaa@nemesis.xlii.org] has joined #launchpad [12:03] ddaa: still got that keyboard huh ? [12:04] Yes, it's cool. There is a hidden way of quitting gaim too... [12:04] Curiously, that shit only happen to me in chat clients. [12:05] lifeless: about buildbot [12:05] I assume, to have 5 successes a day, I need to run several jobs in parallel [12:05] yes [12:05] What's the catch? Use several slaves? [12:05] one slave. [12:06] its threaded. [12:06] Mh. okay... [12:06] It's probably going to complicate debugging, but for the working cases it's okay. [12:07] you could do several slaves if you want, but I'd be wary of spending a bunch of time only to find issues. (such as how to allocate the jobs appropriately) [12:07] Note that my cpu is not too fast (PIII 600MHz) so it's prolly going to be a bottleneck at some point. [12:07] sure. [12:08] go for the low hanging fruit during the day :) [12:08] Yeah. I'd like to postpone the multiple-slave issues as long as I can. [12:08] set things like xserver off when you go to bed :) [12:08] ? [12:08] it would be easier to do parallel buildbots with the same database - dual master + dual slave - for now. [12:09] well small jobs finish quickly [12:09] Good idea. [12:09] I see. I thought you were suggesting to stop X when going to be to release resources :-) === Kinnison drags lifeless over here if he wants to talk about the cscvs stuff [12:47] (unless it's public now) [12:48] lifeless: I knew I was forgetting something at the meeting. [12:49] Kinnison: cscvs is public, always has been. what we are doing with it though.. [12:49] I wanted to poke you about patchlogs which are permanently missing from rocketfuel versions. [12:49] ddaa: arh [12:49] lifeless: I was under the impression the new stuff still wasn't public [12:49] Kinnison: the code isn't yet no. I have the ok to release it, in step with our archives. [12:50] For example the reverted patch from launchpad, or the pre-tag history of soyuz. [12:50] anyhoo, dod you see the bug ? [12:50] ddaa: the pre-tag history of soyuz isn't missing, it was merged in when soyuz was merged in. [12:50] lifeless: Unlikely unless you pointed it to me [12:50] it's annoying because it breaks "missing" and probably "replay". [12:50] ddaa: copy that patch in as a merge, fine by me. [12:51] Okay. So there is no good reason why there are patchlogs permanently missing? [12:51] Kinnison: '/foo/bar' became ':local:foo:/bar' [12:51] ddaa: which ones ? [12:51] lifeless: Eww [12:51] lifeless: wait a min [12:51] lifeless: Oopsie :-P [12:51] :] [12:51] rocketfuel@canonical.com/launchpad--devel--0--patch-422 [12:51] it now uses the cvs protocol for local repositories :) [12:52] ddaa: lifeless> ddaa: copy that patch in as a merge, fine by me. [12:52] rocketfuel@canonical.com/soyuz--devel--0--base-0..patch-29 [12:52] ddaa: you shouldn't be replaying in that branch anyway, ignore that. [12:53] Bad answer. Missing is a useful command and replay is significantly faster than update here, where for some reason it always uses the apply-delta algorithm. [12:54] ddaa: the soyuz branch is not used. [12:54] its a perfectly good answer because of that. [12:54] because missing and replay don't look at that branch. [12:54] It's used by the pqm config. [12:54] ddaa: eh? [12:55] tlash dists> cat configs/canonical.com/pqm/development | head -1 [12:55] ./pqm rocketfuel@canonical.com/soyuz--devel--0 [12:56] ok, thats interesting and all. [12:56] when I next hack on pqm, I'll update the config. [12:56] Okay. [01:16] lifeless: phwoar, big changes on cscvs i see, how's that coming? [01:17] very close to having the default branch support finished [01:17] all the db & parser changes are done, am onto the scanning of tla to recreate what the parser won't tell us. [01:18] I'm hoping to run my first test-on actual data saturday. [01:18] the ProjectProductSetup page is idling though - can you please please please push through the unsconded ones there? [01:21] lifeless: thought we switched to doing this by email? i'm way behind on email [01:22] sabdfl: you were going to finish the current page... otherwise I'm just mailing it to you again. [01:22] are you using the new project/project creation pages, and the links to setup sourcesource there? [01:22] but I can copy the current page and mail to you if you like. [01:22] sabdfl: when they go into launchpad, yes. [01:22] i'll work on it in the wiki [01:23] david is working on the research for this too now, so we'll be putting some volume up I hope. [01:23] would you like me to have him mail you & me rather than adding to the wiki page ? [01:23] hmm.. looks to me like a bunch were added after i thought we switched to email [01:23] sabdfl: nope. [01:23] onest injin [01:24] lifeless: we have a script which will pull details from freshmeat and sourceforge [01:24] celso has been working on that [01:24] sabdfl: cool. I presume we still need to determine the mapping tho. [01:25] What I'd love is to get you off the critical path - so I can go 'hey, xyzlib worked in test, goto launchpad, and just do it' [01:25] yes, what i'm really saying is that there are going to be about 300 projects / products setup automatically [01:25] that will help enormously. [01:25] we are just running some tests this week and next week, then we can put it into production [01:25] the gnome ones helped too, as as they go good we can just make them. [01:27] hmm... project db methinks should be libdb [01:27] berkely database is more than a lib [01:27] the lib is what most folk use is all. [01:28] "db" is confusing, what about berkeleydb [01:28] sure. [01:28] i don't mind the product name [01:28] is libdb1-compat part of the same project? [01:28] IIRC no. [01:29] someone wrote it because they needed it [01:29] I did check that when doing the page. [01:29] if we edit the names in the db, will it break the sync process? [01:29] or does the sync remember what it was told to use when it started? [01:30] once I put the sync gold, the name is set forever, until I go change it [01:30] the project name can vary wildy, it won't care. === limi [~limi@212.80-202-72.nextgentel.com] has joined #launchpad [01:31] spiv: sqlos patch applied [01:31] lifeless: Thanks. [01:31] Hmm, #1922 just struck again. === cprov [~cprov@200.158.100.251] has joined #launchpad [01:31] 3rd time today. [01:31] (after two days nothing) [01:31] and if you change it, does it tag into a new archive/cat-branch-version from the old one? === spiv roboots [01:32] Bug 2005 resolved: Cache prevents to use new users without restarting launchpad [01:32] https://bugzilla.warthogs.hbd.com/bugzilla/show_bug.cgi?id=2005 [01:32] sabdfl: yes, changing involves a manual tag at the moment, and change the sync in launchpad. after that the sync carries on automatically. [01:32] ok [01:32] that is partly by design, partly optimising for the common case. [01:33] the design bit is - we don't want these things randomly changing all over the place. so until we have a well articulated policy, that I can put into code, I don't want the code to have that 'feature'. [01:33] Kinnison: here's an example: [01:33] ':pserver:anonymous@127.0.0.1:/home/robertc/source/canonical/buildbot/launchpad/sourcecode/cscvs/,,repoCatalog' != ':pserver:anonymous@127.0.0.1:/srv/importdtest/buildbot/launchpad/sourcecode/cscvs/,,repoCatalog' [01:34] oh, bah. [01:34] thats just me being brain dead, nm, carry on, etc etc === Kinnison tickles lifeless === limi [~limi@212.80-202-72.nextgentel.com] has joined #launchpad [01:40] Kinnison: this is it (its still happening...) [01:40] print CVS.protocol.CVSRoot('/srv/importdtest/botslave/buildbot-jobs/autoconf-HEAD-import.job/autoconf@arch.ubuntu.com/autoconf--MAIN--0/cvs_temp_repo') [01:40] :local:/srv/importdtest/botslave/buildbot-jobs/autoconf-HEAD-import.job/autoconf@arch.ubuntu.com:/autoconf--MAIN--0/cvs_temp_repo [01:40] I haven't quite fixed it :[ [01:40] lifeless: Aah it's not liking the @ inside the path [01:40] the earlier bug was allowing / in the hostname [01:40] lifeless: Dunno how you'll fix that except to disallow / inside users and passwords in the regexp [01:41] Kinnison: probably break it into ()|() [01:41] possibly [01:41] I'm not sure that the @ is the problem yet [01:42] it smells like it to me [01:42] ok, now I'm sure. [01:42] z.username -> [01:42] /srv/importdtest/botslave/buildbot-jobs/autoconf-HEAD-import.job/autoconf [01:42] yeppers [01:44] thankfully, your test suite is comprehensive enough to catch me being an idiot regression wise :) === Kinnison grins [01:45] username can't have / can it ? [01:45] I sincerely hope not [01:45] that fixes it trivially. [01:46] ^:([^:] +):(?:(?:([^:/] +)(?::([^:/] +))?@)?([^:/] +)(?::(\\d+))?:?)?(/.*)$ [01:46] method username password hostname port path [01:46] yep [01:47] possibly wrong to put the / in the password bit, but hey, cross that bridge when the bug report lands [01:47] aye [01:47] this should finally allow me to not get booted off the kde anonymous server. [01:47] so we can do kdelibs as soon as spiv makes me proud [01:48] hurrah [01:48] (fingers crossed etc etc etc) [01:48] lifeless: :) [01:48] have I mentioned how great it is to have this native facility ? === Kinnison grins. You're welcome [01:50] you might find looking at the current protocol / Protocol / pipes interesting [01:51] you had a design flaw - using a list of handles isn't sufficient to do cleanups properly. [01:52] oh? [01:52] lifeless: ok, project is xiph, product is paranoia, productseries is Paranoia-III [01:52] productseries ? [01:52] yes, name for a set of releases === ddaa gets a tarfile.ReadError and goes to find the bugzilla [01:52] like "head" [01:52] ah. [01:52] or "2.0" [01:52] ok, thats not in production yet. [01:52] or "Paranoia-III" [01:52] lifeless: no, it's last night's work, though the db stuff has been there for a while [01:53] ok [01:53] creating a productrelease will require specifying a series for that release shortly [01:53] this allows us to do better magic when we have intermingled releases of, say: [01:53] apache 1.3 [01:53] no, better example would be: [01:53] gimp1.3 [01:53] gimp2.0 and [01:53] gimp-cvs [01:54] (2.1) [01:54] lifeless: where do the buildbot bugs live? [01:54] create three series, and for each release just tell it which series it belongs to [01:54] ddaa: bugzilla.canonical.com or bugzilla.warthogs.hbd.com - the original main bugzilla [01:54] sabdfl: so a series is roughly a branch :) [01:54] roughly, yes [01:54] I was redirected to bugzilla.no-name-yet.com... [01:54] ddaa: yeah. let me give you a direct url [01:55] https://bugzilla.warthogs.hbd.com/bugzilla/show_bug.cgi?id=2005 [01:55] thats in the right system. [01:55] I think we may need to take out the python tar module, it appears to be slightly crackful. [01:57] thanks [01:59] lifeless: i think keybuk uses that heavily for sourcerer [01:59] sabdfl: he was saying it had really bad problems [01:59] which I filed to look into later [01:59] k [01:59] so, the public name of buildbot is Hoover, right? [02:00] yes [02:00] out install of buildbot sucks. [02:00] s/out/our/ [02:00] :) [02:00] cprov: around? [02:03] Kinnison: got a few minutes ? [02:03] got a change in behaviour from cvs external rlog to the native implementation. [02:04] https://chinstrap.warthogs.hbd.com/importd/devel-status/libbonobo-HEAD-import.job/events/289/log (if you have time) [02:06] sabdfl: yes [02:06] sabdfl: I'm testing the new dump including productreleases [02:06] Just to be sure I grok bugzilla right, there are currently 4 bugs for Hoover whose status is anything but "closed"? [02:06] https://chinstrap.warthogs.hbd.com/importd/devel-status/kdelibs-HEAD-import.job/events/273/log for folk that like to give browsers a hernia [02:06] stub: what's the correct way of doing something like this: [02:06] * SELECT pg_catalog.setval('schema_id_seq', pg_catalog.select(SELECT max(id) FROM Schema"), true); [02:06] ddaa: sounds about right. [02:07] cprov: could you point me at that too please? want to see how it looks [02:07] lifeless: is this a hint that I should enforce mod_deflate? :P [02:07] cprov: do you use the ProductRelease object from canonical.launchpad.database or a custom Soyuz one? [02:07] elmo: wink wink nudge nudge [02:07] sabdfl: Dunno - what are you trying to do? [02:08] elmo: I think that would actually make browsers less happy, not more. [02:08] i was looking to merge your code now if it's still Soyuz* [02:08] Looks like you are trying to reset the primary key sequence to the next highest value? [02:08] stub: trying to be able to add rows of sampledata then reset the serial counter to the max current [02:08] correct [02:08] sabdfl: mailed you a high-priority mapping - kdelibs. [02:08] sabdfl: I'm using custom one, since we aren't use SQLO in those scripts [02:09] cprov: oh, ok [02:09] that's probably fine then [02:09] why not use SQLO? [02:09] DISTINCT etc? [02:09] lifeless: that's just a big ass file, right? [02:09] sane queries etc? [02:09] sabdfl: in zhongshan:/home/cprov/ lp_dump-productreleases-mr.sql [02:09] random failures etc? [02:09] lifeless: buildd.d.o has those too and .bz2's them by default - the browser seems to cope okay [02:09] right, those reasons. [02:10] elmo: dynamic, scrolls as you wait [02:10] sabdfl: Not sure - I've only used alter sequence which won't let you embed a SELECT in it [02:11] stub: no worries, just thought i'd play "throgh crackful queries at the BDBA and see how he handles" today :-) [02:12] sabdfl: yes, many simple feature are harder to do using SQLO, maybe my fault, but using simple "pg" the things shows up faster, same strategy used by elmo and Kinnison [02:12] works for me [02:12] sabdfl: select setval('bug_id_seq', (select max(id)+1 from bug)); [02:13] let's help spiv and bradb by telling them wat tweaks would make sqlo more useful in scripts though [02:13] stub: nice, thanks :-) [02:14] sabdfl: ERROR: column "productseries" does not exist SELECT id FROM ProductRelease WHERE productseries = 466 ? [02:14] elmo: should i be able to scp off zhongshan using my normal ssh keys? [02:14] ok, I'm going to crash, started at 7am :| [02:14] spiv: mail me anything you'd like me to investigate. [02:15] ddaa: good luck, please mail me for info etc. [02:15] cprov: there's an sql patch which went in today for that [02:15] I'm seeing RMS speak tomorrow, so will not be around all that much - will be working offline mostly. [02:15] it was in the pending queue for a day or two [02:15] night all. [02:15] night robert [02:15] lifeless: you have hint about the tarfile error? [02:15] i seem to get that one for everything [02:15] ddaa: oh. [02:15] what ones specifically ? [02:16] if its the X.org ones, its cause that module can't hack the 800MB tarball. [02:16] so far: zenity, yelp, xrestop [02:16] (or, that the URL for the tarball is wrong) [02:16] hmm, dunno offhand. pdb is your friend I think [02:17] lifeless: what i was planning to ho [02:17] run the slave with twistd -nf buildbot.tap [02:17] and throw an import pdb;pdb.set_trace() at the appropriate place. [02:17] ack, I had put that one into a sticky note a few days ago :-0 [02:17] ok, I'm out... [02:17] great. [02:17] night! [02:17] sabdfl: yep, I see, we have now productseries... uhmm it crashes my dump entirely [02:17] sweet dreams [02:18] sabdfl: if you have the .ssh/config set up right, yes [02:18] what's the voodoo for that? [02:18] Host *.ubuntu.com [02:18] ProxyCommand ssh mark@chinstrap.warthogs.hbd.com nc -q0 %h %p [02:18] add that to the other stuff [02:18] for the zhongshan host only? [02:19] do it for *.ubuntu.com, then you get the other ubuntu.com hosts for free as we migrate [02:20] sabdfl: anyway, using that I pointed you, you can see productseries [02:20] can I have separate entries for *.ubuntu.com [02:20] and zhongshan [02:20] ? [02:21] cprov: so your code created productseries entries, and productrelease entries? [02:21] but didn't know about the new field to tell a productrelease what productseries it is part of? [02:23] sabdfl: err, sure if you want, can't imagine why you'd want to tho? [02:24] sabdfl: yes, I missed the field productseries in table Productrelease, so it is almost unuseful :(, sorry [02:32] spiv: is there some magic to prevent twistd from decorating the output of the buildbot? The "2004/10/14 14:31 CEST [Broker,client] " prefix breaks the integration of emacs with pdb. === SteveA [~steve@office.pov.lt] has joined #launchpad [02:35] elmo: wow that was exciting [02:35] just DOS'd myself with a bad .ssh/config [02:38] want to try this at home? [02:38] Host zhongshan [02:38] compression yes [02:38] HostName zhongshan.ubuntu.com [02:38] Host *.ubuntu.com [02:38] ProxyCommand ssh mark@chinstrap.ubuntu.com nc -q0 %h %p [02:38] elmo: was hoping to be able to put the proxycommand in a *.ubuntu.com entry [02:38] and then use separate entries to map host to ubuntu.com [02:40] Kinnison: what do I need from librarian to run gina now ? [02:43] sabdfl: turn on tab completion [02:43] cprov: You need a running librarian [02:44] then it's just zhon, rather than zhonghan or zhongshan.ubuntu.com [02:44] elmo: still not working, even if I specify things completely === limi benchmarks Javascript [02:44] Host *.ubuntu.com [02:44] compression yes [02:44] ProxyCommand ssh mark@chinstrap.ubuntu.com nc -q0 %h %p [02:44] sabdfl: chinstrap.warthogs.hbd.com dude [02:44] sabdfl: yes, I know the half and half things sucks, sorry [02:44] sabdfl: 20K entries is what we are expecting for packages, roughly? [02:45] Kinnison: do you have it zhongshan, in a easy way that I can copy? [02:45] You edit lib/canonical/librarian/server.tac to set a path to the storage and to choose a port for it [02:46] then you run (from the level where you'd make run for launchpad) twistd -noy lib/canonical/librarian/server.tac [02:46] If you don't have it already; then prefix that twistd command with 'PYTHONPATH=lib ' [02:47] Kinnison: ok [02:48] cprov: It's important to remember that the database and filesystem path combination work together for the librarian [02:48] so if you blow the db away; remember to blow the filesystem it created away [02:48] Also if you copy the DB, copy the filestore too [02:48] limi: for a single distro that would be the max for now, yes [02:48] ok [02:48] cprov: *or* mod gina to make the librarian interface optional :-) [02:49] Kinnison: I thinking in this possibility now :) === cprov has just give it up when looks the code [02:54] ew [02:54] the tarfile error is caused by getting a 401 from chinstrap... [02:54] (auth required) [02:55] in zenity.info: [02:55] CvsTarFile: http://chinstrap.warthogs.hbd.com/~jdub/cvsballs/zenity.tar.bz2 [02:57] So... what would be the magic trick to make that work from my workstation? [02:57] elmo: any idea? === SteveA [~steve@office.pov.lt] has joined #launchpad [03:01] cprov: eh? === ddaa investigate having pycurl handling ssl and auth === ddaa wishes his fingers knew how to write english. [03:04] Kinnison: yep, It will take so long to mod gina to work w/o librarian and I'll take the risc of kill you work, I've just perform some surgery on my DB and recover the previous generated gina data [03:05] Kinnison: but I'll be glad if you can do the modularization of gina :) [03:05] cprov: Maybe when I've sorted this poolbuilder === Kinnison is currently eating his lunch; then I'll get back to writing lib.canonical.lucille.publishing.SourcePackagePublisher [03:07] Kinnison: thanks, enjoy you lunch ! === BradB|away is now known as BradB === limi finds out what he suspected: javascript isn't the fastest language to do string comparisons in [03:09] :) [03:26] sabdfl: so, now it is ok, nicole is generating correct productreleases and I modified it in the way that you can take partial dumps when it still running to have a kind of DB snapshot [03:27] sabdfl: I have one im my home with ~10 projects ...if you want you can do you own now (~30) === ddaa wonders if curl could use .netrc of http authentification... === debonzi [~debonzi@200.158.100.251] has joined #launchpad [03:44] Looks like it works... [04:03] cprov: could you do a dump for me please, and point me at it? [04:04] sabdfl: of course, just a second [04:05] limi: figured out how to make javascript fly? [04:05] sabdfl: scp zhongshan.ubuntu.com:/home/cprov/lp_dump-productrelease-partial.sql.gz [04:07] cprov: thanks! [04:07] do I just drop my db and then use this to recreate it? [04:07] sabdfl: "not fly", as the case may be - the conclusion is that we should use XML-RPC with an inline search widget to do this server-side - with a fallback on the button to do a traditional search if you don't have JS [04:07] dropdb foo; createdb -E UNICODE foo; zcat bar.sql.gz | psql foo [04:07] Javascript just won't handle 20K items (tested with 5K) [04:08] limi: absolutely no way to do this client side? [04:09] nope, too many entries - but it is client-side in the sense that you don't leave the form [04:10] stub: Can we drop the dependence on plpython? The security concerns it adds don't seem worth it for 10 lines of Python validation code. [04:10] sabdfl: dropdb XXX; createdb -E UNICODE XXX, psql -f XXX [04:10] What is the security concern? [04:11] The fact that pqm has to run as a superuser on the DB for us to checkin changes ;) [04:12] I don't think that actually works yet, but it was supposed to yesterday, until we hit that roadblock. [04:12] a postgres superuser, although I'm discussing this with steve on habber atm. Why is that a problem? [04:12] stub: It's a big risk to take if the production database is on the same machine as a non-production one. [04:13] And, well, it's 10 lines of Python. :) [04:13] I wasn't aware we had that situation anywhere. [04:14] I'm sure elmo could provide a more detailed explanation of the risks involved though. [04:14] It will be more one day. It is trivial to redo it in pl/pgsql so I'm not fussed about the current code but it is more for future expansion. [04:15] It means that pqm has write access as the postgres user, so they can trash all the postgres databases on that server if they want. [04:15] yep, that's the most obvious risk. there are probably other things though too. [04:16] It isn't a problem on a single database server, since they can trash that single database anyway by dropping tables. [04:16] stub: I'd like to get pagetests running on pqm today. [04:18] Is there a smoother way to run the page tests yet? [04:18] what needs to happen in the set-up scripts for the launchpad_test database to make it so that they can be run by pqm? === BradB see a malone page test failing after a star merge. [04:19] The tests I had did everything - built the database etc. I have no idea what other people are trying to do in their tests. [04:20] cprov: shortdesc seems to be picking up parts of sentences [04:20] for example: [04:20] AbiWord is a cross-platform Open Source word processor. The goal [04:20] why does it get "The goal" at the end? [04:21] fm has the summary and description fields, which should map to our shortdesc and description [04:21] BradB: should the tests all be passing now? === BradB will be really happy of checkins can't happen unless the page tests pass. Then I can be lazy and not have to spend time fixing stuff that was already working in Malone. [04:22] sabdfl: The page tests; there doesn't seem to have been much done yet to make the UT and FT's pass though. [04:22] how do i run just the page tests? [04:23] sabdfl: I run them like this: bradb@ozone:~/launchpad/lp$ PYTHONPATH=~bradb/launchpad/lp/lib python lib/canonical/ftests/test_pages.py [04:23] stub: I need to know what I can hook into my page-test-runner to set up the database [04:23] sabdfl: shortdesc is always unexpectable (it should cut on first dot), I'll request it to Morgan [04:23] stub: right now, it runs "make" in database/schema [04:23] stub: but, pqm can't do that [04:24] Can pqm create and drop databases? [04:24] yes [04:24] so the only problem is plpythonu? [04:24] yes [04:24] err, no [04:24] no? [04:24] I gave pqm createuser last night [04:25] and it already had createdb -> it's postgres superuser === SteveA lets elmo and stub talk about it [04:25] are we going to need to change things on Mawson to get plpgsql to work? [04:25] guys, my requirement is to get pqm to be able to set up the launchpad_test database with appropriate contents and constraints etc., like it would be in production. [04:26] so there shouldn't be any problems then. Running make does that. === carlos [~carlos@69.Red-80-33-181.pooles.rima-tde.net] has joined #launchpad [04:27] cprov: alsa got a project but not a product. why? [04:27] hi [04:27] hi Carlos [04:28] stub: the problem is, requiring superuser in postgres is insane. if, for example, anyone wants to run a launchpad type instance anywhere near the machine running katie, I'm just going to point and laugh at them [04:28] stub, elmo: so I can re-enable running "make" in database/schema on merge? [04:28] even on chinstrap, there's more than pqm using postgres [04:28] SteveA: AFAIK, yes [04:28] carlos: could you update the StaffCalendar page with your new working hours? [04:28] yes [04:28] elmo: is this a temporary hack, or how it will be? [04:29] SteveA: *shrug* I dunno, if none of the other postgres users on chinstrap mind, then it can probably stay, but I'm trying to point out that it's not a sane long term solution IMO [04:30] sabdfl: cause the product algorithm seems to failed to look for alsa-driver, alsa-tools, etc, since nothing is returned with these names from Morgan lib [04:30] ok - then we have to drop using embedded python procedures. I wasn't aware we had that many production databases to worry about. [04:30] carlos, daf: Can we have a rosetta team meeting on #canonical-meeting, now? [04:30] SteveA: yes [04:30] stub: it's not just embedded python - it's the whole drop the db and recreate it - even with just 'createdb' the pqm instance could, f.e. drop the buttress database [04:31] sabdfl: I'm modifying it now to insert default projects either it hasn't a respective SF/FM product [04:31] maybe I'm being over paranoid, and maybe there really isn't a better way to do it in postgres [04:31] elmo: That can be fixed, although it is much more likely to hang. [04:31] elmo: suid script to set up the database we need in the way we need it ? [04:32] The butress database is running on chinstrap??? [04:32] (I know "suid script" doesn't make sense...) [04:32] there's a 'buttress' db on chinstrap - I don't know whether it's the master or what [04:35] createdb/dropdb shouldn't be a problem if we don't mix production and dev on the same server (and neither should plpythonu). [04:46] SteveA: So it sounds like at the moment at least, just running 'make' should be fine for PQM to build the launchpad_test database [04:46] ok, I'm going to try turning this on again. [04:47] elmo: Can you let me know if I need to remove the dependancy on plpythonu on the server, and if PQM needs to lose the ability to drop databases. [04:47] Already fixing Malone bugs that were spotted quickly with page tests. :) [04:48] Losing plpythonu isn't a problem at the moment, but just restricts us a bit. [04:48] (for future work) [04:48] PQM can lose the ability to drop and create databases, but it might cause deadlocks on occasion if a previous run didn't close connections to the database cleanly [04:51] SteveA, daf: Are the page tests docs going to be released today? I'd like somewhere that I can read about how to actually document the page tests (i.e. inside the .txt file itself), and a point of reference showing me how to fix common problems I might encounter (like the IOError and tabs vs. spaces thing from last night) [04:52] BradB: yes. And, I'd like to see if you can reproduce the tabs vs spaces thing with the new system. [04:53] me too :) [04:53] BradB: I need to see instructions on how to reproduce an IOError. Can you file a bug on it, with instructions on how to reproduce it? [04:53] Are we using webunit or something for page tests, or something home brewed? [04:54] SteveA: sure [04:54] stub: the thing from zope3 [04:56] I'm actually working on a better postgresql test harness at the moment so tests are better isolated (without having to use the existing harness that builds the database every single test) [04:56] how would that work? [04:57] By making connection.commit() a noop, meaning I can rollback all changes since the start of the test. It will screw tests that span zope transactions though if they expect an exception to rollback database changes (in some cases) [04:58] Bug 2049 resolved: Link from source package release to binary package release is not displaying version [04:58] https://bugzilla.warthogs.hbd.com/bugzilla/show_bug.cgi?id=2049 [05:02] this won't work well for page tests I think. [05:02] but, should work okay for smaller-scope functional tests [05:04] how comes the buildbot test suite breaks because of some code in cscvs which relies on a contract in pyarch which has changed months ago... [05:04] bob2: any insight? === SteveA will now test that a broken pagetest doesn't get merged === morgs [~morganc@wblv-252-141.telkomadsl.co.za] has joined #launchpad [05:07] ddaa: how long does it typically take to get a new import up to syncing status? [05:08] sabdfl: no idea, still working on my first new import. [05:08] morgs we have your code up and running in "nicole", celsos script [05:08] seems to be a good start, thank you [05:09] If you mean the time of the initial import, in my testing environment, it's of the order of one small hour (slow cpu here). [05:09] morgs: welcome [05:09] how do I do an orderBy in a MultipleJoin? [05:10] Chatting with Celso on jabber too... I'll continue here [05:10] descending, to be cute? [05:10] ddaa: goal is to get the import process to the point where the package maintainer can mostly drive it [05:10] sabdfl: things = MultipleJoin('Thing', orderBy='foo'), I belive. [05:10] cprov: just starting the RSS stuff, I was adding savannah.gnu.org repository but it doesn't have as much detail as sourceforge [05:10] spiv: and descending? [05:11] morgs: rss? [05:11] Ohk, orderBy='foo DESC' :) [05:11] "-foo" [05:11] spiv: initial import of zenity up to the breakage in taxi.py: 25mins [05:11] spiv: thanks [05:11] Or orderBy='-foo' === ddaa audits the changelog [05:12] ddaa: I'm still having trouble makinga testcase for taht :( [05:12] spiv: maybe you should ask bob2... he's the one who wrote it iirc. [05:13] sabdfl: hu see message to spiv 2 mins ago. [05:14] sabdfl: xml feeds [05:14] morgs: ah, ok, thanks [05:14] brb [05:15] morgs: great, I supose it will be easier to grab all the fields than HTML parser [05:17] any advice on xml parsing? I think there is a .deb... [05:21] libxml2 is very full of features, but rather awkward to use from python [05:21] but, you can do xpath with it, which is a very simple way of getting data from xml. [05:21] otherwise, use minidom [05:21] that's in the standard python library [05:22] use the dom methods to get to where you need to be in the tree [05:22] more verbose than xpath, but should be pretty understandable and quick to implement [05:22] SteveA: Great, that should get me what I need. [05:22] elementtree is less verbose than using a DOM, and has primative xpath support now. [05:22] There's a project to wrap libxml2 more pythonically, but it is being done in a couple of people's spare time, and it isn't really progressing. [05:23] what's elementtree ? [05:23] (my dear watson?) [05:23] effbot's XML library [05:24] http://effbot.org/zone/element-index.htm === cprov is now known as cprov_lunch [05:26] There's also beautifulsoup, but it's more intended for HTML scraping. [05:26] No reason why it wouldn't work for XML, though... [05:26] spiv: I'm doing some HTML scraping... what/where is beautifulsoup? [05:27] http://www.crummy.com/software/BeautifulSoup/ [05:27] http://www.crummy.com/software/BeautifulSoup/ [05:27] stub: Drat :) [05:27] Thanks, I'll check it out. [05:28] I've had recommendations for it from people who I trust, but haven't used it personally. [05:28] Heh: "Beautiful Soup defines two parsers: BeautifulSoup, which parses HTML with just enough smarts to be dangerous, and BeautifulStoneSoup, which doesn't try at all to be smart and is therefore well-suited for parsing XML documents and pseudo-HTML with made-up tag names." [05:28] Bug 2051 resolved: Use apt_pkg (from python-apt) to parse dependency lists [05:28] https://bugzilla.warthogs.hbd.com/bugzilla/show_bug.cgi?id=2051 === morgs is now known as morgs-going-to-m === morgs-going-to-m is now known as morgs-supper [05:29] stub: Yeah, I suggested it to Anthony Baxter for TWFY, although I'd never used it, and he came back saying how wonderful it was :) === ddaa looks for a crack pipe in lifeless private stuff [05:33] cscvs test suite blows on the first test... [05:34] Bug 2018 resolved: Complete information for binary package release. [05:34] https://bugzilla.warthogs.hbd.com/bugzilla/show_bug.cgi?id=2018 [05:35] Bug 2010 resolved: Build pages need to be defined for a binary package release [05:35] https://bugzilla.warthogs.hbd.com/bugzilla/show_bug.cgi?id=2010 === debonzi is now known as debonzi_lunch [05:51] erm, how do i make a select a dropdown listbox? [05:51] size="1" or something like that iirc [05:52] that's obvious, why didn't i think of it? [05:52] and, make sure it isn't a multiple select [05:53] BradB: the pagetests are now running on merge. That is, a merge will be rejected if a pagetest fails. Next, I want to talk through it with daf. Daf will update the docs he wrote, and post them to the list. Meanwhile, if you don't mind, I'll move all your (and others') existing page tests into the new location. [05:55] SteveA: By the looks of things, you've removed utilities/page-test-helper, in favour of ./makepagetest.py, right? [05:55] um... removing page-test-helper was sort of an accident [05:55] I wanted to remove it, but not so soon [05:55] but, yes [05:56] hang in there, we'll be out of the tunnel in an hour or so [05:56] great, just in time for me to return from lunch :) [05:56] I'm really happy that we won't be able to merge unless the page tests pass, since if it had been like that before, I'd be looking for other tasks to do, instead of fixing stuff that had already been working in Malone. :) [05:57] ok, have a good lunch === BradB is now known as BradB|lunch === sabdfl scrambles to merge crack before the iron curtain falls [05:58] kidding === ddaa gasps at os.getlogin() raising a IOError on the grounds of "No such file or directory"... === ddaa reads the documentation, blames lifeless, and uses os.environ['LOGNAME'] [06:15] huh? [06:16] would you prefer pwd.getpwuid(os.getuid())[0] ? [06:16] AFAIK that's what os.getlogin() does if LOGNAME fails - why wouldn't it fail with the same IOError if you're really missing /etc/passwd [06:16] SteveA: about those clearer debugging messages [06:16] try this one [06:17] here's the last bit of the traceback: [06:17] Module zope.app.traversing.adapters, line 167, in traversePathElement Module zope.app.traversing.adapters, line 52, in traverse [06:17] __traceback_info__: (, 'releases', [] ) NotFoundError: (, 'releases') [06:17] looks fairly clear [06:17] but releases is in both the interface, and the object definition [06:17] hmm [06:17] further digging: [06:17] os.getlogin raises an IOError in the cscvs test suite for some unfathomable reason, but... [06:18] sabdfl: looks like a traversal directive is missing [06:18] (Pdb) print subject [06:18] [06:18] (Pdb) subject.id [06:18] 4 [06:18] (Pdb) subject.name [06:18] u'firefox' [06:18] (Pdb) subject.releases [06:18] *** AttributeError: 'ProductRelease' object has no attribute 'datecreated DESC' [06:18] (Pdb) [06:18] noo.... [06:18] elmo: thanks for making me think, I should investigate more on what happens in the debugger [06:19] really the problem is with an SQLObject that is not mentioned ever in the traceback [06:20] right, so Zope is trying to traverse from the firefox product to its releases [06:20] and when it tries to do this, it gets a AttributeError [06:20] it dies during the join, yes [06:20] which it translates into a NotFoundError [06:20] very helpfully [06:21] why does dieing during the join lead to an AttributeError? [06:21] elmo: so, os.getlogin fails, but the latter does work... [06:21] rather than a database error? [06:21] for some unfathomable reason. === limi [~limi@212.80-202-72.nextgentel.com] has joined #launchpad [06:22] "dying during the join"? [06:22] how did we work that out? [06:22] daf: look at the pdb output [06:22] something is trying to access a non-existant attribute 'datecreated DESC' [06:23] sabdfl: ideally, what would you like the error to be in this case? [06:23] the problem is with SQLObject, not Zope [06:23] the output of the AttributeError told me exactly what the problem was [06:23] spiv told me to use orderBy="foo DESC" which i did [06:24] I think it's quite reasonable to turn an AttributeError into something not being found [06:24] but i used datecreated [06:24] and the fieldname is actually datereleased [06:24] what was perverse is that the context/releases thing works FINE if there are no releases [06:24] because the orderBy never is used [06:24] it was a fun 20 minutes :-) [06:25] ah... was it a "deeper" attribute error? [06:25] I don't know under what circumstances an AttributeError is generated, but [06:25] a failed JOIN is not "attribute not found" [06:25] that is, the traversal code is interested in an "immediate" attribute error -- that is, the attribute it is trying to get does not exist [06:25] SteveA: yes [06:25] okay... that's really a python limitation [06:25] SteveA: it exists though [06:26] there's a hack to work around it, though. but it ain't pretty [06:26] I'm pretty sure this is a bug in SQLObject [06:26] which results in a misleading exception [06:26] so zope can't tell the difference between trying to access an attribute which does not exist, and an AttributeError raise by an attribute which DOES exist? [06:26] what *should* happen is that the traversal code should be interested in an "immediate" attribute error only, and should just pass on a "deeper" attribute error [06:27] Python can't tell the difference. [06:27] this is one of python's "the snake bites you on the arse when you aren't expecting it" situations [06:27] ok [06:27] the hack involves examining the depth of traceback [06:27] Zope can't tell the difference between an AttributeError that was raised by the object it was looking at directly, and an AttributeError raised by code that runs when accessing an attribute that does exist [06:27] compared to the current stack frame [06:27] is "productrelease" a sane default for traversal past product? [06:28] in doap, at least? [06:28] releases is a magical attribute which runs code when it is accessed [06:28] when that code raises an AttributeError, it looks to Zope like that attribute doesn't exist [06:28] so the problem is with the code run when the property is accessed [06:28] what if SQLObject raised a differnet exception? [06:29] I don't know === SteveA discusses introducing the "pass through deep attribute errors" hack with zope3 upstream [06:29] but if you fix SQLObject to do the orderBy properly, the exception shouldn't occur [06:30] doing the exception-depth magic is a longer-term solution to getting better error messages from Zope === Kinnison [~dsilvers@haddenham.pepperfish.net] has joined #launchpad === Kinnison grrs the librarian for not being able to retrieve by alias alone [06:38] spiv: ping? [06:39] spiv: orderBy='foo DESC' definitely doesn't work [06:40] but orderBy='-foo' seems to work ok [06:42] sabdfl: but what happens if foo is not a number? === SteveA just discussed the traverse / python / deep vs immedate errors problem with Fred Drake [06:44] I'm going to see if I can improve this for traversal in zope3, and thusly for launchpad [06:45] should be a reasonably quick thing to hack up [06:51] carlos: 'foo' is an attribute name [06:52] carlos: i.e. in a string [06:52] sabdfl: I suspect what might have happened is that support for 'blah DESC' was removed in SQLObject 0.6 [06:52] daf: I know, but if the Attribute is not a number, the "-foo" thing will not work, right? === ddaa winces [06:53] the cscvs test suite that the $HOME of lifeless hardcoded... === cprov_lunch is now known as cprov [06:53] carlos: SQLObject translates the '-' into a 'DESC' in the SQL [06:54] daf: ohh, I see [06:54] that's not too userfriendly but if it works... :-P === Kinnison [~dsilvers@haddenham.pepperfish.net] has joined #launchpad [06:54] spiv: reping [06:57] how do I remove a directory in my arch tree? [06:57] can I just remove it, and commit? [06:57] yep === debonzi [~debonzi@200.158.100.251] has joined #launchpad [06:58] hmm, it tells me it is not empty. It contains a .arch-ids directory. [06:58] daf: are you working on https://bugzilla.warthogs.hbd.com/bugzilla/show_bug.cgi?id=2036 ? [06:58] rm -rf [06:58] SteveA: yeah; all dirs contain one [06:59] The .arch-ids dir contains the explicit ids of the directory and files at the top-level. [06:59] ddaa: I'd prefer just rm -r [06:59] SteveA: as you wish [07:00] thanks ddaa and Kinnison [07:00] SteveA: the idea is that you can mv a dir; and arch knows you renamed it rather than deleted a whole pile of shit and re-added it === ddaa decides on $LOGNAME... simpler === BradB|lunch is now known as BradB [07:25] limi: how do i add a picture to a page on the web site? [07:25] we'll need a way of testing emails as well as pages. [07:25] sabdfl: in HTML or Structured Text? [07:25] either? [07:25] Yay! I have finally beaten the cscvs test suite into passing! [07:25] how do i make the image available in the first place? [07:26] "This is image description":img:someimage.jpg [07:26] oh [07:26] just add Image from the pulldown [07:26] cool, thanks [07:27] it will keep its name by default, but you can change it afterwards if you need to [07:28] spiv: Is there a good reason why I can't retrieve a file from the librarian by aliasid alone? [07:31] limi: can i set the size of it in StructuredText? [07:31] sabdfl: no, STX is only for semantic markup in a way - you have to use HTML for layout changes [07:31] but you can mix them [07:32] so use STX, and display the image using limi: or better.. class="borderedImage" and then define the class in the CSS ? [07:38] limi: Or am I abstracting too much? [07:38] Kinnison: well, Mark is making a simple one-instance change here :) === Kinnison gets lost in SQLObject [07:38] hehe [07:38] abstraction alert === Kinnison sets off the sirens [07:39] it would be the right thing to do throughout a site, yes === kiko [~kiko@200.144.121.120] has joined #launchpad [07:39] but "borderedImage" is not really a useful name, to be nitpicky - 2 months down the line you decide to use a different visual effect for it [07:40] yep; well said === limi has seen stuff like class="greenBorder" [07:40] limi: eww [07:40] or, my favorite

[07:40] limi: I may not be good at chosing css class names; but at least I'm not that bad :-) [07:40] ;) [07:40] class="greenBorder normalSize" === Kinnison has things like: