/srv/irclogs.ubuntu.com/2011/07/19/#launchpad-dev.txt

=== wgrant changed the topic of #launchpad-dev to: Performance Tuesday | https://dev.launchpad.net/ | On call reviewer: - | Critical bugs: 238 - 0:[######=_]:256
wgrantsinzui: Did you typo your MUA's display name?00:36
wgrantYou've lost your capital "c"00:36
wgrantsinzui: Nice presentation.00:40
pooliewgrant: ?00:57
pooliei filed https://bugs.launchpad.net/launchpad/+bug/81259700:57
_mup_Bug #812597: gina imports from debian arrive too slowly <gina> <udd> <Launchpad itself:Triaged> < https://launchpad.net/bugs/812597 >00:57
wgrantpoolie: It would only be 7 hours if Debian decided to shift their publishing schedule to right after gina runs.00:59
wgrantThey still only publish every 6 hours, AFAIK.00:59
pooliei see01:00
poolieand we're pretty sure we're in sync with that?01:00
StevenKIt requires checking01:00
wgrantThe current schedule was devised to run just after they published.01:00
wgrantBut they like to change.01:00
wgrantWithout telling anybody.01:00
persiaDoesn't Debian have a push-mirror feature for some mirrors that runs right after publication?  Could one host a mirror, and then use post-mirroring hooks to call gina at the right times?01:13
StevenKwgrant: The amount of QA to do makes me sad.01:14
huwshimihmm... is there any way to test emails on qa staging?01:58
lifelessyes01:59
lifelessfor in you need losa to run the script by hand01:59
lifelessfor out you also need to run the relevant script by hand01:59
lifelessbut then you can access the shared mailbox to see what was sent out02:00
wgrantsinzui: Anything to watch out for in oneiric atm?02:02
lifeless dev/run, ecryptfs were fragile recently, may not be totally sorted yet02:02
lifelesswgrant: what presentation ?02:02
wgrantlifeless: sinzui's velocity presentation from the Prague epic.02:03
lifelessah yes02:03
lifeless+102:03
wgrantI use LUKS rather than ecryptfs, because it's flaky and slow.02:03
huwshimilifeless: Ok, I would like to get a notification of a tag change from a bug. What scripts do I need to run? Is there a wiki page for this?02:04
wgrantStevenK: QA!03:12
StevenKwgrant: I have a better idea. LUNCH!03:12
wgrantI was just doing a pre-lunch QA check myself.03:12
StevenKI shall update mawson now. And then break for lunch.03:12
wgrantThanks.03:13
wgrantHuh.03:13
wgrantJulian used one of Job's YAGNI columns.03:13
StevenKIndeed03:13
lifelesswhich one ?03:14
StevenKrequester03:14
wgrantHm.03:39
wgrantHas someone decided to show the battery indicator at all times in oneiric? :(03:39
* wgrant blames thumper.03:39
thumperheh03:39
thumperwgrant: not me though03:39
thumperbut at least I have a battery indicator now03:39
wgrantBah.03:39
wallyworldthumper: global menus fixed yet?03:49
thumpernah mate03:49
* wallyworld sad03:50
wgrantwallyworld: Hm, we have a picker regression.03:55
wallyworld?03:55
wgranthttps://qastaging.launchpad.net/launchpad/+filebug, the "Assign to" picker has quotes around the text.03:55
wgrantNot there on prod.03:55
LPCIBotYippie, build fixed!03:56
LPCIBotProject db-devel build #731: FIXED in 5 hr 31 min: https://lpci.wedontsleep.org/job/db-devel/731/03:56
wgrantNormal bug assignee pickers are OK.03:56
wallyworldhttps://code.launchpad.net/~wallyworld/launchpad/picker-displays-null-812253/+merge/68318 should fix it03:56
StevenKwgrant: How do you suggest I QA r13455?03:56
wgrantStevenK: Reject a delayed copy.03:57
wgrantwallyworld: Ah, great.03:57
StevenKThat means I need to create one03:57
wgrantStevenK: Easy enough to do.03:57
StevenKAnd delayed copies make me cry.03:57
wgrantYes03:57
StevenKwgrant: So copy something from the soyuz-team P3A to somewhere else?03:58
wgrantStevenK: Something that's not already public, yes.03:58
* StevenK digs03:58
wgrantWhy is syndaemon enabled by default :/03:59
wgrantHow stupid.03:59
StevenKDelayed copy of launchpad-dependencies - 0.39 (source)04:08
StevenKGood enough04:09
=== almaisan-away is now known as al-maisan
StevenKwgrant: But I can't find that delayed copy using queue?04:11
LPCIBotYippie, build fixed!04:14
LPCIBotProject devel build #898: FIXED in 5 hr 34 min: https://lpci.wedontsleep.org/job/devel/898/04:14
wgrantStevenK: You'll have to get the ID manually, I guess.04:17
wgrantHopefully that will work.04:17
StevenKwgrant: scripts/ftpmaster-tools/queue reject -s hardy -Q accepted 2752292 worked fine04:22
StevenKRunning: "info 2752292"04:22
StevenKItem 2752292 is in queue REJECTED04:22
wgrantGreat.04:28
wgrantqa-ok, I s'pose.04:28
StevenKAlready done04:28
* StevenK kicks bigjools for the fun that is r1345904:31
wgrantStevenK: It is flagged.04:31
StevenKThen what's left?04:32
wgrantlifeless: LXC works fairly well in oneiric.04:35
wgrantWith most of those bugs fixed.04:35
wgrantStevenK: Check that the methods don't work, I guess.04:35
StevenKwgrant: For r13459?04:39
wgrantYes.04:40
StevenKAnd lib/lp/soyuz/interfaces/archive.py contains the generic exception ForbiddenByFeatureFlag.04:41
StevenKFail.04:42
wgrantDid somebody break the JS build again?04:42
StevenKOh?04:42
wgrantAh, no, I just killed buildout in a bad place.04:43
lifelesswgrant: I'm hoping my sudo bug is fixed04:46
wgrantlifeless: Not sure.04:48
wgrantStevenK: Erm.04:52
StevenKwgrant: Hm?04:53
wgrantStevenK: [DOGFOOD] [PPA stevenk] [ubuntu/hardy] launchpad-dependencies 0.39 (Rejected)04:53
wgrantThere was a blamer.04:53
wgrantOr SPR creator, possibly.04:53
wgrantflacoste:04:53
StevenKThat is the creator04:53
StevenKwgrant: Neither copyPackage{,s} appears in the API on dogfood for devel :-(04:54
wgrantStevenK: You'll probably need to force it to rebuild the WADL04:56
wgrantStevenK: But this is to be tested on qastaging.04:56
StevenKOh, easy fixed04:56
StevenKwgrant: On qas, I get a 403 calling IArchive.copyPackage()05:00
wgrantGood.05:01
StevenKAh yes, the exception is 40305:02
StevenKShall I qa-ok it?05:02
wgrantStevenK: Might as well.05:08
StevenKGAVIN!05:17
StevenK(The 3 branches linked to bug 809786)05:18
_mup_Bug #809786: Need to filter by package-set on DistroSeries:+localpackagediffs <derivation> <qa-needstesting> <ui> <Launchpad itself:Fix Committed by allenap> < https://launchpad.net/bugs/809786 >05:18
StevenK518 lines, 4720 lines, and 287805:18
=== al-maisan is now known as almaisan-away
StevenKCurious.06:12
wgrantStevenK: Hm?06:19
StevenKwgrant: I finally got +localpackagediffs to load, but I can't tell what Gavin changed/fixed06:20
wgrantStevenK: It was pure refactoring that's landed so far, AFAICT.06:20
StevenKAll 6000 lines of it06:20
StevenKwgrant: So qa-untestable?06:24
wgrantProbably.06:25
StevenKwgrant: So the only thing left is Danilo's revision of doom06:25
=== jam1 is now known as jam
adeuringgood morning07:48
jammorning all07:50
jamI'm trying to understand an LP api timing. I could have sworn I was doing the same request last week, and it was returning in 100-200ms.07:50
jtvhi jam, hi adeuring!07:50
jamBut this week it is more like 700-1000ms07:50
jamhey jtv07:50
jamis there any way to debug these sort of things?07:50
adeuringhi jtv07:50
wgrantjam: What sort of request?07:51
jtvCan we use ++profile++ on API requests?07:51
wgrantjam: 100-200ms is rather quick for LP.07:51
wgrantjtv: Possibly... ++oops++ doesn't work, since it only executes when the template renders.07:51
jamwgrant: getPublishedSources07:51
wgrantjam: Let's see.07:52
wgrantEverything should be faster this week :(07:52
jtvThat's on Archive?07:52
wgrantCould be distroseries.07:52
wgrantOr distribution.07:52
wgrantI guess.07:52
allenapStevenK: Thanks for looking at the changes to +localpackagediffs. As long as it loads, it's fine. The changes so far are almost all moves and shakes.07:52
StevenKallenap: GAVIN! *Scary* branches! RARGH!07:53
StevenKallenap: But yes, it at least loads07:53
jamwgrant: https://api.launchpad.net/1.0/debian/+archive/primary?distro_series=%2Fdebian%2Fsid&exact_match=true&pocket=Release&source_name=%22bzr%22&status=Pending&ws.op=getPublishedSources&ws.size=107:53
jamArchive07:54
jtvYuck.  "clauses.append("SourcePackageName.name LIKE '%%%%' || %s || '%%%%'" % quote_like(name))"07:54
wgrantBut this is exact_match.07:54
wgrantSo that should be skipped.07:54
wgrant(it also shouldn't exist at all, but such is Soyuz)07:54
jtvDo we actually know we're looking at the archive code, actually?07:54
wgrantYes.07:55
wgrantdebian/+archive/primary07:55
wgrantjam: Render time is just about identical compared to 1.5 weeks ago.07:56
wgrantjam: Do you have any hard data to support the slowdown?07:56
jtvThe only recent direct change in that piece of code was in June, where it looks like multi-name matching was introduced.  But that should barely touch the exact-match code path.07:57
jtvUnless the thing gets passed some kind of placeholder that acts like a string but isn't of type str or unicode, but I don't really know whether that's possible or whether it would work if it was.07:58
wgrantAND SourcePackageName.name = E'bzr' AND08:01
mrevellGood morning08:01
wgrantFrom using that URL locally.08:01
bigjoolsmorning al08:01
bigjoolsl08:01
jamwgrant: I know I did timing tests last week, and had some perf tests.08:03
jambut no, I don't have very hard data.08:03
jamwgrant: take that back, I have my .bzr.log08:05
jamfrom today: 6.778  LatestPublication.get_latest_version took 1.524s08:05
wgrantOh, this is for the checking if the branch is up to date thing?08:05
jamfrom last week: 7.974  LatestPublication.get_latest_version took 0.214s08:05
jamwgrant: yep08:05
wgrantHave you moved to Australia lately?08:05
jamso the ones I was testing before is "ubuntu/natty/bzr"08:05
wgrantOh.08:06
wgrantThat's a bit different :)08:06
jambut that is still slow08:06
jam700ms orso08:06
wgrant:(08:06
stubjtv: Storm contains a Like operator that might spell that better if you are coding nearby.08:06
jamjust tested now 939ms08:06
jtvstub: I'm not, but that's part of what made me nauseous.08:06
jtvIt's even easier than Like; IIRC you can use the regular str methods such as startswith.08:07
jtvThere's one for "contains" as well.08:07
StevenKI recently used .contains()08:07
jamwgrant: with bzr.dev tip, you can run this script: http://paste.ubuntu.com/647087/08:07
stubeven better08:07
wgrantwgrant@carob:~$ time curl -H 'Authorization: there is none' 'https://api.launchpad.net/1.0/debian/+archive/primary?distro_series=%2Fdebian%2Fsid&exact_match=true&pocket=Release&source_name=%22bzr%22&status=Pending&ws.op=getPublishedSources&ws.size=1' -s -o /dev/null08:07
wgrantreal0m0.253s08:07
jam(python latest_version.py ubuntu natty bzr)08:08
jtvSomeone went to the trouble of coding up a hybrid storm/sqlobject method so that they could write up that horrible LIKE clause.08:08
StevenKjtv: Can you point out the method so we can have it shot?08:08
wgrantjam: So, it takes around 250ms from the DC.08:09
jtvStevenK: Archive.getPublishedSources08:09
wgrantjam: For natty it's down around 150ms, as expected.08:09
wgrantSince there's a lot less data.08:09
bigjoolsStevenK: please don't shoot that one08:09
wgrant(this isn't exactly optimised)08:09
StevenKIt deserves it.08:10
jamwgrant: I'm seeing 600ms in "_ssl.sslwrap" is that the SSL handshake ?08:10
wgrantjam: I assume so.08:10
jamso that is very different from last week, I believe08:10
wgrantSure your connection latency doesn't suck?08:11
jamwgrant: I'm in Netherlands08:11
jamping says 36ms08:11
wgrantHow long does the curl I listed above take?08:11
jam321ms I believe08:12
wgrantAnd then try a couple of times without the Authorization header.08:12
wgrant(the presence of even an invalid one gets it past Squid)08:12
jamwgrant: but that is because curl fails immediately with "SSL FAILED"08:13
jtvmorning bigjools!08:13
wgrantHelpful.08:13
bigjoolshello jtv08:13
jtvjam: want to meet up later this week?08:13
jambut I can pass -s, I guess08:13
jamI still don't get any output to tell if it is working08:13
wgrantjam: Well, probably better to make it work rather than ignoring the error...08:14
jam:)08:14
jamI wasn't sure what -s did08:14
jamapparently it is --quiet not "--ignore-certificate"08:15
jamwgrant: with -k instead of -s, I get 892ms08:15
wgrantI get no cert error locally or on carob.08:15
jamwgrant: launchpad isn't in my cacerts08:15
jam-k works fine for testing08:15
wgrantMine are stock Natty.08:15
jamso, still the same time for a successful request, about 900ms, which could be 600ms in ssl handshaking08:17
jamwhich is definitely slower than last week08:17
jamI don't know why08:17
jamis there a graph/monitoring of ssl stuff?08:18
wgrantInternally it's fine.08:20
jamwgrant: right, so from devpad, I see 148ms08:23
wgrant900ms from 150ms away08:24
wgrantSo 650ms overhead, which is about four ways...08:24
jamwgrant: I'm 38ms away by "ping"08:24
jamso 17 round tirps08:24
jamtrips08:24
=== almaisan-away is now known as al-maisan
wgrantOdd.08:26
wgrantStill, fine in and next to the DC.08:26
wgrantSo I am tempted to blame your machine/connection.08:26
wgrant.nl hasn't imposed a Great Firewall lately? :)08:27
wgrant:(08:33
wgrantzope.app.testing fixed their Referer default.08:33
wgrantSo 350 or so tests fail with OffsiteFormPost errors.08:34
jamvila: can you try the query from France?08:47
vilaotm08:47
jamtime curl -k 'https://api.launchpad.net/1.0/debian/+archive/primary?distro_series=%2Fdebian%2Fsid&exact_match=true&pocket=Release&source_name=%22bzr%22&status=Pending&ws.op=getPublishedSources&ws.size=1' -o test.txt08:47
rvbajam: real0m0.532s08:48
rvba(from Paris)08:48
wgrantrvba: Is that after a couple of tries?08:49
rvbareal[0m0.362s - 0m0.532s]08:49
rvbawgrant: that's the range I get08:49
rvbaAfter a couple of tries: real0m0.344s08:50
wgrantThat's a bit better.08:50
jamwgrant: of course *right* now it just dropped to 300ms, but now curl varies from 300-953ms08:51
jamand it still is taking 700+ from python08:52
jamwhich still doesn't get close to the 150ms from last week08:53
wgrantWas this on the same machine in the same environment?08:55
jamyes08:55
jamactually, the only difference offhand is that now I'm on a wired connection to my router, while last week I was on wireless08:55
jamI guess it was 200ms last week, which is close to 300.08:56
jamthe variability is surprising08:56
jamand last week was python, not curl08:56
wgrantI get 50ms from squid.08:56
wgrantWhich sucks, but it's always within 1ms.08:57
jamwhat do you mean by "from squid" ?08:57
StevenKWe have a proxy in front of the appservers08:57
jamStevenK: sure, but is he saying bypassing the squid proxy, the appservers respond in 50ms09:00
jamor from the squid machine09:00
jamor he gets 50ms making a request to squid without an appserver09:00
StevenKNo, he is saying that squid replies in 5009:00
wgrant50ms to get the copy that squid has cached.09:00
wgrantie. after requesting a few times without Authorization09:00
jamwgrant: my requests also were without auth, but you're saying from where to squid?09:03
wgrantcarob to squid09:03
wgrantVia the usual HTTPS frontends09:03
wgrantwgrant@carob:~$ time curl 'https://api.launchpad.net/1.0/ubuntu/+archive/primary?distro_series=%2Fubuntu%2Fnatty&exact_match=true&pocket=Release&source_name=%22bzr%22&status=Pending&ws.op=getPublishedSources&ws.size=1' -s -o /dev/null09:04
wgrantreal0m0.050s09:04
wgrant(Debian is, unsurprisingly, identical)09:04
jamsure. Though that just dropped as well09:05
jamit was 150ms for me when I first reported devpad numbers09:05
jameven running a couple of timse09:05
jamtimes09:05
jamwgrant: so I still think there is something that is slowing queries down, that magically disappeared in the last hour09:06
jameg, we're missing some monitoring somewhere that would tell us something is going on for people actually getting a much slower launchpad09:08
wgrantEven in front of squid?09:09
jamwgrant: I ran the same query, and it was 150ms from devpad to api.launchpad.net09:09
wgrantHmm.09:10
jamI think I managed to provoke squid to skipping just now, and I got 400ms09:10
jam(that was to debian, though)09:10
jambut I'm playing with setting Authorization, and I'm getting 50ms09:11
vilajam: first try: 1.355s real, then 0.249s, 0.199s, 0.194s, 0.189s , 0.198s09:11
jamso it sounds like squid doesn't actually pay attention to it.09:11
jamvila: right, so those sound like cached and uncached responses.09:11
vilafurther tries stabilize at 0.200s09:11
jamand arguably we should be really counting the uncached values as our user-experience.09:11
jamwe might need to put this behind a flag09:11
jamvila: I couldn't figure out what you meant by OTM, on-the-mumble?09:13
vilajam: yeah :)09:13
wgrantjam: Apache is configured to send anything with Authorization or Cookie straight to haproxy, not squid, AFAIK.09:13
jamwgrant: tell that to my curl results :)09:13
jamthough I could be typing something wrong09:14
jamah, Authentication not Authorization09:14
jammy bad09:14
jamright, that does bypass it09:15
jamthat's 479ms stabilizing to 230ms on carob bypassing squid requesting debian urls09:15
jamwhich is hopefully worst-case in DC times09:15
jml I clicked "Build now" on https://code.launchpad.net/~testing-cabal/+recipe/testtools-daily and get what looks to be a JS alert saying "Server error, please contact an administrator"09:16
jmlwhat's going on?09:16
jamwgrant: ok, this is weird. curl is going to 91.189.89.224, but python is going to 91.189.89.22509:21
jamand *that* seems to be the difference in timings09:21
jamit is "banana" vs "nutmeg"09:22
StevenKYes, they are two frontends09:22
jamStevenK: any idea why banana would be 300ms slower than nutmeg?09:22
wgrantBoth 50ms cached...09:24
wgrantAnd both 200-250ms uncached.09:25
lifelessstub: hi09:28
stublifeless: hi09:29
lifelessstub: I thought I'd touch base on 'kick connections out and keep em out' implementation choices09:29
bigjoolsanyone got any ideas on where to look to investigate why an api method I added is not available over the wire to dogfood?  It's in dogfood's WADL but launchpadlib  says "'Entry' object has no attribute ...."09:34
=== gmb changed the topic of #launchpad-dev to: Performance Tuesday | https://dev.launchpad.net/ | On call reviewer: gmb | Critical bugs: 238 - 0:[######=_]:256
jamwgrant: how do you get curl to target a server?09:35
jamcan you just do the IP address, or are they vhosts?09:35
jam(I get ACCESS DENIED going to the IP address.)09:36
wgrantjam: They're vhosts on special IP addresses.09:36
wgranttime curl -H 'Authorization: blah' -H 'Host: api.launchpad.net' 'https://91.189.89.225/1.0/debian/+archive/primary?distro_series=%2Fdebian%2Fsid&exact_match=true&pocket=Release&source_name=%22bzr%22&status=Pending&ws.op=getPublishedSources&ws.size=1' -k -o /dev/null09:36
jamwgrant: thanks. So bypassing SQUID going to ubuntu/natty I get a reliable 470-530ms to either server (nutmeg or banana)09:39
jamand without Authorization, I get 330-360 from curl09:41
jamso something about python is provoking something09:41
stublifeless: I think we are shutting down pgbouncer, which kills all the connections with 'connection terminated unexpectedly' exceptions and future connection attempts will timeout as nothing is listening. The 'suspend' and 'pause' don't terminate the active connections and ignore the 'fail if unable to login in this many seconds' parameter.09:46
lifelessstub: ok09:46
lifelessstub: So we'll need the test fixture to be able to shut down and bring back the one config; I'll add that into a 0.2 tomorrow if noone else gets to it09:47
gmblifeless: I don't know if you'll get chance, until your tomorrow, but I've replied to your review of https://code.launchpad.net/~gmb/launchpad/bug-491886/+merge/68174.09:47
stublifeless: We want a pgbouncer test fixture?09:48
lifelessstub: we have one09:48
lifelessstub: the project I put together on Monday09:48
lifelessstub: so we can test this well - we don't need the whole test suite to use pgbouncer, just the tests for interrupt & reconnect detection09:48
stubRight. We used to have some disconnection/reconnection tests that used a simple twisted proxy, but this moved to Storm. We don't need to use pgbouncer if we don't want - as far as PG clients are concerned, they just have a socket.09:50
lifelessstub: its pretty quick to bring up, and the code is written09:50
lifelessstub: I suspect we only need 3-4 tests per daemon09:51
bigjoolslifeless: any chance you can help me debug a webservice problem on dogfood please?09:53
bigjoolsI know it's rather late for you09:54
lifelesssure09:54
lifelesssec09:54
lifelessgmb: reviewed09:54
lifelessgmb: generally, I'm happy for any other reviewer to do a follow up once you consider (and action or reject) my suggestions09:54
gmblifeless: Thanks, and duly noted for the future.09:55
lifelessgmb: I don't think you need to consider the review pinned to me (unless I specifcally say I want that)09:55
gmbOkiedoke.09:55
lifelessgmb: I think the try:except: looks a lot simpler :)09:55
lifelessbigjools: whats up ?09:55
gmblifeless: Agreed. I was trying to avoid the transaction.abort() but simplicity beats I-don't-want-to-have-to-deal-with-transactions.09:56
bigjoolslifeless: I added IArchive.copyPackage to the API. It's on DF's wadl/+apidoc but launchpadlib can't see it :/09:56
lifelessbigjools: delete your lp lib cached09:56
StevenKbigjools: Remove your cached WADL?09:56
bigjoolsGAH09:56
bigjoolsthis old chestnut09:56
bigjoolswhy why why09:57
lifelessif it works, file a bug09:57
lifelessI would -so- like to nuke the use of WADL, but I'm old school.09:57
bigjoolsanything with XML in it reeks of shit09:57
StevenKIt lends itself so well to ridicule, though. "qas is WADLing"09:58
lifelessgmb: oh, and the linty stuff I don't care about really, I mentioned it for completeness09:58
bigjoolsyeah that worked.  thanks lifeless09:59
lifelessbigjools: file a bug ?09:59
bigjoolsyeah doing so09:59
lifeless\o/10:00
bigjoolsbut on where10:00
lifelesslaunchpadlib10:00
lifelessits the thing choosing to use wadllib etc; and its the thing we need a versioned dep on to ensure a newer wadllib if thats how we fix it10:00
cjwatsoncould somebody help me land https://code.launchpad.net/~cjwatson/launchpad/multiarch-translations-schema?10:03
bigjoolslifeless: bug 81279910:04
_mup_Bug #812799: Local cache not updated when it should be <launchpadlib :New> < https://launchpad.net/bugs/812799 >10:04
lifelessbigjools: triage it ! :)10:04
lifeless[I suggest high]10:04
bigjoolslifeless: jeez, gimme time to fart first10:04
lifelessbigjools: well, you're old and all, I don't know I have *that* much time to wait :P10:04
lifelessbigjools: [seriously, when you linked it I figured that meant you'd done all you intended to on it]10:05
bigjoolslifeless: my arthritic fingers at my old age don't work as quick as that :)10:05
stublifeless: pgbouncer might help test suite race conditions tearing down/creating databases10:08
bigjoolsawesome, feature flags are case-sensitive10:08
stubmmm... probably not. its more pg tripping over its own feet and waiting for its internals to catch up rather than inbound connections10:08
lifelessstub: if it could force only one connection at a time to template110:08
stublifeless: it could do that...10:09
lifelessstub: anyhow, my patch for that seem to work, and we're looking at lxc for parallel in the short-medium term anyhow, now.10:09
lifelessbecause there are so many similar glitches in our stack at the moment10:09
lifeless[e.g. launchpadlib test cache dir]10:09
=== al-maisan is now known as almaisan-away
jmlcjwatson: sure.10:16
* wgrant curses Python10:22
wgrantDon't make backwards incompatible changes in 2.7 without a non-monkeypatch way to restore the old behaviour.10:23
StevenKwgrant: Hm?10:23
wgrantThe default continuation whitespace character in generated email headers changed from tab to space.10:23
wgrantAnd this is set in an _method of a class.10:24
wgrantSo it's not really overridable.10:24
jmlYeah, python-devel seems to hate backwards compatibility10:24
jmlHmm.10:24
wgrantAh, mercurial indeed fix it by monkeypatching.10:24
jmlIt turns out I'm not able to land branches any more, because I'm not allowed to edit commit messages on merge proposals for Launchpad.10:24
wgrantYeah, the emeritus stuff isn't really set up yet.10:25
wgrantMost people are still in ~launchpad.10:25
wgrantWant me to land it instead?10:25
jmlwgrant: sure. here's the line I'm using:10:26
jml./utilities/ec2 land https://code.launchpad.net/~cjwatson/launchpad/multiarch-translations-schema/+merge/67992 --commit-text="Add include_long_descriptions column to DistroSeries to enable future work separating long descriptions out of Packages file" --no-qa10:26
jmlwgrant: I would like my commit access back too please10:27
wgrant:(10:31
cjwatsonshall I set the commit message in the MP?10:33
wgrantIt's done.10:33
wgrantec2 is almost running.10:33
cjwatsonthanks10:34
jmlcjwatson: even if it were set, I would need privileges to edit it, since our landing tools push some metadata onto it.10:34
* cjwatson nods10:34
wgrantDown to 12 2.7 test failures, plus two tests that hang.10:36
wgrantNearly there.10:36
jml\o/10:38
jmlwgrant: so the plan is to switch to 2.7 ahead of the next LTS?10:38
wgrantjml: Yes.10:38
jmlthat'd be nice10:39
wgrantWe have a Lucid backport already, I believe.10:39
wgrantOK, down to gratuitous difflib formatting changes plus XML-RPC rewrite.10:39
wgrantYay10:39
wgrantdanilos: Does the zero-width space fix the slight underline you get on hover in WebKit?10:40
jmllifeless: does fixtures have a way of combining two fixtures?11:14
jamwgrant: to add love to this investigation, if I do 2 api requests back-to-back in the same process, the first takes 700ms, the second 200ms. something seems really wonky about ssl.sslwrap11:23
jammaybe it caches host identities or something11:23
nigelboh yay, is loggerhead down?11:26
* nigelb moves to the correct chanel11:27
wgrantjam: Hmm.11:33
wgrantjam: It's possibly also doing OCSP or something slow like that.11:33
lifelessjml: yes, useFixture, or the hypothetical graph stuff we've talked about vis-a-vis replacing testresources11:39
lifelessjml: re emeritus - yes, its not setup yet; you didnt have to leave ~launchpad :P.... anyhow, getting it setup means moving stuff that references launchpad directly to reference role teams, and I don't know how long the piece of string is yet.11:40
=== henninge is now known as henninge-lunch
=== almaisan-away is now known as al-maisan
bigjoolswoo, timeouts creating an MP11:56
wgrantbigjools: The branch scanner is slow, as it has to insert like 70000 branchrevisions.11:59
wgrantThat keeps things locked for a while.11:59
bigjoolswhat did someone just do?11:59
gary_posterpersonally, I was typing.12:00
bigjoolsbadumtish!12:00
gary_poster:-)12:00
bigjoolsStevenK's humour has been rubbing off on gary_poster12:01
gary_posterheh12:01
bigjoolsI say humour in the loosest sense12:01
gary_posteryeh yeah yeah12:01
bigjoolsthat reflects more on him than you though :)12:01
gary_poster:-)12:01
wgrantgary_poster: Hi.12:03
gary_posterhey wgrant.12:03
wgrantgary_poster: Our XMLRPCTestTransport is broken in Python 2.7. getresponse now takes a buffering kwarg, and it's not trivially obvious to me how to make our stuff work with 2.7. I noticed that there's zope.app.testing.xmlrpc.ZopeTestTransport which does a similar thing in a different, 2.7-compatible way. But compared to ours, it doesn't send a Host header or preserve the original interaction.12:06
wgrantI presuuuume we probably want to rebase our stuff around ZopeTestTransport.12:06
wgrantBut was wondering if you had any ideas.12:06
wgrantGiven your Zopey foundationsness.12:06
gary_posterheh12:06
gary_posterwgrant, in the abstract, I no longer feel that there's a huge win to trying to coordinate with the Zope community, given its ever-vanishing existence/relevance.  Maybe that's unfair.12:13
gary_posterIn any case, I'm in favor of a relatively attractive path of least resistance, whatever that is.  If it would be relatively easy to make the Zope version do what we want, benji and I still have commit and release privs.12:13
gary_posterThat could potentially get us in one of the "update the Zope world" situations, but if there has been as little activity there as I believe, that might be a non-problem.12:14
gary_posterI'm afraid I don't have anything to offer other than that ATM, though if you'd like me to dig in a bit to understand the problem you describe and have a more concrete opinion on a way forward, I'm happy to.12:14
wgrantI'm happy to dig myself, was just wondering if you had any immediate insights.12:15
gary_poster'fraid not. :-/12:15
wgrantThanks anyway.12:17
gary_posternp12:17
wgrantShouldn't be too hard to sort something out.12:17
gary_posteragreed12:17
wgrantbigjools: :( that needs to be advertised everywhere12:22
bigjoolswgrant: ?12:22
wgrantbigjools: I'd not seen that matcher before, and it seems useful in an awful lot of tests.12:23
bigjoolsyes, it freakin rocks12:23
bigjoolsseriously, read the matcher docs12:23
daniloswgrant, yes (regarding zero-width space)12:34
wgrantdanilos: Great!12:35
jkakarbigjools: +1000. :)12:38
jkakartesttools in general is super awesome.12:38
=== henninge-lunch is now known as henninge
=== Ursinha is now known as Ursinha-bbl
deryckMorning, everyone.13:07
=== matsubara-afk is now known as matsubara
jtvhi deryck13:15
bigjoolsjkakar: yes!13:35
=== mbarnett` is now known as mbarnett
jtvgmb: could I trouble you for review of an oversized refactoring?  https://code.launchpad.net/~jtv/launchpad/pre-788078/+merge/6837413:38
gary_posterHey deryck.  You know I'm excited, so please just forgive me :-) but where is the thunderdome JS test stuff?  Have you started actually writing tests with it, and are you reasonably pleased with it?  (I am reminded because I'm writing some JS in a module w/out tests)13:47
deryckgary_poster, I have sadly not got very far with it.  I had one other remaining thunderdome bit which has taken all my attention....13:48
deryckgary_poster, a branch for fixing bug heat.13:48
derycksorry :(13:48
gary_posterderyck, oh :-( .  understood.  I am about to finish a bugfix branch myself, so maybe I'll race you for the last bit? :-)13:48
gary_posterderyck, if you really want to do it, that's ok.  I got to play on it quite a bit13:49
deryckgary_poster, ok :)  I hope to either resolve the heat work or turn to the js stuff while waiting on more info on the heat work today. But I'll see how it goes.13:50
gary_poster:-) k cool13:50
deryckgary_poster, but I'll keep you better informed on my progress, too. :)13:50
gary_poster:-) thanks for humoring me13:50
derycknp!13:52
henningeHow do I get the dev server to not use launchpad.js but the single js files?13:58
gary_posterhenninge, afaik there is no way.  when people talk about that stuff they are talking about running YUI tests.  If you discover differently, I would *love* to know too :-)13:59
wgrantThat ability was removed once we stopped using the bundled up YUI thing.14:00
wgrantBecause it ended up making 400 requests to the local appserver on each page load, which took a couple of minutes.14:00
gary_posteroof14:00
henningeAh, but my memory was not wrong.14:00
wgranthenninge: I know 'make jsbuild' isn't perfect, but it should be pretty fast.14:00
wgrantNo need for appserver restarts or anything.14:00
wgrantPlus you should be doing TDD ;)14:00
henningewgrant: no, it's about debugging the js file in the browser14:01
henningelots of scrolling and searching in launchpad.js14:01
wgranthenninge: The Launchpad bits should be unminified.14:01
wgrantBut yeah.14:01
gary_postermy concern with it is that neither chrome debug tools nor firebug handles the large file well14:01
wgranthenninge: You can't debug in the testrunner?14:01
gary_postertrying to set breakpoints sucks in that case14:01
henningehm, could try that14:02
gary_posterwgrant we have lots of legacy JS code without tests14:02
wgrantgary_poster: Indeed :(14:02
gary_posterhenninge both firebug and chrome debug let you search for a string14:02
gary_posterthat's what I use14:02
gary_posterI find navigation not too bad14:02
henningeme, too14:03
gary_posterthe debugger bugs are the killer14:03
gary_posterfor me14:03
henningewould be cool if the file had comments with the originial file name whereever that file starts14:04
henningewgrant: where is the code that creates launchpad.js?14:04
wgrantlp.scripts.utilities.js.jsmin14:05
henningethanks14:05
wgrantWell.14:05
wgrantThat minifies it.14:05
wgrantBut the input to that is just cat run over all the files.14:05
henningeand the combining?14:06
wgrantcat :)14:06
henningeah14:06
henninge;)14:06
wgrantThat's all in Makefile.14:06
wgrantrm lib/canonical/launchpad/icing/build/launchpad.js; make jsbuild14:07
wgrantWatch your screen and sob.14:07
gmbjtv: Sure, I'll take a look at that branch for you.14:07
jtvgreat, thanks14:07
* henninge sobs14:09
* StevenK sees that henninge is following wgrant's instructions14:10
henningeyup, always14:10
=== al-maisan is now known as almaisan-away
cjwatsonwgrant: whee, thanks14:55
cjwatson(there's still multiarch-translations proper, but need to wait for a lucid-updates landing and sort out PPA bits first)14:56
=== salgado is now known as salgado-lunch
=== jtv is now known as jtv-afk
gmbjtv-afk: r=me on the kilobranch. Nice work.15:12
nigelbHi, how do I update my branch? My devel branch is getting a rocketfuel-get.15:23
nigelb*my feature branch15:24
jmlnigelb: you merge devel (or stable) into it15:47
nigelbok, doing that now.15:47
=== salgado-lunch is now known as salgado
jmlnigelb: you need to commit after merging15:54
jmlnigelb: pretty important that you do that.15:54
nigelbjml: oh, ok!15:55
nigelbdid windmill get nuked?15:55
jmlnigelb: yes.15:55
jmlnigelb: at the rally15:55
nigelbthat explains the conflict.15:56
=== Ursinha-bbl is now known as Ursinha
nigelbwhere is the ideal place to put a utility function (sorting)16:05
mhall119quick question, can I query the launchpad api using a user SSO identity URL yet, or is it still just username?16:06
jmlnigelb: lp.services.utils16:21
jmlnigelb: tests in lp.services.tests.test_utils16:21
nigelbjml: thanks again. Where is everyone today? :)16:22
jmlnigelb: no idea.16:22
bigjoolsjml: "byEquality" - it just gets betterer and betterer :)16:23
jmlbigjools: heh16:23
bigjoolsnigelb: I guess we must be busy hacking or something16:23
bigjoolsjml: have you got anything to do with Murdoch taking a pie to the face?16:24
jmlbigjools: sadly no.16:24
nigelblol16:24
nigelbI wonder if I should agree to take a pie to the face.16:25
nigelbFor fixing all critical bugs before uds-p16:25
* jml has a side bet for UDS-P. It involves drinking.16:25
nigelbjml: In that case I'll hear that one out :D16:26
nigelbUDS-P should of course be written as UDS :-P16:26
jml:)16:27
nigelber, I'd like some help.16:31
nigelbHow would I generalize the sort in this http://paste.ubuntu.com/647431/16:33
nigelbI want to avoid the bit where I'm doing the lower() and move it to a common function.16:33
nigelbBut I'm not sure how to make it generic enough to be used everywhere16:33
jmlnigelb: I don't think it's worth generalizing, tbh.16:34
jmlnigelb: hmm.16:34
jmlIf you had to...16:34
nigelbjml: That's what I've been told to do. https://code.launchpad.net/~nigelbabu/launchpad/spec-sub-sort/+merge/6331516:34
jmldef case_insensitive_sort(xs):16:34
jml    return sorted(xs, key=lambda x: x.lower())16:35
nigelbSo, the only bit I'd avoid would be the lower() thingy.16:36
jmlwell, yeah16:36
nigelbI guess jtv could help throw some insight on whether that's enough. He helped with the initial review.16:37
nigelbjtv: ping16:37
jmlnigelb: I don't have the willpower to read through that whole discussion...16:38
jmlnigelb: you could also try this:16:38
jmldef sort_for_display(things_to_display):16:38
nigelbjml: Now you know why it took me this long to do this :)16:38
jml  return sorted(things_to_display, key=lambda x: x.displayname.lower())16:39
jmlor even16:39
jmldef sort_for_display(things_to_display, display_attribute='displayname'):16:39
jml  return sorted(things_to_display, key=lambda x: getattr(x, display_attribute).lower())16:39
nigelbok, ^ sounds like what's probably generic enough.16:40
jmlnigelb: yeah, you've got some fairly blocky reviews16:40
nigelbIt was hard to get out of that pit of demotivation I fell into after this.16:41
jmlnigelb: well, basically, the thing is that as long as there's a *name* for what you are trying to do, the exact level of abstraction doesn't matter.16:41
jmlbecause it'll probably change as we get it better anyway16:41
jml(and the name in this case is the function)16:41
jmlnigelb: *nod* I know what you mean16:42
nigelbjml: I was trying to use this function in different use cases and I couldn't abstract it will.16:42
jmlnigelb: ahh, ok.16:42
nigelb*well16:43
nigelbjml: I had to take some time off Lp dev before I could get myself back :)16:44
* nigelb is now back :D16:44
nigelbjml: in that above function, should x also be a parameter to the function?16:46
nigelb*shouldn't16:46
jmlnigelb: no. it's the argument to the key function.16:46
jmlnigelb: please bear in mind that I typed it out into my IRC client, and thus am missing my usual crutches and so I've probably made a NameError or SyntaxError16:47
nigelbI'm testing it on python shell first16:47
jmlgood idea16:48
=== cr3_ is now known as cr3
=== gmb changed the topic of #launchpad-dev to: Performance Tuesday | https://dev.launchpad.net/ | On call reviewer: - | Critical bugs: 238 - 0:[######=_]:256
nigelbgmb: are you EOD-ing or was the topic stale?17:09
nigelbOuch, my launchpad.dev isn't working. Is anyone around to help me figure out how to get it working?17:11
nigelbI guess I'll be back tomorrow.17:13
=== beuno is now known as beuno-lunch
derycksinzui, ping18:18
sinzuideryck, otp18:19
derycksinzui, ok, just a question about a bug when you have a moment.18:20
=== beuno-lunch is now known as beuno
sinzuihi deryck19:25
deryckhi sinzui.  Was only wanting to ask you about bug 812790 since you guys are working on stuff related to people, pickers, etc.19:26
_mup_Bug #812790: strange "no such person" popup on user-relative branch url <Launchpad itself:Triaged> < https://launchpad.net/bugs/812790 >19:26
derycksinzui, not sure if that bug is at all related, but it seems fixed on qastaging.  So wondering if the bug should be closed?19:26
deryckor I can't repro on qastaging anyway.19:26
sinzuiWe have landed a few fixes in the last 24 hours19:27
sinzuiI do not see anything that specifically fixes it though19:28
derycksinzui, ok, thanks.  I'll leave as is then until we have more info.19:29
lifelessmorning19:57
jelmerhi lifeless20:01
lifelesshi jelmer20:05
lifelessgary_poster: hi20:05
gary_posterhey lifeless20:05
lifelessgary_poster: that UNION -> OR may be performance sensitive; am just reviewing.20:05
gary_poster(on call(20:05
lifelessyah, no worries, just giving you a headsup20:06
gary_posterlifeless, wondered about that bu seemed unlikely.  I can just revert that part.  k20:06
gary_posterlifeless, that user_ids_affected_with_dupes code (UNION -> OR) has never been changed since it was originally written.  Tempted to revert anyway.  Thoughts?20:21
lifelessgary_poster: reverting would be conservative and make qa easier20:22
lifelessgary_poster: thats about it :)20:22
gary_posterlifeless, k, agree, will do, thx.20:22
gary_posterlifeless, "in the client side js, the limit of checking whether the task was20:24
gary_posternew seems unnecessary - isn't it reasonable to just cross check20:24
gary_posterregardless?" :20:24
gary_posterSince we don't do this generically (yet) I thought it would be surprising to update other tasks in response to saying "me too" because people might confuse coincidence with consequence.  I'd be fine if we always kept statuses up-to-date with long-poll later, for instance; then we would not need this "something happened" code path at all.20:24
gary_posterFor now, though, I stand by my call there.20:24
lifelessits up to you20:25
lifelessthere are a couple of bugs I can point you at :)20:25
lifelesswhere we suffer inflight collisions [particularly on the non-ajax forms] due to stale statuses20:25
gary_postersure20:25
gary_posterbut that's not what this is about20:25
lifelessI do see your point20:25
lifelessuhm20:25
lifelessperhaps huw can suggest some way [in the future] for us to show 'updated by you' and 'updated by $other'20:26
lifelessgary_poster: e.g. standing by your call is fine.20:26
gary_postercool lifeless, thanks20:26
lifelessgary_poster: it was only a thought in the first place; perhaps I wasn't clear enough about that.20:27
gary_posterlifeless, sure.  if you didn't want us to respond to them, though, why express them? ;-)20:27
lifelesswell, I do like responses :)20:28
lifelessI suspect I should eat before I fail-at-communication-more20:28
gary_posterwell, and they are good thoughts that merit a response.  so...20:28
gary_posterlol20:28
lifelessI don't know entirely what I meant from 'it was only' .. on.20:30
lifeless-> food20:30
gary_poster:-)20:30
lifelesswgrant: lxc in oneiric may be better, but check out bug 80298520:44
_mup_Bug #802985: [lucid] /var/lib/dpkg/tmp.ci/preinst: 399: arithmetic expression: expecting EOF: "3.0-0-generic" <lucid> <patch> <debootstrap (Ubuntu):Triaged> <eglibc (Ubuntu):Invalid> <debootstrap (Ubuntu Lucid):Invalid> <eglibc (Ubuntu Lucid):Triaged> <debootstrap (Ubuntu Maverick):Invalid> <eglibc (Ubuntu Maverick):Triaged> <debootstrap (Ubuntu Natty):Invalid> <eglibc (Ubuntu Natty):Triaged> <debootstrap (Ubuntu Hardy):Invalid> <eglibc (Ubunt20:44
gary_posterlifeless, do you recommend bothering with lxc for LP right now?  I've been considering it, but the page you and William have maintained makes it look pretty...fragile.20:45
lifelessgary_poster: I have a few thoughts20:45
lifelessI don't think that right now the incremental learning for the team is all that great from someone just running a single test suite under it20:45
gary_poster(just a vm is my other option, and what I have do noe)20:45
gary_posternow20:45
lifelessI think its good to be familiar with lxc though, because I think its got legs20:46
gary_postersure20:46
gary_posterI hope so20:46
lifelessand we're likely to want to implement parallel testing on top of it20:46
gary_postermakes sense20:46
lifelessthe page does cover a bunch of detail today, but automation is improving, especially if using oneiric or the lxc ppa20:46
gary_posterwas going to try oneiric, but I've spent my OS configuration budget for the month.  August, maybe. :-)20:47
lifelessheh, i hear you!20:48
lifelessI've just made a KVM oneiric instance20:48
lifelessto kick the tires on lxc and ensemble a bit20:48
gary_postercool, yeah, I made a vm too20:49
gary_posterbut lxc on a vm seemed a curiosity and a good way to explore lxc, more than a practical approach. :-)20:49
gary_posteralthough it would still give some benefits, I suppose20:50
lifelessso lxc is a moving target20:50
lifelessthe container rules and capabilities are changing fast enough that 6 months can easily get bug fixes relevant to us.20:51
gary_posteryeah, makes sense20:52
lifelessanyhow, if you want to do any of the following, lxc +1:20:52
lifeless - parallel test20:52
lifeless - test with different / new dependency stacks20:52
lifeless - run more than one lp up at a time [e.g. to play with federation]20:52
lifelessor if you have any memory pressure issues on your machine (e.g. during bin/test)20:53
lifelessit shouldn't be too bumpy a ride - it will either be easy, or totally impossible :>20:53
gary_posterlol20:53
gary_posterok cool20:53
gary_postermight give it a try soon then20:53
gary_posterthanks lifeless :-)20:53
lifelessnp :)20:54
lifelesswallyworld: wow, early up?20:57
lifelessjelmer: hi21:04
lifelessjelmer: why do you think bug 628323 is a subunit package bug ?21:04
jelmerlifeless: see max's comment21:06
lifelessI don't have that21:06
lifelesswhat did he sya ?21:07
maxbsomeone talking about me? :-)21:07
jelmerlifeless: "This FTBFS is the result of the python-subunit package not being21:07
jelmerproperly built for all current Python versions"21:07
lifelessI don't follow.21:07
lifelessdoes it need a no change rebuild because the set of supported python packages has finally changed in debian ?21:08
lifelesss/packages/versions/21:08
maxbIs that bug number wrong, or private?21:08
lifelessits in Debian21:08
jelmerlifeless: it has already had the no-change rebuild, I'vconfirmed 0.0.6-2 was only built for 2.6, 0.0.6-3 was built for 2.6 and 2.721:13
lifelessI'm not sure what to conclude from that :)21:14
lifelessI have to pop out to see the midwife, be back in <= 90m21:14
jelmeraccording to the logs the bug was filed with 0.0.6-2 installed21:14
jelmerso I'm going to indicate that in the bts21:14
jelmerlifeless: ok. I hope all is well21:15
lifelessjust routine meeting21:15
wgrantlifeless: Hah, nice.21:22
wgrantlifeless: Oddly, lxc debootstrapped it fine for me.21:22
=== matsubara is now known as matsubara-afk
wgrantwallyworld: Hi21:34
wallyworldhi21:34
wallyworldlifeless: sorry, didn't see your ping. irc notifications in unity aren't as good as i'd like so i often miss them21:35
wallyworldwgrant: what can i do you for?21:36
wgrantwallyworld: Was just going to tell you that the "true"/"false" thing was a real bug, but I see you've fixed it already.21:36
wgrant(not sure why it can't be in the main JSON dict, though)21:37
wgrantNice simple test, too.21:37
wallyworldwgrant: yes. i reverted that tales snippet. it's all a mess. today i plan to do some more yak shaving. there's 2 ways of getting a picker (via create or addpickerpatcher) and you get different results depending on which one is used :-(21:38
wgrantwallyworld: Yeah.21:38
wgrantStill, we are getting better.21:38
wgrantSlowly.21:38
wallyworldplus the config vs json_config issue21:38
wallyworldat first i didn't see why there was the need to hack in those booleans in the tales21:39
wallyworldall pretty ugly21:39
wgrantBut fixable :)21:39
wallyworldi added 2 tests to the yui test case - one for the textfield plugin and one for the extra message. the textfield plugin refactoring from the tales was good because it enabled the yui test to check the proper code path21:41
wallyworldwhereas before the test wired it up manually21:41
wgranti only see one new test, but I guess you adapted an existing one to test the whole thing?21:42
wallyworldwgrant: yes - i modified an existing test and added some extra setup guff to support it21:43
wgrantThe best kind of new test.21:43
wallyworldthe modifed test is the one i was referring to above when i said it's now a better test21:43
wgrantYup21:43
LPCIBotProject devel build #900: FAILURE in 5 hr 33 min: https://lpci.wedontsleep.org/job/devel/900/22:12
jelmerI'm getting exceptions starting RabbitMQ during the testsuite; is that a common issue?22:30
wgrantjelmer: Shouldn't be any more. Is your hostname configured correctly?22:31
jelmer'hostname -f' returns 'localhost6', so I guess not..22:31
wgrantThat could be a problem.22:32
wgrantIt should be mostly immune to that now, but maybe IPv6 combined with dodgy hostname is doing bad things.22:32
jelmerhmm, that doesn't seem to work either22:42
jelmerI'll have a look tomorrow22:42
wgrantsinzui: We can hear you and each other.23:04
wgrantCan you hear us?23:04
sinzuino.23:04
wgrantThat is suboptimal.23:04
poolieit would be cool if launchpad could show a per-pillar view like this for the hottest bugs: http://mail.google.com/support/bin/static.py?page=suggestions.cs23:14
wgrantlifeless: Hi23:36
lifelesshi23:37
wgrantlifeless: The malone.advanced-subscriptions.enabled, malone.advanced-structural-subscriptions.enabled, code.recipes_enabled flags are on by default and have been removed from the code. Can we remove the rules?23:40
lifelessof course23:40
wgrantThanks.23:40
pooliehi wgrant, lifeless23:45

Generated by irclog2html.py 2.7 by Marius Gedminas - find it at mg.pov.lt!