[00:11] james_w: ah [00:11] james_w: does it give you a hint of some sort? [00:11] james_w: yes, -foundations [00:11] lifeless: well, it says the nonce was used before [00:12] yes [00:12] I don't know if that indicates a bug in your code, in launchpadlib, or in the lp code. [00:13] my code?!? [00:13] never! [00:13] its quite possibly lp itself [00:14] one user with many api clients at once isn't a common occurence today [00:14] That shouldn't cause nonce reuse... [00:14] without checking the implementation, who knows. [00:15] wgrant: do you know how the nonces are generated and stored? [00:15] OAuthNonce table [00:15] LP stores them in the OAuthNonce table. [00:15] Yeah. [00:15] I wonder who generates them. [00:15] second highest oops yesterday I think [00:15] are they an 8 bit number and stored forever, or are we talking heat death of the universe before this should happen? [00:15] not quite, bit lower: [00:15] 8 / 112 RootObject:+login [00:16] oauthnonce__access_token__request_timestamp__nonce__key" UNIQUE, btree (access_token, request_timestamp, nonce) [00:16] the nonce itself isn't constrained to be unique. [00:17] Right, the timestamp has to match. [00:17] So a collision is less likely. [00:18] wgrant: python-oauth by the look of it [00:18] nonces/timestamps are only SHOULD IIRC ;-) [00:18] def generate_nonce(length=8): """Generate pseudorandom number.""" return ''.join([str(random.randint(0, 9)) for i in range(length)]) [00:18] Yeah. [00:19] 10^8 space per second? [00:20] Looks like it. [00:20] I think this is the error you get if you don't re-sign on redirect, but I don't know why these requests would have been redirected [00:21] What are the requests? [00:21] this one was sp.getBranch(pocket="Something") [00:22] if you take n samples randomly from N options, you have about 1 - math.exp(-n*(n-1)/2.0*N) chance of getting a collision [00:22] Hmm. [00:23] 1 - math.exp(-n*(n-1)/(2.0*N)) ? [00:23] um [00:23] or 1 - math.exp((-n*(n-1))/(2.0*N)) [00:24] james_w: well those last two are the same, right? but they're both better than what i said, indeed [00:24] lifeless: I think having our chat after lunch would be best for limiting the disruptions [00:24] yeah [00:24] bug 645640 [00:24] <_mup_> Bug #645640: API gives 401 errors on occasion [00:25] sure [00:25] i did this math for http://bugs.python.org/issue6598 fwiw :) [00:25] how many nonces do we generate in a second? [00:26] mwhudson: I don't know, but 100-1000 probably [00:27] assuming they are per-access-token too [00:27] in this case I got three 401 in 2 minutes, so I don't think it's a collision [00:27] RootObject:+login 204961 [00:28] 2-3 [00:28] james_w: yes, that seems pretty unlikely [00:28] unless its per-request [00:28] lifeless: my comment on reliability was more about tuning than large changes, so while I agree that using Launchpad's facilities makes sense, that is a fair amount of work, and doesn't buy us much in the fine-tuning department. [00:29] james_w: fair enough; my main goal is to get the system 'owned' by someone with it as part of their work portfolio [00:29] lifeless: so I was thinking of things such as better heuristics for transient failures, better automatic categorisation of problems, automated linking to bugs [00:29] james_w: you do great, but its hardly your primary thing [00:29] lifeless: and I'm certainly down with that [00:29] lifeless: What does +login have to do with OAuth? [00:30] james_w: OOPS-1725D1955 [00:31] grah the pageid on that is sad [00:32] won't load for me [00:32] james_w: if you only need readonly access you could be anonymous [00:32] and avoid the issues [00:32] james_w: https://lp-oops.canonical.com/oops.py/?oopsid=1725D1955 works for me [00:33] poolie: this is read-write on occaision [00:33] poolie: anonymous may not be what you think it is ;) [00:33] lifeless: does now [00:33] poolie: I'm fairly sure it allocates a token on the fly [00:33] poolie: which is gross in some ways [00:33] lifeless: i think (imbw) it's only inside the app server, not in http [00:33] that's probably me [00:33] i agree it is a bit gross [00:34] poolie: its in the appserver, but thats where this problem may be too :) [00:35] OAuthAccessToken.sourcepackagename? [00:36] https://bugs.edge.launchpad.net/launchpad-foundations/+bug/645651 about the oops itself [00:36] <_mup_> Bug #645651: pageids on nonce failures are a bit bogus [00:36] james_w: no idea why thats there [00:37] ah, because they can be restricted in scope [00:38] lifeless: i think i'll keep pushing on flags today [00:38] poolie: \o/ [00:38] i meant to give it a one or two day timeslice but it feels like there's a fair momentum/inertia effect [00:38] iow launchpad is heavy :) [00:38] yes [00:38] I wouldn't stress about doing it the right way though. [00:38] oh btw [00:39] Like, practicality of purity. [00:39] if i change my mind about naming something, i can just change it, right? [00:39] james_w: 755156 API requests a day. [00:39] there are no internal api stability rules other than that the whole thing must pass it's tests [00:39] poolie: yes you can but its desirable, if folk are building on something, that they be able to CP it to production. [00:39] (i don't mean gratuituously; i realize it may break outstanding branches etc) [00:39] right [00:39] poolie: so do consider leaving a forwarder behind. [00:40] sure [00:40] you odn't have to [00:40] some of the naming about features vs flags etc is a bit inconsistent [00:40] yeah [00:40] that's one nice thing compared to bzrlib [00:40] so as of yesterday i have a working readonly anonymous view of the rules [00:40] james_w: so, 100 per second or so [00:41] i was wondering about proposing that in parallel with adding access control [00:41] just to be stepwise [00:41] and because there's probably nothing privacy critical at the moment [00:41] poolie: separate review, sure, but I wouldn't land it solo [00:41] k [00:41] james_w: Scope restriction doesn't actually work, unfortunately. [00:41] It would be really nice if it did. [00:41] What with the rather damaging APIs that are around now... [00:41] Like, say, syncSource into the primary archive. [00:43] james_w: oauthnonce || 79.96 tuples/sec [00:43] wgrant: did you get those buildd-manager logs you were after? [00:43] mwhudson: I didn't. [00:44] wgrant: can you tell me what to look for? [00:46] mwhudson: a startBuild(http://.buildd:8221/, firefox, 3.6.10+build1+nobinonly-0ubuntu3 [00:46] I think. [00:46] I may have the buildd bit wrong. [00:46] But it will mention adare or ross in there somewhere. [00:47] wgrant: any idea when? [00:48] date_first_dispatched [00:48] 2010-09-22 07:48:40.575908+00:00 [00:48] ah, there we go [00:48] mwh@devpad:/srv/launchpad.net-logs/production/cesium$ grep startBuild.*adare buildd-manager.log | grep firefox [00:48] 2010-09-22 07:48:40+0000 [-] startBuild(http://adare.buildd:8221/, firefox, 3.6.10+build1+nobinonly-0ubuntu3, Release) [00:48] 2010-09-22 07:53:13+0000 [-] startBuild(http://adare.buildd:8221/, firefox, 3.6.10+build1+nobinonly-0ubuntu3, Release) [00:48] 2010-09-22 07:55:09+0000 [-] startBuild(http://adare.buildd:8221/, firefox, 3.6.10+build1+nobinonly-0ubuntu3, Release) [00:48] 2010-09-22 13:46:15+0000 [-] startBuild(http://adare.buildd:8221/, firefox, 3.6.10+build1+nobinonly-0ubuntu3, Release) [00:48] What happens immediately after the first startBuild? [00:49] (the 13:46 one is from when Julian retried it last night -- we presumed it was transient) [00:49] looks normal [00:49] I wonder why it's retried 5 minutes later, then. [00:49] http://pastebin.ubuntu.com/498821/ [00:49] 2010-09-22 07:52:19+0000 [-] communication failed (User timeout caused connection failure.) [00:49] 2010-09-22 07:52:19+0000 [-] failure (None) [00:50] Gah. [00:50] Non-virt builders *do not do that*. [00:50] Ah. And I bet that retrying the build didn't reset the failure count. [00:51] So it was immediately killed the next time. [00:51] actually [00:51] http://pastebin.ubuntu.com/498823/ [00:51] that dispatching line at the bottom is normal? [00:51] I believe so. [00:51] ah haha [00:53] wgrant: http://pastebin.ubuntu.com/498826/ [00:53] That's more like it. [00:54] * thumper goes to find lunc [00:54] lunch that is [00:54] However, that wasn't terminal, since the builder and job failure counts are the same. [00:54] adare seems to be timing out quite a lot [00:54] Hmm. [00:54] not like every build [01:04] wgrant: http://people.canonical.com/~mwh/ppc-buildd-fail.txt [01:05] mwhudson: Are builders from other archs so bad? [01:07] wgrant: hard to say, it's pretty bad though [01:09] wgrant: yuck: http://pastebin.ubuntu.com/498830/ [01:10] (adare and ross are the worst though) [01:11] wgrant: ppa builders, for contrast: http://pastebin.ubuntu.com/498831/ [01:15] Hm, quite a difference. [01:15] still seems like a whole lotta time out for one day's log [01:38] adare looks reasonably dead now :/ [01:39] And ross just started building something that has been attempted a couple of times befor.e [01:39] I wonder what's going on. [01:52] hi wgrant, mwhudson [01:53] StevenK: what happened to https://code.edge.launchpad.net/~mbp/launchpad/mbp-trivial/+merge/32173 ? [01:53] Morning poolie. [02:04] wow, lp certainly has a big backlog of approved non-landed mps [02:06] They're mostly/all from people with PQM privs, though. [02:07] Hm. Half are more than a week old. [02:08] lifeless: ping [02:14] thumper: sec [02:14] np [02:21] thumper: skype? [02:21] lifeless: just a few secs [02:23] lifeless: yep [02:28] https://lp-oops.canonical.com/oops.py/?oopsid=1723XMLP310 [02:44] OOPS-1727EA92 [02:44] That same filename works fine as an attachment on launchpad.dev [02:44] But not on edge. [02:46] (yes, yes, I do name files strangely to try to break LP) [02:59] \o/ bugtask:+index landed. [02:59] * lifeless dances the happy happy dance. [02:59] lifeless: What did you get it down to? [02:59] wgrant: what I posted to the list [03:00] wgrant: just had various test fallout to fix - things I had missed subtlties of [03:00] Ah, right. [03:00] Yep. [03:00] lifeless: At what stage did the OOPS I referenced about occur? [03:01] The initial traversal, or some librarian trickery? [03:01] please wait, your enquiry is important to us [03:02] do doo doo, do do do-doo [03:02] I bet sodium has been exposed to water again [03:02] Heh. [03:02] Sounds like it needs replacing. [03:02] its been replaced. [03:03] ... [03:03] the problem is possibly the fs [03:03] Well then. [03:03] or similar [03:03] it does not like that oops [03:03] will try again ... later [03:03] k [03:03] Thanks for trying. [03:16] thumper: mwhudson: I need your rubber stamps, again. [03:16] lifeless: otp [03:17] lifeless: Where to now? 11565? [03:17] https://code.edge.launchpad.net/~lifeless/launchpad/cp/+merge/36406 when you get a chance; just code-ok :) [03:17] wgrant: yes [03:17] Excellent. [03:27] spm: if you're not flat chat, could you drop http://pastebin.com/DZcgFsA1 onto staging as a cowboy? want to see if its going to blow up tonight so I can start on it now, if needed. [03:27] sure, gimme a bit tho, chasing logs for beuno atm [03:27] ta [03:30] evil: http://samy.pl/evercookie/ [03:33] for some reason i feel like viewing that url with netcat [03:35] lifeless: you ain't kidding. that's horrid. [03:35] * jcsackett ponders. [03:35] i believe i heard about that as being by the same person who came up with the twitter xss hack. [03:36] security is damn hard :) [03:39] ain't that the truth. [03:39] at least this evercookie is open; someone can develop a plugin to kill it. [03:41] jcsackett, is that the cookie that uses the HTML5 database to store itself? [03:42] mars: yeah. and uses png cookies, and every other storage mechanism. [03:42] heh, nasty [03:42] link was posted above by lifeless; it's a pretty freaky thing. though tech wise, it *is* sort of impressive. [03:43] yeah, I'm not clicking that link. I stopped clicking when the 'Gmail contacts stealing' example page was published [03:44] although I suppose I could use a Chrome incognito browser to look at it [03:44] Unless it uses Flash cookies too. [03:44] :/ [03:44] double yuck [03:44] the png cookies are especially beautiful [03:45] Indeed. [03:46] The Web really needs to just die. [03:50] ah, here it is, the recent debacle around ad agencies starting to use these nasty techniques: http://arstechnica.com/apple/news/2010/09/rldguid-tracking-cookies-in-safari-database-form.ars [03:52] mwhudson: so, can has stamp ? [03:52] thumper: ping [03:52] lifeless: yus? [03:52] please rc stsamp https://code.edge.launchpad.net/~lifeless/launchpad/cp/+merge/36406 [03:53] lifeless: still otp [03:53] mwhudson: sorry, thought you weren't from the netcat comment. It doesn't actually a review, only a 'review'. [03:54] mars: perhaps you could ? [03:54] looking [03:55] lifeless, does the report give you that output right now? Or did you construct it yourself? [03:55] mars: myself from the report [03:56] lifeless, what am I supposed to do with this? rubber-stamp it? [03:56] mars: yes, its just so that ec2land will work [03:56] ok [03:57] lifeless, done [03:57] thanks! [03:58] lifeless, we can add that data to the text report later as well. I already publish most of that info in the HTML report. [03:58] mars: it shouldn't go in the txt report [03:58] storing cookies in etags is an impressively evil idea [04:00] what we should do though is start showing more info about the other commits [04:02] if only we could figure out how to make the vaccines for these internet diseases spread as quickly as the infections themselves! :) === Ursinha is now known as Ursinha-afk [04:03] it would make an awesome doctoral thesis topic [04:04] spm: taptap :) [04:06] still busy :-) [04:07] you might want to grab the pastebin before it times out ;) [04:10] * beuno starts talking more loudly than lifeless [04:10] beuno: you could do these things during your workday :P [04:11] lifeless, I remember the days when I could... [04:11] they where great and people didn't yell at me during a movie [04:11] lifeless: if you'd put your paste into a WEBSCALE pastebin, like, eg sqlite backed, timeout wuolnd't be a problem. that is all. [04:12] :-) [04:33] lifeless, poolie, here is the new feature flags fixture in action: http://pastebin.ubuntu.com/498904/ [04:39] mars, sweet! [04:39] is that already in devel? [04:39] jdEC2 [04:39] poolie, not yet, the branch is here: lp:~mars/launchpad/add-profiling-feature-flag [04:39] also, i guess in non doctest code you'd use self.useFixture(FeatureFixture...) [04:39] needs landing polish [04:39] irssi FAIL [04:40] hi everyone! [04:40] poolie, yes, and it is very nice. with statements also work. [04:40] hello beuno; i just got a survey from LAN the other day saying "where would you like to go in South America" and I thought of you :) [04:41] poolie, I would love to repeat our mini-sprint [04:41] or just skip the sprints and do all meals! [04:41] heh [04:41] i wish code branches easily showed you the submit diff, even before the mp exists [04:41] I think everything should just have a WIP MP. [04:41] fileabug [04:41] mars, so how flags work out for you? [04:42] wgrant, that'd be another way to do it [04:42] lifeless: i'm pretty sure there is already one [04:42] mars i think we need more systematic naming of flags [04:42] i don't want to sound bureaucratic here but i just think it will help when we get more of them [04:42] poolie, I think they will work well. It will be nice to turn the request_profiling feature on and off from the UI and also control access by scope [04:43] maybe yours should be something like webapp.profiling.enabled [04:43] poolie, yes, I pondered 'services.profiling' [04:43] or that [04:43] or services.profile.enabled [04:43] exactly [04:43] matching the python module name is good, i think [04:43] ugh [04:43] what does FF do? [04:44] we have many terrible python names [04:44] FF? [04:44] foo.bar.enabled: True ? [04:44] firefox [04:44] so I wouldn't match them. [04:44] if theres a good name, use it. [04:44] I thought about how it would look in a UI [04:44] but often a flag will be interpreted in many modules. [04:44] Feature: request_profiling - ON [04:44] so for instance [04:44] i think 'memcache' is a poor name [04:45] unless it really is going to turn all of memcache on and off [04:45] it turns all of memcache on and off [04:45] if it's controlling memcache tal stuff more specifically, we should say that [04:45] or vice versa [04:45] yes, that makes sense [04:45] poolie: it turns it on and off for all uses we have of memcache atm. [04:46] tal and API [04:46] yes, that too [04:46] so the current use makes sense [04:46] but if you ever get anything more fine-grained, then maybe some naming could make the flag's function more obvious [04:47] tight [04:47] bah [04:47] right [04:48] mars that branch looks nice [04:48] thanks [04:48] ok, my vision is starting to get blurry - time to sign off. I'll land the FeatureFixture tomorrow. [04:48] Good night! [04:48] you can cc me to the review if you like [04:48] poolie, ok, will do [04:48] one thing, can you mention this in the servicres.features doc string? [04:49] ok [05:17] spm: so.. [05:17] * spm should kick lifeless from the U1 channel.... :-P [05:18] gimme 10, just doccoing what we did and taknig a bio break [05:19] kk [05:23] here's something to try if you want to cry: [05:23] host 'sdasd asdas' ns4.vpweb.com [05:23] (this is the other kind of evil to what lifeless posted earlier) [05:26] Hard / Soft Page ID [05:26] 122 / 338 BugTask:+index [05:26] 91 / 212 CodeImportSchedulerApplication:CodeImportSchedulerAPI [05:26] 48 / 13 Distribution:+search [05:26] 45 / 25 DistributionSourcePackage:+filebug [05:27] 18 / 85 Distribution:+bugtarget-portlet-bugfilters-stats [05:27] 13 / 47 Milestone:+index [05:27] 13 / 40 Distribution:+bugs [05:27] 10 / 11 POFile:+translate [05:27] 9 / 8 Person:+translations [05:27] 8 / 18 DistroSeries:+queue [05:38] lifeless: Is that daily lpnet? [05:38] yes [05:39] Not bad. [05:51] lifeless: patched and restarting .... [05:51] fwiw. '10' in sysadmin speak is typucally out by a factor of 3 when dealing with actual time. fwiw. [05:52] so you're || close to being a black hole? [05:53] StevenK: hi? [05:54] poolie: Hi. You didn't get a mail from ec2? [05:55] nup [05:56] did you? [05:56] I certainly did, the tests failed [05:56] Interesting, the exact same tests failed for my branch that I submitted at roughly the same time :-( [05:57] We are in test fix. Is the lp.poppy.tests.test_poppy.TestPoppy.test_full_source_upload(sftp) test being looked at? [05:58] LayerIsolationError: Test left new live threads: [] <-- suprious? [05:58] spurious even? [06:00] wgrant: bug 1 in the ubuntu context - 179 queries [06:00] <_mup_> Bug #1: Microsoft has a majority market share

< [06:00] bug 1 is such a good example of why we should add task deletion :) [06:00] <_mup_> Bug #1: Microsoft has a majority market share

< [06:06] and why _mup_ should have rate limiting [06:10] lifeless: I recall it being roughly three times that... [06:10] Not bad. [06:11] wgrant: 179 to the point it times out ... :P [06:11] lifeless: got a minute? [06:11] sure [06:12] Ah. === almaisan-away is now known as al-maisan [06:20] * mwhudson goes home [06:27] thumper: ping? [06:27] wallyworld_: otp [06:28] thumper: np. just wanted to touch base. later. [06:29]