[12:04] three strikes! [12:04] jblack: Don't worry too much about it.. my coffee was extremely strong today. [12:05] Anyway.. FOOOD! === kiko-zzz WAVES [12:05] have fun === lohmatii [n=chatzill@ts1-a85.Saratov.dial.rol.ru] has joined #launchpad === mantiena [n=mantas@ctv-217-147-43-171.init.lt] has left #launchpad ["Ieinu"] === lohmatii [n=chatzill@ts1-a85.Saratov.dial.rol.ru] has left #launchpad [] [12:39] How do I make an anchor for a moin wiki doc header, like "== Foobar =="? I want it to keep looking like it does now (i.e. not like a link), but be able to link directly to that section from somewhere else. [12:40] HelpOnMacros should have anchor instructions [12:40] bradb, ^ [12:40] bradb: Are you doing skype or h.323? [12:41] HelpOnLinking was the first doc I tried, of course. [12:41] if its just a table of contents you're after, use the [[TableOfContents] ] macro [12:41] mdke: Nope. [12:41] ok, anchor macro then === bradb tries it [12:42] jblack: skype [12:44] mdke: you rock, thanks [12:44] np [12:46] mpt: around? i've got a UI to show you shortly [01:12] kiko-zzz: around? === bradb heads off === bradb [n=bradb@modemcable033.209-70-69.mc.videotron.ca] has left #launchpad [] === mhz [n=mhz_chil@pc-252-84-215-201.cm.vtr.net] has joined #launchpad === wby87 [i=wby87@dialup-4.158.198.44.Dial1.Chicago1.Level3.net] has joined #launchpad [01:57] re [01:59] hi, i got a question, how long does it usually take for the account registration email to be send? [02:05] few minutes max === mdke_ [n=matt@ubuntu/member/mdke] has joined #launchpad [02:51] ddaa's merges in the pqm queue don't look like they'll succeed ... [02:57] Seveas: ping === stub [i=stub@sweep.bur.st] has joined #launchpad [03:02] stub: at some point, could you check that the sections I added to the production, staging and dogfood configs look okay? [03:02] Merge to devel/launchpad: [r=SteveA] New page layout: CSS columns, an embryonic sitemap, and a vertical nested menu instead of tabs. (r2893: Matthew Paul Thomas) [03:02] (the errordir key in particular) === SloMoSnail [n=slomo@p5487FDF3.dip.t-dialin.net] has joined #launchpad [03:21] jamesh: Sure. I'll be rolling that code out to staging today so we can see it in action. === stub turns his sound on [03:50] when was udu? [03:51] pardon, ubz [03:51] ~ nov 1-14 [03:55] I'm looking at a map of cities within a day's drive.. [03:56] I'm in a very good place. it looks to me like I've got over a 100 million people within a day's drive. [03:57] heh, I've got 16 million in this entire country ;) [03:57] I'm 2 hours from new york, 5 hours from washinton DC, 2 hours from philly, 4 hours from baltimore... [03:57] boston is probably 6 or 7.. [04:00] BjornT: is https://wiki.launchpad.canonical.com/MaloneEmailInterfaceUserDoc up to date? === ajmitch_ has < 1 million within a days drive [04:04] Really? I thought sidney had millions? [04:04] it does [04:05] but it's about 3 & 1/2 hours flying time away [04:09] lifeless: Are you still waiting on a chroot jail for the new PQM box? [04:10] stub: no, its all in my court at the moment [04:11] lifeless: ok. Any rough eta? PQM on breezy/PG8.0 is the next step before we can do staging and production upgrades to breezy/PG8.0 [04:11] ah, did not know there was processes depending on me [04:11] I have to flush a couple of things for bazaar [04:12] before I get back to testing it [04:12] in theory its all ready to go modulo an update to the latest launchpad-bzr [04:13] I'm thinking of only taking one weeks of leave, so ideally pqm could be running sometime next week and we can update staging around Mon 19th if elmo is free [04:13] in theory thats not a problem [04:13] ok. If it slips it doesn't really matter [04:24] Kinnison: Do you have the SQL fragments for adding the new distroarchrelease's for breezy and hoary? They got nuked on staging when I resynced from the production db. [05:02] Launchpad will be going down in 30 minutes time, which will also put the Ubuntu wikis into read only more. Estimated down time is 30 minutes. [05:04] spiv: I'll need to update the librarian too [05:04] stub: Ok. Need anything from me for that? [05:05] spiv: Nah - I think all the keys are currently setup for me so easier if I do it [05:05] Ok. [05:09] lifeless: These tagging instructions I got from you appear to be out of date - they assume 'launchpad/devel' is a branch relative to the PQM user's home directory. [05:09] lifeless: I need to tag r2888 as production 1.41 === pu [n=ydnar@cpe-67-10-187-94.houston.res.rr.com] has joined #launchpad [05:23] And the bzr.integration in rolloouts is still broken :-( [05:24] I can't find one person on the internet with a installed gnomemeeting and a actual microphone [05:24] I'll try integration head [05:32] stub: the launchpad branch is fixed [05:32] p.u.c/~robertc/baz2.0/launchpad [05:32] I have to upload that into sourcecode [05:33] stub: paste the instructions .. [05:33] ok - thanks [05:33] sudo to pqm [05:33] bzr branch -r 2848 launchpad/devel launchpad/production/1.40 [05:33] in the pqm home [05:33] and copy that to the public copy in /home/warthogs/archives/ [05:33] I think it is all fine except the launchpad/devel is now found in the archives directory? [05:33] right, I assumed you'd cd to ~/archives/rocketfuel ;0) [05:34] ok ;) [05:35] Branching now === sabdfl [n=mark@ubuntu/member/pdpc.silver.sabdfl] has joined #launchpad === sabdfl [n=mark@ubuntu/member/pdpc.silver.sabdfl] has left #launchpad [] [06:17] database is up (so authserver is fine), but launchpad and the librarian will be down for at least another hour while I branch and test a fresh bzr [06:19] mmm [06:19] why do you need to do that ? (I'm not sure why you 'install' bzr at all, just run the copy from sourcecode... [06:19] lifeless: I'm branching that p.u.c/bzr/launchpad branch to chinstrap so I can rsync it. Can you make it go faster? [06:20] just rsync it from rookery [06:20] I don't have a rookery account [06:20] or take a copy of sourcecode/* and then do a bzr pull [06:20] yes you do [06:20] rookery - everyone has an account [06:20] I do? [06:22] The reason I need to install bzr is so I can do automated installs, rather than manual and error prone installs, and I need to rewrite that bit of the scripts to run stuff direct from the rsynced bzr branch [06:23] ok [06:23] I was meaning 'why dont the scripts run from the branch as is' [06:23] anyway, you should be able to get bzr in about 3 seconds by rsync [06:27] Mainly because I hadn't realized it was runnable, and doing the install step made it identical to the paramiko, pybaz, config_manager steps. [06:27] oh, so paramiko pybaz and config_manager are all runnable in situ [06:28] do you have the bzr you need ? [06:30] Getting there [06:31] have you rsynced it ? [06:31] rsync -a rookery:/home/robertc/public_html/baz2.0/launchpad === jblack drops his jaw. [06:32] gnomemeeting actually works with us dsl to new zealand dialup [06:34] lifeless: Got that branch, but it is still failing on the revert step [06:35] stub: what branch is it reverting ? [06:35] Ahh... the traceback tells me 'files from a previous merge'. I'll clean things out [06:35] bzr-tree-change [06:35] if that exists remove it [06:38] rsync --recursive --archive --hard-links --compress --delete-after --partial --include=/.bzr --include=/.bzr/** --exclude=* stub@chinstrap.ubuntu.com:/home/warthogs/archives/rocketfuel/launchpad/devel/ /srv/launchpad.net/production/mirror/rocketfuel/launchpad/devel [06:39] Running /srv/launchpad.net/production/bin/bzr revert [06:39] bzr: ERROR: lib/canonical/zeca/ftests/keys/0x046C6D63.get is already versioned [06:39] eeeeeerk [06:39] can I give you a workaround ? [06:39] bzr branch /srv/launchpad.net/production/mirror/rocketfuel/launchpad/devel new-devel [06:39] that will give you a good tree in new-devel [06:40] Mmm... .but sloooow..... [06:40] thats local [06:40] should be fast [06:40] I'll try pulling the prebuilt rocketfuel and 'pull --override' the production branch [06:41] that too should be fast [06:41] it updates quite reliably with cm update ;0 [06:42] Last time I tried config manager to build stuff on gangotri it took over 20 minutes. [06:42] hi [06:42] is launchpad off for maintainance? [06:43] Is it using faster transports now, or has bzr's sftp been sped up yet? [06:44] stub: 'update' not 'build' [06:44] stub: we *have* a good tree, we should be able to manipulate it without doing fresh ones each time [06:44] ahh... yes. update didn't exist then ;) [06:45] So rsync prebuilt launchpad and cm update to build a fresh tree, and just cm update to update one [06:45] I think thats a workable theory [06:45] we'll need to give cm 'switch' or 'overwrite' rather than plain ol update [06:45] but thats not hard [06:46] I'm adding that to my TODO [06:46] Oh... yes. That would be good. I'll try it with just build for the time being. [06:52] stub: that gpg key file was removed and replaced by a symlink [06:54] $ bzr pull --overwrite sftp://stub@chinstrap.ubuntu.com/home/warthogs/archives/rocketfuel/launchpad/production/1.41 [06:54] bzr: ERROR: Not a branch: sftp://stub@chinstrap.ubuntu.com/home/warthogs/archives/rocketfuel/launchpad/production/1.41 [06:54] Anyone see a typo in that, or is this the error jblack confirmed for me the other day? [06:54] maybe //home instead of /home [06:54] depending on how old the bzr is [06:55] Double slash looks like it worked [06:55] bah - pull overwrite didn't ;-( [06:55] stub: and sftp transport should be a bit faster than it was now, since you can use compression [06:55] 0 revisions pulled :-( [06:57] grrr [06:57] truncate .bzr/revision-history [06:57] just by 3-4 lines [06:57] or 10-20 [06:57] I've just added a todo to triple check the overwrite code [06:58] Chop the tail you mean? [06:59] Still 0 revisions pulled (although bzr log reports an earlier revision now) [06:59] yah [07:00] if want want to cheat [07:00] chop it back tot he revision you want at the tip [07:00] (log --show-ids | head) in the production branch will give you that [07:00] But I'd then need to revert [07:00] yes [07:00] garh right [07:00] pull DOES a merge though [07:00] you can't avoid that code path === GoRoDeK [n=gorodek@p5083F040.dip.t-dialin.net] has joined #launchpad [07:01] I'll need to use config manager to build a fresh tree I think - only way I can think of to avoid needing to pull --overwrite or revert (?) [07:02] Apart from manually assembling it [07:02] so [07:02] do that on chinstrap [07:02] then rsync on top of an existing tree [07:02] for max speed [07:02] Good idea. [07:08] Ok - all build on chinstrap. [07:16] Merge to devel/launchpad: [trivial] Add all arcchitectures to staging's Gina configuration (r2894: Stuart Bishop) [07:20] librarian back up [07:23] launchpad back up [07:23] :-) [07:51] stub: re: bug 5338, would this mean that the same auth cookie would work on both production instances? [07:51] Malone bug #5338: Zope read and write conflicts when reading and saving session data In: launchpad (upstream), Severity: Normal, Assigned to: Stuart Bishop, Status: New https://launchpad.net/bugs/5338 [08:16] jamesh: Yes [08:16] jamesh: So we can turn off server affinity [08:20] stub: okay, it'll be interesting to see how long I stay logged in after the change then ... [08:20] How long do you stay logged in for now btw? [08:22] morning [08:23] stub: depends. Sometimes a day, sometimes an hour [08:23] SteveA: Morning [08:24] Ooh.... pqm spam. [08:25] stub: at the moment I stay logged in for < 1 form [08:25] i.e. login [08:25] fill out form [08:25] discuss a bit [08:25] click save, get 'not logged in error' [08:27] lifeless: i *still* need to fix the Vary header thing [08:27] ;0 [08:27] i need to locate what branch that work is on and revive it === stu1 [i=stub@sweep.bur.st] has joined #launchpad [08:29] Wheee.... locked up except for my caps lock and scroll lock lights flashing in unison [08:29] dude [08:29] whack ;0 === robitaille [n=robitail@ubuntu/member/robitaille] has joined #launchpad === gml_ [i=gml@dyn-83-156-0-5.ppp.tiscali.fr] has joined #launchpad === mpt [n=mpt@200-158-80-35.dsl.telesp.net.br] has joined #launchpad === zyga [n=zyga@2-mi2-1.acn.waw.pl] has joined #launchpad [09:34] stevea: You're UP! [09:34] ewwww [09:34] TMI [09:34] tmi.. [09:34] too much imagination. [09:34] lol === SteveA [n=steve@195.182.78.95] has left #launchpad ["Leaving"] [09:36] hmm. kind of hard not to take that the wrong way [09:36] morning all [09:38] hi sivang [09:39] lifeless: Where do we want the new mirrors to go? (the old ones go to http://mirrors.sourcecontrol.net) [09:39] bzr mirrors ? [09:39] yeah [09:39] they should be published on bazaar.launchpad.net [09:40] isn't that where the imports go? [09:40] using the supermirror file system hierarchy naming scheme [09:40] yes [09:40] we're publishing them all in one namespace [09:40] the bzr sftp server will be listening on bazaar.launchpad.net too [09:40] evening [09:40] so there is no conflict with the old arch based service [09:41] elmo has a new ip address on the production supermirror to allow this for sftp already [09:41] Ok. so the current imports are listed at bazaar.launchpad.net and mirrored to mirrors.sourcecontrol.net [09:41] no [09:41] they are listed at bazaar.ubuntu.com [09:41] bazaar.launchpad.net is currently unused [09:41] +click+ [09:41] ;) [09:42] Not sure why we need an extra ip though. virtual hosting doesn't require it [09:42] SFTP handshaking does [09:42] like ssl in that respect, host id occurs after key exchange === SteveA [n=steve@195.182.78.95] has joined #launchpad [09:44] I want a new documentroot. [09:45] I don't think we specify the root. I'd like /src/sm-ng [09:57] stevea: I did manage to get a working gnomemeeting going. [10:00] gnomemeeting is kinda tricky to get working unless you can configure your NAT routers [10:02] which I had to do. [10:02] took some effort. I was double natted before I started. [10:03] skype doesn't run here. [10:03] jblack: you probably need to statically compiled version with QT in [10:03] jblack: that will save you lots of hand dependency resolution of missing deps [10:03] skype has sources? === poningru [n=poningru@n128-227-82-53.xlate.ufl.edu] has left #launchpad [] === sivang checks [10:05] hmm, seems not [10:06] Would surprise me. skype is supposed to be proprietary software. === poningru [n=poningru@n128-227-82-53.xlate.ufl.edu] has joined #launchpad [10:06] gnomemeeting was a lot a lot easier to set up this time. relatively, that is. [10:06] Now they tell you how the firewall is in the way, and waht specifically you need to do. [10:07] I can even _receive calls_ (oohhh, ahhh) [10:07] Talked with an old friend from New Zealand [10:07] jblack: skype tells you cannot receive calls? [10:07] skype doesn't start at all. [10:08] no output, no anything. [10:10] that happened to me [10:10] then as soon as I started stracing it to see what it was doing, it showed a GUI [10:10] I'm happier that way anyways. gnomemeeting is free software. [10:11] I haven't managed to get gnomemeeting working [10:11] poll( [10:11] shtoom is great when it works [10:11] if it was as just-works as skype, it would be wonderful [10:11] when did you last try gnomemeeting? [10:12] a while ago [10:12] does it do SIP now? [10:12] its still h323. [10:12] ah, ok [10:12] I know even less about h323 than I do about SIP :) [10:12] and 326 or 328 or something that helps with nat [10:12] You still have t o map a tcp port, but that's generally pretty easy. [10:13] natted nat confused it, but when I stuck my wireless ap on the outside of the local firewall, punched.. 1070 or some such, it worked great [10:13] jblack: when you say "skype doesn't run", did you try sladen's apt repository? [10:13] thats where I got it from, yes [10:14] you know that it takes 45 secs to start? [10:14] I may not have waited that lone. [10:15] it takes AGES! [10:15] gosh [10:15] gah. it did come up. [10:15] I still want to use gnomemeeting [10:15] i can't punch holes in my nat [10:16] SteveA: slande's repo has a package for the statically linked version? [10:16] i have no idea [10:16] it works for me on breezy [10:16] works for me on dapper [10:16] jblack: if it still doesn';t start after you wait 45 seconds, then fetch the tar.gz that stub did, it shoudl work [10:17] this license agreement says it can install spyware [10:18] ohhh. looks like this may be an illegal version of skype too, unless Paul got an excpetion to 3.1 [10:24] stevea: you have to punch for skype too, for inbound calls [10:25] i didn't have to [10:25] jblack: I called Steve yesterday with skype, and neither of us did any manual punching [10:26] i think it uses outbound HTTP polling [10:26] Hmm. [10:26] in a number of weeks, we'll be using sip / asterisk kinda stuff [10:26] when it is set up properly in the DC [10:27] maybe the advanced settings page is wrong [10:27] until then, i'd like people to use skype, and not get everyone setting up several different systems [10:28] respectfully, I'm not happy about skype === stub [i=stub@sweep.bur.st] has joined #launchpad [10:31] asterik is cool [10:31] I understand the difficulties of coordinating an organization. I've got it, and I'll use it. [10:32] But I'm disatisfied with this decision. [10:33] jblack: i appreciate what you're saying. [10:35] I'm ready to receive that phone call when you want. I'd like to keep it short and professional [10:35] stub: hi. when will staging next be updated to use new code? [10:47] bug 5411 [10:47] Malone bug #5411: "list all packages" times out In: launchpad (upstream), Severity: Normal, Assigned to: Nobody, Status: New https://launchpad.net/bugs/5411 [11:11] bug 1 [11:11] Malone bug #1: Microsoft has a majority market share In: Ubuntu, Severity: Critical, Assigned to: Mark Shuttleworth, Status: Accepted https://launchpad.net/bugs/1 === Keybuk [n=scott@descent.netsplit.com] has joined #launchpad === zyga [n=zyga@2-mi2-1.acn.waw.pl] has left #launchpad ["Leaving"] === beyond [n=beyond@200-171-140-32.dsl.telesp.net.br] has joined #launchpad === Kinnison workraves === mpt [n=mpt@200-171-140-32.dsl.telesp.net.br] has joined #launchpad === niemeyer [n=niemeyer@200-171-140-32.dsl.telesp.net.br] has joined #launchpad [11:50] SteveA: Staging was updated earlier today, before you came online. [11:53] Morning! [11:53] Morning === WaterSevenUb [n=WaterSev@azevedo.astro.up.pt] has joined #launchpad [11:56] stub: looks like the error reports are getting generated okay [11:56] (for staging) [11:57] jamesh: the OOPS ids look a little short [11:57] OOPS-S6 [11:58] i was expecting them to have the same length each time [11:58] SteveA: yeah. I guess I could pad them [11:58] mpt: which do you think is better for end userse? [11:58] does the "S" in "S6" change? [11:58] S is for staging [11:59] daf: S == staging [11:59] in production it will be A or B [11:59] ah, I see [12:01] new layout is on staging now [12:02] Goooooooooooooood morning Launchpadders [12:03] SteveA, which out of what? [12:04] OOPS-S6 vs. OOPS-S0006 [12:04] I don't think it matters in the slightest :-) [12:04] Are they being lined up in a table somewhere? [12:04] ok, then we'll leave it as is [12:04] not for users [12:05] maybe for developers some day [12:05] jamesh: this is really cool. [12:05] so, what happens next? analysis scripts/ [12:05] mpt: also, could you check the wording I added on the exception pages? [12:06] SteveA: yeah. I'll base them on what kiko is using [12:06] jamesh, URL? [12:06] let me check [12:06] https://staging.ubuntu.com/asdasd [12:07] launchpad-notfound.pt, launchpad-oops.pt and launchpad-requestexpired.pt [12:07] jamesh, that's fine [12:08] I might do some tweaking later to further discourage people from quoting the never-changing boilerplate in bug reports [12:09] Should we have a 'report a bug' link, which prefils the OOPS id, on those pages? Or is that just asking for trouble and dupes? === stub is sure this was probably discussed in Montreal [12:10] stub: not sure. The fact that the oops ID has been shown to the user means we have a record of the crash === matsubara [n=matsubar@200-171-140-32.dsl.telesp.net.br] has joined #launchpad [12:10] mpt: on the new layout, the menus have a clickable area that is just the text. [12:10] good morning! [12:10] stub: we don't want a situation where we have a bug report for every oopsid with a description of "asdf" [12:10] stub, kiko's reports are more accurate and less noisy, and what jamesh said [12:10] is there a way to make it the entire row area? [12:11] SteveA, ah yes, I tried to fix earlier but it was screwing with the bullets [12:11] I'll have another go [12:11] do we need bullets? [12:11] i mean, at the top level [12:11] i guess so... [12:12] Well now they're not tabs, they can have proper icons [12:12] the funny thing is, i keep trying to scroll up [12:12] because the top of the page is so much thinner than i'm used to [12:13] cripes, it's on production already! [12:13] mpt: do we need "Launchpad >>" at the start of the hierarchy? [12:13] no it isn't [12:13] it's on staging [12:13] oh === mpt stares at his URL field [12:14] how did that happen [12:14] oh, I used Ctrl+Enter [12:14] i asked stub to update staging [12:15] 25% is a lot when i'm using the full width of my 1280 screen [12:15] yes [12:17] I think it would be odd to have a hierarchy that *didn't* start with the front page [12:17] there's the icon [12:17] }=> === Kinnison turns his head to one side to make the icon look right [12:18] mpt: are the columns 25% including margins and gutters ? [12:19] mpt: the new menu stuff in staging is interesting [12:19] do you think it would work to have the whitespace on the extreme LHS and RHS less wide? [12:20] daf: the menus top left aren't finished. i need to write some code for them [12:20] SteveA, they're 24%. The gutters are 1% each. [12:20] I like the fact that the tabs are gone, I think [12:20] I think they were misleading [12:21] mpt: can you make the whitespace on the extreme LHS and RHS a fixed width? [12:21] yes, I never liked them where they were, but I wasn't in charge of their position [12:21] SteveA, it already is, 1em [12:21] so the layout is [12:21] try 0.5 em [12:22] [1em margin] [[24% column] [1% space] [50% column] [1% space] [24% column] ] [1em margin] [12:23] SteveA, I think it would look wrong for the sidebars to be closer to the edge of the window than they are to the rest of the layout (0.5 em < 1% for most screen widths) === mpt tries === cprov [n=cprov@200-171-140-32.dsl.telesp.net.br] has joined #launchpad [12:25] hello morning people [12:25] jamesh, would you like a new copy of the analysis script? [12:25] it fixes the bug ddaa reported [12:25] yeah, it's a bit odd [12:25] morning guys [12:25] the mainloop is braindead in that it read()s the whole files [12:26] but it's probably not too difficult to convert it into a state machine that readline()s the file [12:26] or perhaps you will do things using a different model [12:26] (is jamesh around?) [12:26] daf, what's up with bug 5411? [12:26] Malone bug #5411: "list all packages" times out In: launchpad (upstream), Severity: Normal, Assigned to: Nobody, Status: New https://launchpad.net/bugs/5411 === salgado [n=salgado@200-171-140-32.dsl.telesp.net.br] has joined #launchpad [12:27] hey salgado [12:27] yo kiko [12:27] what's cookin [12:28] need to check if everything is okay with shipit reports and get it running on production [12:29] stub, ping [12:29] mpt: i looked in my browser at various window sizes (but not various font sizes), and there just seemed to be too much whitespace around the edge of the content/boxes area [12:29] kiko: just what it says -- if you ask for a list of all packages, it oopses with a timeout [12:30] daf, have you checked out what sort of SQL is being executed for that page? [12:30] mpt: and, comparing to production, a little more than before [12:30] I suspect we may need a view for that [12:30] kiko: not yet -- just wanted to make a note of it [12:30] SteveA, ah, true, it was 0.75em before [12:30] I'll trim it down [12:31] mpt: you and i have about half a week to get this polished off, before it goes out to production. i'll get cracking on the code soon. [12:33] daf, I am saying that because I suspect an sql view will be required to render that.. [12:36] salgado: pong [12:37] kiko: even with a view, the amount of time needed to transfer the data from the DB and render it might still be large [12:38] Can individual pages request a bit longer to do their job [12:38] E.g. the +all pages could say "I take a while" [12:38] and get maybe 30 seconds or something [12:38] Kinnison: no. it would be possible to make it so. [12:38] i don't think it's all that good an idea [12:38] salgado: I've run the country/continent update [12:38] daf, perhaps. it's like 12K packages IIRC [12:38] kiko: yeah, could you send it to me? [12:39] jamesh, sure. [12:39] stub, do we have revision 2883 in staging? if so, could you run the shipit-reports cronscript there again? [12:39]