[03:04] <micahg> lifeless: I just had convert to question time out on non-edge, do you need the OOPS?
[03:05] <lifeless> micahg: no we have the one from earlier
[03:05] <lifeless> micahg: edge and nonedge run the exact same code now
[03:05] <micahg> lifeless: ok, that second thing was what I was wondering, thanks :)
[03:06]  * micahg tries again to convert to question
[08:09] <rryan> push/pull from bazaar.launchpad.net isn't working right now.. "bzr: ERROR: Connection closed: Unexpected end of message. Please check connectivity and permissions, and report a bug if problems persist." anybody else seeing this?
[08:10] <spiv> rryan: pastebin the full output?
[08:10] <rryan> http://pastebin.com/Uxp6yzeK
[08:10] <webrsk> same error occurs for me too
[08:10] <spiv> Oh, hmm, I do see an issue.
[08:11] <spiv> LOSA ping: codehosting appears to be busted
[08:11] <spm> codehosting???
[08:11] <spiv> spm: see that pastebin link, I get it too
[08:11] <spiv> "#
[08:11] <spiv> ValueError: bad marshal data" importing bzrlib.
[08:11] <spm> wow
[08:12] <spiv> spm: which implies srv/bazaar.launchpad.net/production/launchpad-rev-9885/eggs/bzr-2.2.0-py2.6-linux-x86_64.egg/bzrlib/builtins.pyc is broken
[08:12] <spiv> Which would be very weird, but perhaps try deleting that .pyc file I just mentioned?
[08:12] <spiv> Or better, rename it out of the way
[08:13] <spiv> so we can look at it later to see if there's any clues to be found in it :)
[08:13] <spm> renamed, try now?
[08:14] <spiv> spm: looks better now
[08:14] <spm> interesting. same file size, but diff reports differences. dunno if significant differences or not but... curious.
[08:14] <spiv> spm: hmm!
[08:14] <al-maisan> I was able to push my branch now. Thanks!
[08:15] <spiv> spm: it's EOD for me, so I guess post to the lp list about it
[08:15] <spiv> spm: but I'm very curious :)
[08:15] <spm> spiv: it's EOD for me too :-)
[08:15] <spiv> spm: hopefully it's not a sign of hardware flipping bits... :/
[08:16] <spm> truly
[08:17] <mthaddon> ok, let's not rush to blame hardware here guys :)
[08:21] <webrsk> now works fine !! able to push and pull :)
[08:26] <al-maisan> I am getting a "ssh: connect to host bazaar.launchpad.net port 22: Connection refused" when pushing a branch
[08:27] <al-maisan> Is this expected or looked at already?
[08:28] <al-maisan> losa ping re. ^^
[08:29] <spm> I think a restart might be warranted...
[08:29] <spm> ew. a ps listing looks unhealthy. definitely a restart needed.
[08:29] <al-maisan> ouch
[08:30] <spm> oh. it's crashed. that's why.
[08:30] <spm> al-maisan: try now?
[08:31] <mthaddon> seems better from here
[08:33] <al-maisan> spm: worked now, thanks!
[08:34] <spm> coolio
[08:34] <al-maisan> one only realises how critical LP is when it stops working :P
[08:37] <spm> :-)
[08:38] <vila> can we please keep it down a few hours so we can give better error messages from all apps relying on it ? :-D
[08:38] <al-maisan> he-he
[08:38] <vila> but I still get 'Connection refused' here...
[08:39] <vila> on both 91.189.89.223 and 91.189.89.222 (if that helps)
[08:41] <vila> wow wow, it came back online briefly in the last minutes (my hudson botnet was able to connect successfully once out of 3 attempts (from 3 different jobs))
[08:50] <vila> mthaddon: is lp still reachable for you and should I search for something specific to me or did it get down again ?
[08:51] <mthaddon> vila: can you be more specific - what part isn't working for you?
[08:51] <vila> http://paste.ubuntu.com/520056/
[08:52] <mthaddon> vila: I don't think you've ever been able to ssh to launchpad.net have you?
[08:52] <vila> no, but shouldn't it display the debug messages for keys attempted ?
[08:52] <mthaddon> vila: I mean, try bazaar.launchpad.net
[08:53] <vila> argh, yes, just realized and verified that :(
[08:53] <vila> sorry for the noise
[08:53] <mthaddon> k
[09:05] <wgrant> [A3
[09:05] <wgrant> Bah.
[09:12] <fta> OOPS-1760N420
[09:13] <fta> (can't login using the API on LPNET_SERVICE_ROOT)
[09:14] <fta> started less than 3h ago
[09:41] <bac> fta: what were you trying to do?  you say you can't login but the OOPS shows that you were logged in, i think.  i'm currently able to use lplib against production.
[09:44] <fta>     builds = ppa.getBuildSummariesForSourceIds(source_ids = chunk)
[09:44] <fta>   File "/usr/lib/python2.6/dist-packages/lazr/restfulclient/resource.py", line 528, in __call__
[09:44] <fta>     url, in_representation, http_method, extra_headers=extra_headers)
[09:44] <fta>   File "/usr/lib/python2.6/dist-packages/lazr/restfulclient/_browser.py", line 277, in _request
[09:44] <fta>     raise HTTPError(response, content)
[09:44] <fta> lazr.restfulclient.errors.HTTPError: HTTP Error 503: Service Unavailable
[09:45] <fta> bac, ^^
[09:45] <bac> fta: i was wrong, i see you were not logged in
[09:46] <bac> fta: and this failure is ongoing?
[09:46] <fta> bac: yes, still there
[09:47] <fta> it's a timeout in getBuildSummariesForSourceIds() so i'm logged in (confusing error page btw)
[09:48] <bac> fta: i'm having an admin check.  we saw a similar problem over the weekend.  thanks for reporting it.
[09:59] <pmjdebruijn> silly question, when I add my PPA, my packages are also available via software center, however, my package does not have a screenshot... how do I upload a screenshot for it... please do note, there is no equivalent package in the main repos
[10:03] <wgrant> pmjdebruijn: I'm not sure that's possible. Ubuntu Software Center uses screenshots.debian.net.
[10:04] <bac> fta: it looks like you are requesting summaries for a large number of source ids at once and the query is timing out.  can you restructure your code to request fewer in one go?  i've done that interactively and see that it works.
[10:05] <pmjdebruijn> wgrant: I was afraid of that :)
[10:05] <fta> bac, i fetch by chunks of 25 source ids, it worked for months
[10:06] <pmjdebruijn> wgrant: I guessed as much, thanks for confirming
[10:08] <fta> bac, http://paste.ubuntu.com/520087/
[10:39] <bac> fta: https://bugs.edge.launchpad.net/soyuz/+bug/662523 ... you filed this bug last week?  it seems to be the same issue.
[10:40] <fta> bac, that was against edge, i've been told to use the production server instead and it works for ~2 days without timeout
[10:41] <fta> -works+worked
[10:41] <bac> fta: ok.  might be good to add that extra info to the bug if you would
[10:47] <leoquant> anyone a linkage to the livestreams from the uds orlanda? (wiki maybe?)
[10:55] <fta> bac, done
[10:56] <bac> thx
[11:05] <lifeless> fta: edge and production run the same code
[11:05] <lifeless> fta: using production is good, thank you for doing that.
[11:05] <lifeless> until we fix the bug though, it will timeout from time to time.
[11:06] <fta> lifeless, atm, it timeouts 100% of the time. even with chunks of 15
[11:06] <fta> trying 10
[11:07] <lifeless> fta: yeah, it sucks.
[11:08] <lifeless> fta: its the pg8.4 transition; our previously tuned queries are fail against pg8.4, we need to retune the *entire app* - thats hundreds of qqueries.
[11:08] <lifeless> your one is in the pile of worst-offenders.
[11:08] <fta> doh!
[11:17] <poolie> istm the broken people vocabulary is a bit critical
[11:17] <poolie> you cannot add new users to a code review for example
[11:18] <poolie> oh ok it did work after a few attempts, but i'm seeing mostly timeouts
[11:22] <lifeless> poolie: its being worked on
[11:37] <poolie> vila, if you have time you could look into bug 666642
[11:45] <MrJones> hi
[11:45] <MrJones> I just attempted to register an account since I want to report a bug (and I guess an account is required for that) and I just get "Invalid OpenID transaction" when clicking on "Login/Register"
[11:47] <MrJones> oh well... enabling cookies for launchpad.net helped. maybe that error message should be changed to something more obvious :x
[11:49] <poolie> good idea
[17:46] <odony> Hello, we have a bazaar branch that seems corrupted on the Launchpad servers,  we have opened a bug on launchpad-code project, but no-one seems to handle it there... anything else we could do to repair this branch?
[17:47] <odony> (for reference, that's bug 666642)
[18:23] <odony> I wonder what all these connected users are... only bots?
[18:23] <odony> no-one from Launchpad here?
[18:37] <jcsackett> odony: it's UDS--a lot of the lp developers are there. let me see if i can get someone to help you.
[18:39] <thekorn> jcsackett: too slow ;) he already left
[18:40] <jcsackett> thekorn: ah, so he did.
[19:43] <mwhudson> i want the title links on bug 1111 in text back
[19:43] <mwhudson> if it works by ajax that's fine
[19:56] <mtaylor> weird problem - my branch (https://code.edge.launchpad.net/~mordred/drizzle/bug666814) has been "updated branch" for ... a while
[19:59] <shtylman> does the self hosted launchpad install support building packages as well?
[19:59] <shtylman> i.e. can I build packages against various internal PPAs etc?
[20:00] <mwhudson> with sufficient effort, sure
[20:00] <shtylman> how much "effort" are we talking :)
[20:02] <mwhudson> shtylman: https://dev.launchpad.net/Soyuz/HowToUseSoyuzLocally is a start
[20:04] <shtylman> mwhudson: thx
[20:05] <mtaylor> mwhudson: launchpad hates me today
[20:11] <lbieber> This branch was uploaded over an hour ago, but still says updating.... https://code.launchpad.net/~mordred/drizzle/bug666814
[20:11] <lbieber> why the hangup?
[20:32] <lifeless> mtaylor: ask your question
[20:34] <mtaylor> weird problem - my branch (https://code.edge.launchpad.net/~mordred/drizzle/bug666814) has been "updated branch" for ... a while
[20:36] <mtaylor> and when I say "a while" - I mean over 4 hours
[21:41] <henninge> Hi henninge_!
[21:41] <henninge_> Hi henninge! ;)
[21:52] <shtylman> I am trying to do the "make schema
[21:52] <shtylman>  part of the lp install
[21:52] <shtylman> and it is complaining because it can't find bin/py
[21:52] <shtylman> in my devel tree
[21:52] <shtylman> this is right after doing the rocketfuel-setup step
[21:53] <danilos> hi henninge_ and henninge!
[21:54] <henninge> Hi Daniels!
[21:54] <henninge> danilos
[21:54] <henninge_> Auto complete playing a trick on henninge
[21:55] <danilos> henninge: *who* are you? :)
[21:55] <henninge> We?
[21:55] <henninge_> We?
[21:55] <danilos> aaah, my eyes, my eyes, double henninge
[21:56]  * danilos runs away in terror
[21:56] <henninge_> henninge: Go away, you are hurting him!
[21:56] <henninge> No, you go!
[21:58]  * henninge leaves
[21:59] <jcsackett> that was just weird.
[21:59] <jcsackett> funny, but weird. :-P
[21:59] <henninge_> finally ...
[21:59] <henninge> jcsackett: IRC from an Android phone is phun ... ;)
[22:10] <bigjools> shtylman: did anyone answer you?
[22:11] <shtylman> bigjools: nope... but I deleted the tree and am rerunning the rocket script again
[22:11] <shtylman> just to see if I messed anything up there
[22:12] <bigjools> shtylman: you need to run utilities/link-external-sourcecode
[22:12] <bigjools> and then make
[22:12] <shtylman> bigjools: k.. I will try that once the momma sources download
[22:47] <shtylman> bigjools: I did that... but make fails with the same problem
[22:55] <bigjools> shtylman: so link-external-source completes with no error?
[22:56] <shtylman> assuming I did it right
[22:56] <shtylman> in devel I did: ./utilities/link-external-sourcecode ../../lp-sourcedeps/
[22:56] <shtylman> but that might not be correct?
[22:56] <bigjools> perfect
[22:56] <shtylman> it seemd to be fine
[22:56] <bigjools> what does "make compile" say?
[22:58] <shtylman> currently sitting at: utilities/shhh.py PYTHONPATH= ./bin/buildout \ configuration:instance_name=development -c buildout.cfg
[22:58] <shtylman> is that normal?
[22:59] <bigjools> yeah it takes a while
[22:59] <shtylman> k
[23:09] <shtylman> it finished
[23:10] <shtylman> seemingly successfully
[23:10] <shtylman> running "make" now
[23:43] <shtylman> hm
[23:43] <shtylman> now I just get an ssl connection error message
[23:44] <wgrant> shtylman: Try restarting Apache.
[23:44] <shtylman> nope
[23:44] <wgrant> What's the error?
[23:44] <shtylman> also.. it said that /var/tmp/bazaar .. ppa.. archive
[23:44] <shtylman> do not exist
[23:44] <shtylman> when restarting apache
[23:45] <wgrant> That's fine.
[23:45] <shtylman> k
[23:45] <shtylman> Unable to make a secure connection to the server. This may be a problem with the server, or it may be requiring a client authentication certificate that you don't have.
[23:45] <shtylman> that is the error when I go to launchpad.dev
[23:45] <wgrant> With which browser? Is there a more detailed error code?
[23:46] <shtylman> Error 107 (net::ERR_SSL_PROTOCOL_ERROR): SSL protocol error.
[23:46] <wgrant> Is there anything in the Apache error log?
[23:46] <wgrant> And you did *restart* Apache, not just reload it?
[23:46] <shtylman> yep
[23:47] <shtylman> [Tue Oct 26 18:46:55 2010] [error] [client 127.0.1.1] Invalid method in request \x16\x03
[23:47] <shtylman> that is the only thing that seems of interest
[23:47] <wgrant> It sounds like it's trying to speak HTTPS to an HTTP port.
[23:48] <shtylman> hm
[23:48] <shtylman> the launchpad install is http by default
[23:48] <shtylman> right?
[23:48] <wgrant> No.
[23:48] <wgrant> It redirects to HTTPS.
[23:48] <shtylman> right
[23:48] <shtylman> k
[23:48] <shtylman> that is what I see happening
[23:49] <wgrant> What happens if you wget the HTTPS URL directly?
[23:50] <wgrant> Hm.
[23:50] <wgrant> Which URL are you going to?
[23:50] <shtylman> Connecting to launchpad.dev|127.0.1.1|:443... connected.
[23:50] <shtylman> OpenSSL: error:140770FC:SSL routines:SSL23_GET_SERVER_HELLO:unknown protocol
[23:50] <wgrant> Your /etc/hosts is broken.
[23:50] <shtylman> Unable to establish SSL connection.
[23:50] <shtylman> :(
[23:50] <wgrant> Did you not use rocketfuel-setup?
[23:51] <shtylman> I did... but then something else might have fixed my /etc/hosts after that
[23:51] <shtylman> (company runs puppet...)
[23:51] <shtylman> so that might have... ahem.. "corrected" it
[23:51] <wgrant> It should have added this to it:
[23:51] <wgrant> # Launchpad virtual domains. This should be on one line.
[23:51] <wgrant> 127.0.0.88      launchpad.dev answers.launchpad.dev archive.launchpad.dev api.launchpad.dev bazaar-internal.launchpad.dev beta.launchpad.dev blueprints.launchpad.dev bugs.launchpad.dev code.    launchpad.dev feeds.launchpad.dev id.launchpad.dev keyserver.launchpad.dev lists.launchpad.dev openid.launchpad.dev ubuntu-openid.launchpad.dev ppa.launchpad.dev private-ppa.launchpad.dev       shipit.edubuntu.dev shipit.kubuntu.dev shipit.ubuntu.dev ...
[23:51] <wgrant> ... testopenid.dev translations.launchpad.dev xmlrpc-private.launchpad.dev xmlrpc.launchpad.dev
[23:51] <wgrant> 127.0.0.99      bazaar.launchpad.dev
[23:51] <wgrant> 127.0.0.77      vostok.dev archive.vostok.dev
[23:52] <shtylman> wgrant: thank you
[23:58] <shtylman> personal launchpad for the win!