[11:46] <kittykitty> uhh, launchpad is down?
[11:48] <cjwatson> kittykitty: Network issue, I think
[11:49] <kittykitty> thought they fixed it 2 days ago cjwatson ?
[11:49] <cjwatson> kittykitty: Different one :P
[11:49] <kittykitty> god, again?!
[11:49] <cjwatson> kittykitty: I don't know any details, sysadmins are looking into it
[11:49] <kittykitty> cool cool , thanks cjwatson
[11:49] <acheronuk> ok, so not just me then... phew
[11:52] <wgrant> cjwatson: Oh, sorry.
[12:08] <shagan> still down :/
[12:08] <shagan> is there a mirror we could reach out?
[12:08] <shagan> trying to download : https://launchpad.net/gcc-arm-embedded/5.0/5-2016-q2-update/+download/gcc-arm-none-eabi-5_4-2016q2-20160622-mac.tar.bz2
[12:15] <cjwatson> shagan: I'm not aware of any mirrors of product release files.
[12:15] <cjwatson> shagan: Hopefully it won't be long.
[12:15] <shagan> @cjwatson hopefully not :/
[12:22] <amar__> Hello
[12:22] <axino> shagan cjwatson : it should be back
[12:23] <amar__> Launchpad is down?
[12:23] <cjwatson> amar__: Just back now.
[12:23] <shagan> it started, thanks guys!
[12:23] <amar__> ah great thanks
[12:24] <lawdpad> \o/
[12:28] <fhf> it up again
[13:40] <dholbach> hiya
[13:41] <dholbach> do I need to be of a special team to register a sprint for Ubuntu? I can register a general sprint on https://launchpad.net/sprints/+new (no option to choose the LP project) - but will that make it show up on https://launchpad.net/ubuntu/+sprints?
[13:51] <cjwatson> dholbach: Sprints aren't themselves directly related to specification targets such as distributions.  You get a sprint to show up on /ubuntu/+sprints by accepting Ubuntu-targeted specifications as topics for the sprint.
[13:51] <dholbach> ah ok
[13:51] <dholbach> thanks cjwatson
[14:06] <acheronuk> LP down again?
[14:06] <axino> acheronuk: yes, please bear with us
[14:06] <acheronuk> np
[14:06] <axino> acheronuk: it will be back soon
[14:07]  * acheronuk takes coffee break then :)
[14:12] <oparoz> Launchpad is still down
[14:12] <foobar_> Hi. It seems launchpad still down for me. Any news? According to twitter status it's up
[14:13] <oparoz> Same here
[14:13] <wgrant> The dodgy hardware has just dropped out again.
[14:13] <wgrant> It was back for a couple of hours, and will be back again shortly.
[14:13] <oparoz> Thanks! And good luck :)
[14:17] <kashyap> Hi folks; any ETA on when LaunchPad will be back up?
[14:19] <axino> kashyap: not yet unfortunately
[14:19] <kashyap> axino: Whoops.  Okay, so today I can defer my Openstack upstream bug triage, then... :-)
[14:22]  * cjwatson updates twitter status again belatedly
[14:22] <wgrant> cjwatson: Too late
[14:22] <LocutusOfBorg> hi launchpad, why you 503 since this morning?
[14:23] <cjwatson> wgrant: Oh, blah, tweetdeck confused me again
[14:23] <cjwatson> LocutusOfBorg: see /topic
[14:23] <andykimpe> hello please rapid solve Uh oh!  Something has gone wrong. We're sorry! If we are in the middle of an update, Launchpad will be back in a couple of minutes. Otherwise, we are working to fix the unexpected problems. Check @launchpadstatus on Twitter for updates.  If the problem persists, let us know in the #launchpad IRC channel on freenode.  Technically, the load balancer took too long to connect to an application server.  Reloa
[14:23] <cjwatson> andykimpe: see /topic, sysadmins are working on it
[14:24] <monsta> hi all. is packages.ubuntu.com on the same server as LP?
[14:24] <cjwatson> monsta: No, but it's behind some of the same network infrastructure.
[14:25] <LocutusOfBorg> !topic
[14:25] <LocutusOfBorg> thanks
[14:25] <monsta> cjwatson: ah ok, thanks, that explains why it's also unreachable
[14:26] <cjwatson> monsta: Right, it's a network switch problem not a problem with Launchpad per se.
[14:26] <monsta> ok I see
[14:27] <platzhirsch> Any known problems with the packages? https://launchpad.net/ubuntu/trusty/amd64/libssl-dev
[14:27] <LocutusOfBorg> platzhirsch, launchpad is down
[14:28] <platzhirsch> LocutusOfBorg: would that affect thepackages as well?
[14:28] <platzhirsch> the repositories
[14:28] <platzhirsch> because I can't install thepackage
[14:28] <cjwatson> platzhirsch: Not the Ubuntu archive, but PPA hosting is affected
[14:28] <wgrant> platzhirsch: What's the error?
[14:28] <wgrant> cjwatson: It's actually not :)
[14:28] <wgrant> PPAs are in GS2
[14:28] <cjwatson> Oh, good point, but "add-apt-repository" will be :)
[14:28] <wgrant> Ah yes
[14:29] <cjwatson> Anyway, yes, state the error.
[14:29] <platzhirsch> wgrant: cannot find package
[14:29] <andykimpe> I crazy about your topic  It is almost 17 hours and since no way this morning to connect  I'm asking is that his works  a server that is for fonctioner 24h / 24, 7 days 7
[14:29] <cjwatson> platzhirsch: No, copy and paste the error message, along with saying exactly what command you ran.
[14:29] <andykimpe> Server is made to run 24h / 24, 7 days 7
[14:30] <wgrant> andykimpe: There was a 30 minute outage two hours ago, and 25 minutes just now, due to hardware failures. I don't know what you mean when you say 17 hours.
[14:30] <acheronuk> andykimpe: to be fair, it has been up *most* of today
[14:30] <lamont> should I expect that I'm getting "Uh oh!" pages from lp right now?  (https://launchpad.net included)?
[14:30] <cjwatson> andykimpe: The network switch situation isn't good, but (a) it's outside the control of the Launchpad developers, we're waiting on sysadmins to fix the switch whose hardware is failing, (b) Launchpad has certainly not been down for 17 hours.
[14:30] <cjwatson> lamont: Yes, see topic.
[14:30] <lamont> doh
[14:31] <platzhirsch> Sorry, here http://pastebin.com/n810YqMz cjwatson wgrant
[14:31] <cjwatson> platzhirsch: OK, that's entirely client-side and nothing to do with Launchpad.  Try "sudo apt-get update" to start with, then "apt-cache policy libssl-dev"
[14:32] <platzhirsch> cjwatson: ah thanks, I'll try the cache policy
[14:33] <cjwatson> "sudo apt-get update" may fix it depending on what's wrong, but if not then "apt-cache policy libssl-dev" will tell us more.
[14:33] <andykimpe> I not hardly this page since 0:00 a.m and is 4:32 p.m. (GMT +1)
[14:34] <cjwatson> andykimpe: "this page"?
[14:34] <cjwatson> andykimpe: Launchpad has definitely been up for most of that time.  Perhaps you're conflating multiple problems.
[14:35] <platzhirsch> cjwatson: working now. I had apt-get update in before but apparently.. I don't know. Works now
[14:35] <cjwatson> platzhirsch: OK, great
[14:36] <platzhirsch> Thanks a lot, silly mistake :)
[14:50] <barry> ah, looks like launchpad is having hiccups
[14:50] <martin1> launchpad being down right now is really amazingly bad timing!
[14:50] <martin1> never mind :-(
[14:50] <martin1> hope you can get it back up and running soon
[14:50] <smcginnis> Sorry if it's been asked already lately, but anyone have status on what's going on and any ETA? I can try to spread the news in the OpenStack community.
[14:50] <martin1> i uninstalled PHP7 and went to install Ondrej's PHP5 but of course launchpad no worky, can't add the PPA... now I have a temporarily dead website :-)
[14:51] <mapreri> is patches.ubuntu.com in the same DC of launchpad?  (that one is down too…)
[14:51] <mg983> there's an outage, right?
[14:51] <martin1> mg983: yeah
[14:51] <axino> there is a network outage yes
[14:51] <mg983> an eta?
[14:52] <axino> no ETA right now
[14:52] <mordof> martin1: can you grab it from source somewhere else in the meantime?
[14:52] <wgrant> martin1: You can actually use PPAs, you just can't use add-apt-repository.
[14:52] <mg983> darn ok
[14:53] <wgrant> (add-apt-repository uses the webapp, which is dependent on servers sitting on the dead switch. but ppa.launchpad.net itself is fine.)
[14:53] <wgrant> So if you add the PPA to your sources.list manually, all should be good.
[14:53] <mordof> oooh
[14:53] <mordof> that's good to know
[14:54] <martin1> wgrant: oh, thanks
[14:54] <martin1> wgrant: how would I add e.g. ppa:ondrej/php ?
[14:55] <martin1> the URL is https://launchpad.net/~ondrej/+archive/php which is obviously also down
[14:55] <dobey> you have to wait for LP to be back up again
[14:55] <wgrant> martin1: 'deb http://ppa.launchpad.net/ondrej/php/ubuntu SERIES main'
[14:55] <wgrant> dobey: no
[14:56] <martin1> wgrant: isn't that then going to poll ppa.launchpad.net ? or is that a separate server that's not down?
[14:56] <cjwatson> Getting the key to add a PPA by hand is tricky, though.
[14:56] <cjwatson> martin1: ppa.launchpad.net is up
[14:56] <martin1> cjwatson: thanks
[14:56] <wgrant> martin1: ppa.launchpad.net is in the other datacentre, so it's fine
[14:56] <martin1> aah thanks
[14:56] <martin1> much appreciated
[14:56] <mordof> wgrant: was that an intentional decision to separate the two?
[14:56] <wgrant> It's just the switch with critical single-point-of-failure master database infrastructure that's failed.
[14:57] <cjwatson> (I don't think the key is currently available anywhere other than the database/webapp)
[14:57] <wgrant> cjwatson: Indeed...
[14:57] <wgrant> mordof: Not exactly.
[14:57] <dimitern`> any idea how long a downtime are we looking at?
[14:58] <cjwatson> dimitern`: We haven't been given an ETA by sysadmins yet, I'm afraid.
[14:58] <dimitern`> cjwatson: right, ok - thanks
[14:59] <Rumbles> yey my ppa works again now :D
[14:59] <mg983> mine too Rumbles
[14:59] <mordof> wgrant: alright. i'm learning about building systems that could have considerations like that, so was thinking it'd be interesting to get insight into that if it were intentional
[14:59] <mordof> if it was*...?
[15:00] <mordof> had been?
[15:00] <mordof> lol
[15:00] <cjwatson> "if it were" is right here, since it's a counterfactual :)
[15:00] <mordof> ok
[15:00] <mordof> thanks :)
[15:00] <cjwatson> (or "if it had been", I guess.)
[15:01] <mordof> maybe the two are interchangeable here? they both feel correct. not sure
[15:01] <cjwatson> Different tense.
[15:01] <wgrant> mordof: In an ideal world we'd have one of everything in each of our London DCs. But there are a couple of things that aren't: one of them is ppa.launchpad.net, because it's very large and not currently practical to replicate live; and another is the master DB infrastructure, because there has to be a single master at any one time.
[15:02] <mordof> cjwatson: english is hard sometimes
[15:02] <mordof> wgrant: that makes sense :) thanks
[15:03] <cjwatson> We do have some tentative plans for de-single-point-of-failure-ing ppa.launchpad.net; in doing that we would probably want to think about attempting to preserve the useful property of independence from master DB problems that we've seen here (though doing so fully may be hard, since the entire reason it's independent is that it has a multi-terabyte copy of lots of stuff locally ...)
[15:03] <wgrant> cjwatson: That's already solved by the multiple pgbouncer solution which is planned.
[15:03] <seme> with ppa down will I be able to do things like add apt repositories on my system?
[15:03] <wgrant> Since they can reasonably run off a slave.
[15:04] <cjwatson> wgrant: That's better robustness of the core rather than independence of PPA, but yes
[15:04] <cjwatson> seme: Once added they'll work, but add-apt-repository won't work at the moment.
[15:04] <cjwatson> (Well, "add-apt-repository ppa:..." that is.)
[15:04] <mordof> wgrant: is pgbouncer a solution for handling many read-nodes, with a way of distributing writes to all of them?
[15:05] <seme> damn
[15:05] <seme> thanks
[15:05] <wgrant> mordof: Well that's a funny story actually
[15:05] <cjwatson> https://pgbouncer.github.io/
[15:06] <cjwatson> https://wiki.postgresql.org/wiki/PgBouncer is perhaps slightly more readable ...
[15:06] <dobey> mostly what i'm missing right now is IRC :)
[15:06] <mordof> yeah, their main page doesn't explain things that well, lol
[15:06] <mapreri> is patches.u.c in the same place?  or, why is it down too?
[15:06] <cjwatson> mapreri: It's behind the same failed switch.
[15:06] <mapreri> oh, now it replied, damn.
[15:07] <mordof> wgrant: i'd be interested in hearing about it if you've got the time/interest :)
[15:07] <mapreri> cjwatson: i suspected it.  though now patches.u.c replied, but lp doesn't :|  I suppose that's enough for now for me :)
[15:07] <dobey> oh well, hopefully it'll be back when i return from lunch
[15:07] <dimitern`> paste.u.c is back at least
[15:07] <cjwatson> mapreri: Earlier some packets were getting through but not all; it may be, er, patchy.
[15:08] <mapreri> ic
[15:10] <dimitern`> and LP's back! \o/
[15:10] <dimitern`> (most of it - some imgs still not appearing though)
[15:10] <cjwatson> Probably not reliably.
[15:10] <andrew90> hi guys!
[15:10] <andrew90> what happened?
[15:10] <mordof> oh, it seems pgsql has been adding replication into their own stuff.. we'd been using pg 9.1 up until recently, so i hadn't noticed
[15:10] <andrew90> trying to download a jar from launchpad, it seems offline?
[15:11] <dimitern`> down again :/
[15:11] <smcginnis> andrew90: Network outage is being worked on.
[15:11] <smcginnis> No ETA at the moment.
[15:11] <andrew90> ok, thank you smcginnis
[15:13] <andrew90> are there any other mirror available? I'm trying to get https://launchpad.net/sikuli/sikulix/1.1.0/+download/sikulixsetup-1.1.0.jar
[15:15] <cjwatson> andrew90: I'm not aware of any other mirrors for project release files like that in general, no, although of course individual projects might happen to host them elsewhere as well.
[15:21] <andrew90> Thank you again then :)
[15:21] <andrew90> bye!
[15:32] <madjedj> Hi all, is some know how long Launchpad will be unavailable ?
[15:32] <mordof> no ETA provided as of yet
[15:34] <jfjeronimo> Hola! Tengo problemas con el repositorio pi-rho, ¿alguien sabe si lo solucionarán pronto? Gracias!
[15:35] <axino> jfjeronimo: hi, launchad is unreachable because a network outage
[15:35] <rashar> Is ppa/arand down?
[15:35] <madjedj> mordof: ok thank's and good bye. :(
[15:35] <jfjeronimo> hi axino, thank you!
[15:35] <brookshank> ETA on it coming back?
[15:35] <axino> no ETA yet
[15:35] <brookshank> damn
[15:38] <BlackMage> what kind of network outage?
[15:39] <wgrant> BlackMage: The switch with the master database server on it failed.
[15:40] <BlackMage> wgrant: a joke?
[15:41] <wgrant> BlackMage: No.
[15:43] <mordof> wgrant: did it take any other hardware with it? or it just died itself?
[15:57] <jfjeronimo> Ready! Thanks!
[15:57] <axino> OK
[15:57] <axino> Launchpad should be back
[15:57] <cjwatson> axino: Likely to stay up now?
[16:00] <axino> cjwatson: likely, yes.
[16:03] <mordof> hurray for launchpad!
[16:03] <mordof> good job to all involved for identifying and solving the problem :)
[16:03] <mordof> thanks for keeping things running ^_^
[18:12] <JeanLuc_> Okay this is the 3rd channel I was sent to, so please don't just respond with "ask in $SOME_OTHER_CHANNEL". I want to build my vala/gtk-application on launchpad as part of a ppa. But I am facing a strange issue: First a vapi-file is being created during the build, then right after the build fails complaining it can't find the exact same file.
[18:12] <JeanLuc_> error: /«BUILDDIR»/feedreader-2.0.0~r1264+pkg25~ubuntu16.10.1//«BUILDDIR»/feedreader-2.0.0~r1264+pkg25~ubuntu16.10.1/obj-x86_64-linux-gnu/FeedReaderCommon.vapi not found
[18:12] <JeanLuc_> https://launchpadlibrarian.net/286704177/buildlog_ubuntu-yakkety-amd64.feedreader_2.0.0~r1264+pkg25~ubuntu16.10.1_BUILDING.txt.gz
[18:15] <nacc> JeanLuc_: to repeat what i said in #ubuntu:
[18:16] <nacc> 2.0.0~r1264+pkg25~ubuntu16.10.1//«BUILDDIR»/feedreader-2.0.0~r1264+pkg25~ubuntu16.10.1/obj-x86_64-linux-gnu/FeedReaderCommon.vapi
[18:16] <dobey> well, this is the wrong channel, as this doesn't have anything to do with launchpad itself. it seems there is a problem with the build rules, and the vapi file is either not being created, or created in a different location than the latter rules expect it
[18:16] <nacc> error: /«BUILDDIR»/feedreader-2.0.0~r1264+pkg25~ubuntu16.10.1//«BUILDDIR»/feedreader-2.0.0~r1264+pkg25~ubuntu16.10.1/obj-x86_64-linux-gnu/FeedReaderCommon.vapi not found
[18:16] <sarnold> wow there's a lot of warnings in that thing, it's hard to even spot an error among all the warnings...
[18:19] <sarnold> sure enough that last "/usr/bin/valac -C -b ..." command refers to that filename with the funky doubled-BUILDDIR, near the end
[18:20] <dobey> indeed
[18:20] <dobey> fix your cmake :)
[18:21] <sarnold> whether that means there's a missing filename or missing file or needless reference to a variable, I can't guess, I don't know Cmake nearly well enough
[18:22] <dobey> the CMakeLists.txt is definitely wrong somewhere