[15:41] <karz> Guys, I cannot login to my account. Every time I try there's oops something went wrong
[15:42] <karz> This is not the first time. I don't remember how many times I have done it. forgetting password and yet it seems to not care
[15:42] <tomwardill> karz: send an email to feedback@launchpad.com with your account name and copy and paste the OOPS-ID you get, and someone will look at it
[15:42] <karz> hmm okay.
[15:43] <tomwardill> there's some cases that can cause your account to end up in a state that means you can't login, but they're usually straightforward for us to fix
[15:43] <tomwardill> ping SpecialK|Canon ^^
[15:46] <karz> Thanks tomwardill I have sent it.
[16:09] <SpecialK|Canon> tomwardill: karz has gone, but if they return, can you direct them to feedback@launchpad.net (not com) please
[16:09] <tomwardill> oh whoops, sorry
[16:09] <xnox> SpecialK|Canon: can the topic be updated to state that is is known that lcy builders are disabled, and are being investigated to bring them back online? And that buildd farm is not healthy.
[16:11] <xnox> i.e. that 9.5% of builders are down.
[16:49] <SpecialK|Canon> xnox: buildd farm should be healthy (minus a few builders), though my apologies for not reporting the previous issues here (but they should have been primarily about throughput)
[16:49] <SpecialK|Canon> xnox: are you experiencing current issues?
[16:54] <xnox> SpecialK|Canon: all of CPC, Foundations, Server, Desktop team developers are continiously experiencing lack of communication "is it a know throughput issue, or should it be reported, and a way to check the status of it"
[16:54] <xnox> SpecialK|Canon:  there are 9.5% builders down, where is the status about that?
[17:01] <SpecialK|Canon> xnox: this is the first I've heard of that being an issue; I'll reach out to ensure we can deal with that
[17:04] <SpecialK|Canon> xnox: the lcy builders being down is now in the topic, as you've seen, and while I'd like them back, they're 26 builders out of 262; I appreciate we've got a backlog of 30k jobs from the rebuild and that more builders always helps, but fundamentally there's 30k jobs
[17:05] <SpecialK|Canon> (the majority of which are from the test rebuild and should be appropriately scored to not block time-critical builds)
[17:16] <xnox> SpecialK|Canon:  "this is the first I've heard of that being an issue" => this year doko/vorlon/I/etc had to continiosuly notice, point out, report, and aska bout stuck builders.
[17:17] <xnox> SpecialK|Canon:  it's not past the point of being concern, and will now be turning into request/requirement, for LP team to self-monitor automatically, have alerts, and up to data status data about buildd farm issues.
[17:17] <xnox> SpecialK|Canon:  thanks for the status update in topic.
[17:18] <xnox> maybe prepend it with "Known issues: "?
[17:18] <xnox> such that once lcy builders are resolved it can be changed to "Known Issues: none, please report"
[17:25] <SpecialK|Canon> xnox: If I may direct your attention to our (internally published) roadmap, featuring our plans to improve our metrics infrastructure, for purposes such as this
[17:27] <SpecialK|Canon> xnox: I hope it comes as no surprise when I say that we would very much like to be in a better position there
[17:29] <SpecialK|Canon> xnox: And that I would much rather hear from a Nagios alert than you
[17:29] <SpecialK|Canon> *of issues
[17:30] <SpecialK|Canon> xnox: Launchpad's build infrastructure is unfortunately necessarily quite complex, with interesting security/privacy requirements
[17:34] <SpecialK|Canon> We're already doing what we can to improve the monitoring and tooling there
[17:43] <SpecialK|Canon> (Incidentally if you'd like to follow along at https://code.launchpad.net/launchpad you'll note that build farm improvements form a significant chunk of our ongoing work)