[12:14] <mdeslaur> been hitting a _lot_ of timeout errors in launchpad for the past 2 days...for example OOPS-e9f1c257441a1b45a234846b52494b5d
[12:27] <cjwatson> I've never properly understood those BugTaskFlat update OOPSes; they spike from time to time and then go away.  They show up on our 5xx graphs, but not usually so high as to indicate a major outage.  In the past I believe they've been correlated with large amounts of DB replication traffic, but there's nothing particularly obvious on the bandwidth graphs.
[12:28] <cjwatson> (for the DB servers)
[13:12] <kyrofa> Can someone explain how the snap builders upload to the store? I see completed builds, but no new revisions in the store
[13:16] <kyrofa> cjwatson, any idea how to debug that? I just get http://pastebin.ubuntu.com/17641432/ at the end... no "uploading to the store" type messages like I expected
[13:21] <cjwatson> kyrofa: Can you give me the URL to the snap, please?
[13:21] <cjwatson> kyrofa: The builder isn't the part that does the store upload, so you won't see anything in the build log about it.
[13:21] <kyrofa> cjwatson, ah, okay. You mean this URL? https://code.launchpad.net/~kyrofa/+snap/nextcloud
[13:22] <kyrofa> cjwatson, also, an important note: I don't own the snap in the store, but it was shared with me so I can publish to it
[13:22] <kyrofa> Perhaps that is a problem
[13:23] <cjwatson> It is quite possible.  Right now it's nearly impossible to debug this; I happen to be working on changes right now to make it easier.
[13:24] <kyrofa> cjwatson, okay, no big deal, I can download and publish myself, but getting this working would be great!
[13:24] <cjwatson> Right, sorry, all I can tell right now is that we tried to upload it, and it's either been rejected by the store or there's still a scan in progress; I can't tell which.
[13:25] <cjwatson> It's probably more productive for me to continue to work on the changes to make this more robust and visible than to try to dig through store logs to figure out what happened in this particular case.
[13:25] <cjwatson> If you have access to our Trello board, this is "Fix SnapStoreUploadJob to check that SCA's scan succeeds" on https://trello.com/c/ph5dn8hx/286-polishing-of-launchpad-snap-builds-to-the-store
[13:27] <kyrofa> cjwatson, sounds good, thanks!
[13:35] <kyrofa> cjwatson, what's the plan there, by the way? Another log for uploading?
[13:35] <cjwatson> kyrofa: You mean in terms of UI presentation?
[13:35] <kyrofa> cjwatson, right
[13:36] <cjwatson> kyrofa: A log is overkill; it'll be either an error message or a URL to the package in the store, depending on scan status.
[13:36] <cjwatson> (Or a pending indication.)
[13:36] <cjwatson> Maybe in future an option to retry the upload, although probablynot in the first iteration.
[13:36] <cjwatson> *probably not
[13:37] <kyrofa> cjwatson, yeah you're right. Would it email? Or just show an error on the build page?
[13:37] <cjwatson> I wrote the email code earlier today.
[13:37] <cjwatson> I'm writing the bits to show it on the build page now.
[13:40] <kyrofa> Alright great :)
[13:41] <kyrofa> cjwatson, in other news, what's up with the arm builders? I've been waiting over an hour and my build hasn't even started yet for armhf or arm64
[13:41] <kyrofa> cjwatson, is there a problem, or are they just in high demand?
[13:44] <cjwatson> kyrofa: There's a test rebuild in progress at the moment.
[13:44] <kyrofa> Ah ha
[13:44] <cjwatson> kyrofa: But actually, this is probably virtualised, right?
[13:45] <cjwatson> kyrofa: In which case I think I can now do a bit of rebalancing to help.
[13:45] <kyrofa> cjwatson, I'm not sure how it's setup, I just checked those archs in the snap config
[13:45] <cjwatson> kyrofa: I reassigned eight of the virtualised ARM builders to play non-virtualised builders on TV last week so that the test rebuild would finish quicker, but I can reverse that now.
[13:45] <kyrofa> Hahaha
[14:16] <cjwatson> kyrofa: Queues should be clearing out now.  (Sorry for delay, was called away.)
[14:16] <kyrofa> cjwatson, yay! No problem, looks like they're building now, thanks!
[15:06] <cjwatson> kyrofa: https://code.launchpad.net/~cjwatson/launchpad/snap-upload-poll-status/+merge/298030 just FYI
[15:07] <kyrofa> cjwatson, awesome, thank you!
[15:34] <free`> cjwatson: hey there, I'd need to rename a new project that I created a few days go. Should I file an LP question for that?
[15:42] <cjwatson> free`: yes
[15:52] <free`> cjwatson: cheers
[15:57] <free`> cjwatson: are questions all public and visible by anybody?
[15:58] <cjwatson> free`: Indeed, questions have no privacy support.  If it relates to a private thing you can ask in #launchpad-ops internal (or #webops if it's something that I'm not going to have access to see)
[16:03] <free`> cjwatson: gotcha
[18:02] <ayan> is there documentation on how to create an ad hoc git repository on launchpad under https://code.launchpad.net/~user/+git ?
[18:03] <ayan> i opened lp1590560 to add this to the git repository listing similar to the bzr instructions on the https://code.launchpad.net/~user page.
[18:03] <dobey> add a remote for ~/+git/repository and then push there
[18:04] <dobey> https://help.launchpad.net/Code/Git
[18:15] <ayan> dobey: ah!  awesome.  maybe a link to that page should be included on the code page?
[18:20] <dobey> maybe
[18:36] <ayan> okay -- i asked about it, was told to open a bug, so there it is.