[06:42] <RikMills> packages built 7 hours ago, still not published. idle x86 builders, but 44 jobs in the queue.
[06:43] <RikMills> not nagging. I don't envy LP engineers sometimes!
[06:44] <wgrant> Those 44 jobs are stuck because the publisher is, let's see
[07:45] <oSoMoN> good morning launchpad!
[07:45] <oSoMoN> I filed bug #1836010 re- the question I asked yesterday about webhooks
[08:44] <cjwatson> oSoMoN: Confused, because I already gave you the existing bug yesterday, didn't I?
[08:44] <cjwatson> I guess this is slightly different
[08:58] <oSoMoN> cjwatson, sorry I must have missed your message yesterday
[08:59] <oSoMoN> what was the existing bug?
[09:03] <cjwatson> You referred to it actually
[09:03] <cjwatson> I've just added a backlink from https://bugs.launchpad.net/launchpad/+bug/1638333
[09:08] <oSoMoN> cjwatson, thanks. I guess I should have started by looking for existing bugs before asking my question yesterday
[09:09] <oSoMoN> cjwatson, does the use case make sense?
[09:11] <cjwatson> It's reasonable, just not sure when we'd fit it in.  (Though it's probably not completely inaccessible for a new developer)
[10:01] <Nakato> Is there anyone here able to help me out with launchpad erroring out on a bug submission?
[10:02] <cjwatson> Nakato: Do you have an OOPS ID?
[10:02] <Nakato> cjwatson:  OOPS-be239c7023dce658b606040ab1522a2f
[10:03] <cjwatson> Nakato: There's no source package in Ubuntu called "linux-snap"
[10:04] <Nakato> Oh, okay, where can I file a bug for https://git.launchpad.net/~ubuntu-kernel/ubuntu/+source/linux-snap/+git/xenial/tree/Makefile ?
[10:04] <cjwatson> Nakato: I don't know, you'll have to ask the kernel team
[10:04] <ginggs> Is there a known problem uploading to PPA?  I keep getting [Errno 104] Connection reset by peer
[10:04] <cjwatson> Nakato: Try #ubuntu-kernel
[10:05] <cjwatson> ginggs: Could be related to the rollout we did to make the uploader IPv6-ready I suppose
[10:05] <cjwatson> ginggs: Are you using FTP?
[10:05] <ginggs> cjwatson: yes, using ftp
[10:10] <cjwatson> ginggs: Can you try stracing it from your end?  Since it's FTP the trace shouldn't contain any secrets
[10:10] <cjwatson> ginggs: strace -f -s 1024 -tt
[10:16] <ginggs> cjwatson: it seems to be disconnected after PASV
[10:16] <ginggs> let me know if i should send the trace somewhere
[10:18] <cjwatson> ginggs: cjwatson@canonical.com please
[10:19] <cjwatson> ginggs: In the meantime you should be able to use SFTP
[10:20] <Nakato> Is it a bug that I was able to get to that bug-submission page that errored out?  I started at this url: https://code.launchpad.net/~ubuntu-kernel/ubuntu/+source/linux-snap/+git/xenial then clicked "bugs" in the title, then "Report a bug".
[10:23] <cjwatson> Nakato: Yes, it's https://bugs.launchpad.net/launchpad/+bug/1635118
[10:24] <Nakato> Thanks cjwatson
[10:24] <cjwatson> The fix that caused it was for a more important problem, so it needs some more digging rather than just reverting the change that introduced it
[10:53] <ginggs> cjwatson: SFTP worked, thanks
[10:56] <cjwatson> ginggs: strace> interesting - I have a nearly identical strace here except I get a proper response to PASV and everything works fine
[10:59] <cjwatson> ginggs: What time zone is this strace in?
[11:00] <ginggs> south africa, GMT +2
[11:00] <cjwatson> Thanks, even after all this time my interhemisphere timezone guessing still sucks
[11:06] <RikMills> normal ftp stalled here as well. sftp succeeded
[11:31] <cjwatson> I really don't understand this - it works fine for me, with the company VPN disabled
[11:32] <cjwatson> Try "telnet ppa.launchpad.net 21", then "USER anonymous", "PASS testing", "PASV" and see what you get?
[11:44] <ginggs> cjwatson: Connection closed by foreign host.
[11:44] <cjwatson> What the heck is going on
[11:45] <cjwatson> ginggs: Could I see the full transcript of that telnet session?
[11:45] <ginggs> cjwatson: so at home, earlier today, dput was just stalling and never timing out. now at the office i'm getting the connection closed
[11:45] <cjwatson> Including the "Trying ..." line at the start
[11:46] <ginggs> https://paste.ubuntu.com/p/pwDvKDkWbk/
[11:47] <cjwatson> For me it's fine from home and from chiark, hangs from master.debian.org
[11:50] <GyrosGeier> if I send "QUIT" right after PASV, I get 550 Requested action not taken: internal server error
[11:50] <GyrosGeier> 221 Goodbye.
[11:50] <GyrosGeier> so two responses to one command
[11:54] <cjwatson> GyrosGeier: FTP permits multiple replies
[11:54] <cjwatson> GyrosGeier: See RFC 959 section 4.2
[11:54] <cjwatson> GyrosGeier: "Every command must generate at least one reply, although there may be more than one"
[11:57] <cjwatson> GyrosGeier: I don't know if it's correct in this case, but it seems to be permitted in general, even though it's bizarre.  But presumably in your case PASV had already produced a response?
[12:23] <cjwatson> ginggs: Would you mind filing a bug on https://bugs.launchpad.net/txpkgupload about this?  I managed to reproduce it in an environment where I might be able to debug it
[12:23] <cjwatson> Maybe
[12:24] <ginggs> cjwatson: doing
[12:25] <cjwatson> Also confirmed that reverting the recent deployment fixes it, so now just to work out why
[12:29] <GyrosGeier> cjwatson, yes, it did
[12:29] <GyrosGeier> there was 30 seconds pause after the PASV, which got a positive reply
[12:30] <ginggs> cjwatson: LP: #1836059
[12:30] <GyrosGeier> so I'd say the error was definitely in response to the QUIT
[12:31] <cjwatson> ginggs: thanks
[12:32] <ginggs> cjwatson: np
[12:38] <cjwatson> Huh, running txpkgupload on a different port makes it go away
[12:38] <cjwatson> I wonder if we're managing to confuse some conntrack nonsense or something
[12:39] <cjwatson> Oh, or maybe the problem is a leading ::ffff: in the PASV response
[13:50] <cjwatson> ginggs,RikMills,GyrosGeier: Worked around for now
[13:51] <RikMills> ty
[15:40] <ginggs> cjwatson: workaround working, ta!