[10:33] <rbasak> If I give https://api.launchpad.net/devel/ubuntu/+archive/primary/+sourcepub/752367 to ubuntutools.archive.UbuntuSourcePackage and use its pull method, it tries (based on my debugging) to fetch https://launchpad.net/ubuntu/+archive/primary/+files/akonadi_1.1.2.orig.tar.gz but then raises:
[10:33] <rbasak> ubuntutools.archive.DownloadError: File akonadi_1.1.2.orig.tar.gz could not be found
[10:33] <rbasak> wget on that URL gives me a 303 to https://launchpadlibrarian.net/26538825/akonadi_1.1.2.orig.tar.gz that appears to work.
[10:33] <rbasak> spphr.packageupload.sourceFileUrls() points straight to the launchpadlibrarian.net URL and appears to work.
[10:33] <rbasak> Is the same bug that caused us to create a "pull overrides" mechanism in git ubuntu so that we could override particular specific URLs? I could add an entry for this one that I think would work, but I wanted to check it's not a new different issue.
[10:33] <rbasak> I could investigate to see why ubuntutools.archive can't download from that URL but wget can.
[10:34] <rbasak> But I thought I'd ask first as I think Colin might know the answer already?
[10:34] <rbasak> Possibly it's the redirect? But why is akonadi 1.1.2 special, when the importer generally works for almost all other package publications?
[10:47] <cjwatson> rbasak: UbuntuSourcePackage('akonadi', '1.1.2-1ubuntu1').pull() worked for me.  Could you give me some kind of reduced sample code that reproduces the problem?
[10:48] <cjwatson> (UbuntuSourcePackage doesn't seem to take API URLs)
[10:50] <cjwatson> The one thing I know about is that ubuntutools.archive.*SourcePackage would be slightly better off using the new +sourcefiles traversals, which avoid ambiguity in some cases.  But I'm not convinced that's relevant here
[10:50] <cjwatson> (In particular it should only be relevant to the Debian import, not to Ubuntu)
[11:01] <rbasak> Yes I'll try and reduce it to not use any gitubuntu bits
[11:08] <rbasak> cjwatson: UbuntuSourcePackage('akonadi', '1.1.2-0ubuntu1~jaunty1').pull()
[11:08] <rbasak> but
[11:08] <rbasak> UbuntuSourcePackage('akonadi', '1.1.2-0ubuntu1').pull() fails for me
[11:09] <rbasak> Maybe it's the version of ubuntutools
[11:10] <rbasak> 0.161 on Bionic seems to fail
[11:11] <rbasak> Using Python3
[11:11] <cjwatson> Reproduced.  In meetings, will look a bit later
[11:12] <rbasak> Thanks!
[11:12] <cjwatson> "Error: Checksum for akonadi_1.1.2.orig.tar.gz does not match." is almost certainly more relevant than the "could not be found" lie
[13:08] <cjwatson> rbasak: So in fact it is the thing I first thought of: somehow we've ended up with .orig mismatches between two different versions of akonadi in Ubuntu (presumably a historical bug).  In https://code.launchpad.net/~cjwatson/launchpad/archive-unambiguous-files-traversals/+merge/345118 I added a URL variation that lets us cope with this.  Would you mind trying the lightly-tested ...
[13:08] <cjwatson> ... https://paste.ubuntu.com/p/npjZbpMFVD/ and seeing if that fixes things for you?
[13:08] <cjwatson> rbasak: (akonadi 1.1.2-0ubuntu1~jaunty1 and 1.1.2-1ubuntu1 work with that.  1.1.2-0ubuntu1 fails because there was no such version of akonadi in Ubuntu.)
[13:46] <rbasak> Thanks - I'll try that with my full failure scenario. It will take a while.
[13:49] <cjwatson> I don't know what your "pull overrides" mechanism was in response to - it may have been the same thing.
[13:50] <rbasak> An example is ipvsadm 1:1.24-2 - https://launchpadlibrarian.net/15550377/ipvsadm_1.24.orig.tar.gz
[13:50] <rbasak> Where the - is some kind of None sentinel value for the dsc
[13:51] <rbasak> Which I think means "when fetching ipvsadm 1:1.24-2 then use https://launchpadlibrarian.net/15550377/ipvsadm_1.24.orig.tar.gz instead of the normal route to get that file"
[13:56] <cjwatson> rbasak: I think it's likely that you needed that because https://launchpad.net/ubuntu/+source/ipvsadm/1.24-1 and https://launchpad.net/ubuntu/+source/ipvsadm/1:1.24-2 have .orig.tar.gz files with the same filename but different contents.  The patch above would deal with that too.
[13:56] <cjwatson> i.e. it's the same thing
[13:56] <rbasak> OK thanks
[13:57] <rbasak> What I might do then is confirm this patch fixes it, and if so retry the imports that we have the overrides for without the overrides. If it's good then we could drop the overrides
[13:58] <cjwatson> If it works then I'll propose it as an MP to ubuntu-dev-tools
[13:58] <rbasak> Sounds good
[13:59] <cjwatson> I was a bit more pedantic about quoting as well, which could conceivably help in some cases
[15:03] <rbasak> cjwatson: I'm struggling to test this in a useful way. git-ubuntu is bound to Xenial or Bionic because we distribute it as a snap. Until there's a Core 20. Your patch doesn't apply cleanly against Bionic's ubuntu-dev-tools. I tried backporting Focal's ubuntu-dev-tools but that led me to a debhelper rabbit hole.
[15:03] <rbasak> Maybe we should land your patch if it fixes my reproducer case on Focal though? But you've already tested that presumably. I can double check if you like.
[15:05] <rbasak> In the meantime I can use pull overrides for git ubuntu.
[15:05] <rbasak> But it'd be nice to set things in motion so that one day we can drop them so I'm in favour of your change to ubuntu-dev-tools in principle certainly.
[15:06] <rbasak> Or we could backport your patch properly to Bionic and either SRU or apply during git ubuntu's snap build.
[15:09] <cjwatson> I tested it in-tree with python3 on bionic.  Backporting the whole package indeed isn't where I'd start.  Let me see if I can produce a targeted patch.
[15:11] <rbasak> BTW, Bionic is just my git ubuntu testing environment
[15:11] <rbasak> The actual runtime environment is more complicated. IIRC it's Core "16" based because Core 18 didn't exist at the time
[15:12] <rbasak> Ah, and it's using 0.161 from a tarball
[15:12] <cjwatson> Which is slightly pre-bionic
[15:12] <cjwatson> Probably no particular reason, just timing?
[15:12] <rbasak> Probably yes
[15:13] <rbasak> I can probably bump that up to a tarball based on Focal (or a future Focal) quite easily.
[15:14] <rbasak> I can test with your patch added to Focal's version and a "custom" tarball. I just have to build a snap to do it.
[15:14] <cjwatson> https://paste.ubuntu.com/p/jmpMvmbNbM/ is an untested backport to bionic
[15:14] <rbasak> Thanks!
[15:14] <rbasak> Let me try that first. That should be quick.
[15:15] <cjwatson> Seems not totally broken at least
[15:19] <rbasak> I think that applies against /usr/lib/python3/dist-packages/ubuntutools/archive.py on Bionic
[15:20] <rbasak> Running a test import of akonadi now
[15:20] <rbasak> 01/21/2020 15:20:14 - ERROR:stderr: pristine-tar: /home/ubuntu/git-ubuntu/akonadi/.git/git-ubuntu-cache/akonadi_1.1.2.orig.tar.gz does not match stored hash (expected 8c05a4f154fd41d6aa01145c58537f67dbe86a6666873cd58d765566167234f9, got 62fd42b603b8542b8cd1305d9140224b3f56445e93af6a9c1cfae1d4119d42a3)
[15:20] <rbasak> I think that means it worked
[15:20] <rbasak> (it's expected that the pristine-tar branch has an undefined orig tarball)
[16:05] <rbasak> The imported completed successfully. Thanks!
[16:05] <rbasak> I'm going to create an LP bug to track this so I can reference it from various places
[16:08] <cjwatson> Cool, let me know the bug number and I'll attach it to MPs
[16:38] <rbasak> cjwatson: for now, https://bugs.launchpad.net/ubuntu/+source/ubuntu-dev-tools/+bug/1860456; I'll write up details later.
[16:38] <ubot5> Ubuntu bug 1860456 in ubuntu-dev-tools (Ubuntu) "ubuntutools.archive.UbuntuSourcePackage().pull() fails" [Undecided,New]
[16:41] <cjwatson> thanks