[09:18] <tintou_> I've filled an issue in Launchpad itself a few weeks ago, but would it be possible to reassign all the signon-* and libaccounts* projects from https://launchpad.net/online-accounts to https://launchpad.net/~accounts-sso-devs which contains the team from https://gitlab.com/accounts-sso
[10:07] <cjwatson> tintou_: Could you point me to the issue you filed?
[10:20] <tintou_> cjwatson: arf, sorry for the noise, I've probably used the "contact this team admins" here instead https://launchpad.net/~pspmteam
[10:22] <cjwatson> That is ... probably not very useful
[10:23] <rbasak> Launchpad is building git-ubuntu snaps broken currently (we embed tests and they fail). I haven't been able to reproduce locally - my snap builds from the same source tree seem to reliably work.
[10:23] <cjwatson> tintou_: Could you file a ticket on https://answers.launchpad.net/launchpad/+addquestion ?  We may need to consult a bit.
[10:23] <rbasak> Do you have any details on how I could reproduce Launchpad's builds exactly please?
[10:23] <rbasak> Separately I'm trying to follow up by investigating exactly how the Launchpad built snap is broken.
[10:24] <cjwatson> rbasak: I guess it would be necessary to set up launchpad-buildd in a VM.  It's a bit of a long tail to get to "exactly".
[10:25] <rbasak> I'm using "snapcraft cleanbuild" locally
[10:25] <wgrant> rbasak: Have you tried building it in a buildd chroot?
[10:25] <wgrant> cleanbuild does not currently use a properly clean chroot AFAIK
[10:25] <rbasak> wgrant: no. That sounds like a good idea.
[10:26] <wgrant> It probably uses something big based on a cloud-image.
[10:26] <rbasak> I'm seeing this, which I suppose is expected:
[10:26] <rbasak> -prefix=/root/build_git-ubuntu/stage/usr
[10:26] <rbasak> +prefix=/build/git-ubuntu/stage/usr
[10:26] <rbasak> But also this, which I'm more surprised to see
[10:26] <rbasak> -CPPFLAGS=      -I. -IInclude -I$(srcdir)/Include  -I/root/build_git-ubuntu/stage/usr/include -I/root/build_git-ubuntu/stage/usr/include/x86_64-linux-gnu
[10:26] <rbasak> +CPPFLAGS=      -I. -IInclude -I$(srcdir)/Include  -I/build/git-ubuntu/stage/usr/include
[10:27] <rbasak> LDFLAGS not affected though
[10:28] <tintou_> cjwatson: question opened https://answers.launchpad.net/launchpad/+question/670044 thanks :)
[10:28] <wgrant> rbasak: I'd suggest building in a chroot that contains a sane set of packages (eg. chroot_url from https://api.launchpad.net/devel/ubuntu/xenial/amd64)
[10:28] <wgrant> It will probably tell you which dep you're missing
[10:29] <rbasak> wgrant: so put that in schroot or a container or similiar, apt install snapcraft and run that inside it?
[10:30] <cjwatson> LP converts it to a container, but that's probably only necessary if you build-depend on snaps.
[10:31] <wgrant> Indeed.
[10:31] <rbasak> OK I'll try that thanks
[10:31] <rbasak> (using schroot)
[10:31] <wgrant> tintou_: Can you explain the justification and history in the ticket?
[10:32] <wgrant> It's very exceptional for us to reassign ownership of projects like this.
[10:32] <wgrant> We'll need to do some research.
[10:39] <rbasak> Is it the buildd user I should run snapcraft against?
[10:39] <tintou_> I just added more information, but yeah I understand that it might require to ask people for the feasibility :)
[10:56] <rbasak> Hmm. snapcraft seems to want sudo.
[10:56] <cjwatson> rbasak: We currently run snapcraft as root.
[10:57] <cjwatson> https://bugs.launchpad.net/launchpad-buildd/+bug/1702656
[10:57] <ubot5> Ubuntu bug 1702656 in launchpad-buildd "Run snapcraft as non-root (with passwordless sudo)" [High,Triaged]
[10:57] <rbasak> Ah OK. Easier for me to replicate then, thanks :)
[10:58] <rbasak> Running snapcraft now. I did it in a plain chroot as I realised I'm doing this all in a test VM anyway
[11:00] <cjwatson> Yeah, that's basically what Launchpad did until we switched to a container.
[13:22] <rizzitello> Hi all im working on a project and we are in current discussion to add submodules to our project. How is this going to affect ubuntu builds? last I tried launchpad didn't work right with git repos with submodules . Is there a workaround or has this been fixed ?
[13:25] <cjwatson> rizzitello: What sort of builds?  Source package recipes, snaps, ...?
[13:25] <rizzitello> packages
[13:26] <cjwatson> Sounds like https://bugs.launchpad.net/git-build-recipe/+bug/1733603, which we haven't got round to fixing yet
[13:26] <ubot5> Ubuntu bug 1733603 in git-build-recipe "recipe builds for git projects don't work with submodules" [Undecided,New]
[13:26] <cjwatson> git-build-recipe is open source, like the rest of Launchpad; I'd be happy to review a tested patch to it
[13:27] <rizzitello> source repo?
[13:27] <rizzitello> and how best for me to test my patch?
[13:27] <cjwatson> https://git.launchpad.net/git-build-recipe
[13:28] <cjwatson> "python3 setup.py test" should work
[13:29] <rizzitello> any pointers on where to squash this bug?
[13:30] <cjwatson> That's most of the work; if I had an immediate answer to that I'd already have fixed the bug :)
[13:30] <rizzitello> lol ok just trying to save anytime on the search :P
[13:31] <cjwatson> But it's quite a small project, only a couple of thousand lines of code outside tests
[13:32] <cjwatson> Maybe pull_or_clone, or somewhere near there
[13:32] <rizzitello> ok thanks cjwatson ill see what i can do ..
[13:32] <cjwatson> Cheers
[13:52] <rbasak> I've not been successful at reproducing the problem.
[13:52] <rbasak> Now that I've got things close, I'll see if a diff against the Launchpad build log gives me anything.
[14:03] <rbasak> Could there be some different in network environment perhaps? Eg. pip falling back to something if it can't fetch from pypi?
[14:05] <rbasak> That kind of thing anyway - LP's build log shows pip working normally AFAICT
[15:01] <cjwatson> rbasak: There is certainly such a difference - LP builders run with very constrained network access and have an authenticated outbound proxy
[15:02] <cjwatson> rbasak: This is regrettably quite hard to reproduce exactly locally
[15:13] <rbasak> Thanks. I'm trying to pin it down from the other direction.
[15:13] <rbasak> ImportError: /snap/git-ubuntu/x8/lib/python3.6/site-packages/../../../usr/lib/x86_64-linux-gnu/../libgcrypt.so.20: symbol gpgrt_get_syscall_clamp, version GPG_ERROR_1.0 not defined in file libgpg-error.so.0 with link time reference
[15:13] <rbasak> That's the actual specific problem.
[15:13] <rbasak> I'm building both libgcrypt.so.20 and libgpg-error.so.0 as part of the snap build, from upstream source
[15:13] <rbasak> I don't see that it has appeared in my local chroot via snapcraft installing deps before it begins
[15:14] <rbasak> Any chance it could arrive in the chroot on Launchpad in some other way?
[15:14] <rbasak> Another difference is that on Launchpad snapcraft is using make -j4
[15:15]  * rbasak looks up objdump syntax
[15:28] <cjwatson> rbasak: launchpad-buildd also installs   snapd fuse squashfuse udev (you may have difficulty installing these ones in a chroot, but check their deps) python3 socat snapcraft, and either bzr or git depending on what sort of source for the snap you gave it
[15:29] <cjwatson> reasonably believable that something there could pull in gpg-error
[15:32] <rbasak> Hmm.
[15:32] <rbasak> On the Launchpad build, that symbol is missing. On my working build, the symbol is presne.t
[15:32] <rbasak> That narrows it down at least.
[15:33] <cjwatson> You should be able to see what packages are being installed in the build log on each side
[15:33] <rbasak> ack
[15:34] <cjwatson> Also make sure sources.list is the same (you can see that near the top of the LP build log, in one of its initialisation commands)
[15:34] <cjwatson> override-sources-list
[15:34] <cjwatson> You'll have to replace "ftpmaster.internal" with "archive.ubuntu.com" but otherwise should be able to use it intact
[15:43] <cjwatson> (It is just about technically possible to set this all up locally, aside maybe from the strange DNS view hacks that I've never seen directly.  But having been beating my head against it for the last two days I can't yet say I'd recommend trying it.)
[15:44] <rbasak> I'm starting to suspect that libgpg-error has a bug related to concurrent builds
[15:44] <rbasak> The configure output is identical
[16:00] <nacc> rbasak: but the issue we're seeing is unrelated to the symbol being present or not, it's the rpath thing snapcraft does
[16:01] <nacc> rbasak: that is, check if the libgpg-error.so has the symbol in the snap (not the one used by _pygit2). if it does, the build of libgpg-error isn't the problem.
[16:02] <nacc> rbasak: if it does, check the `ldd` output for _pygit2, you'll see in the working cases it uses our snap's build. In the failing cases it uses the core snap's. I pinged kyrofa on this, and he said they are working on making the build/stage order determinate
[16:02] <nacc> not sure if that will just break us everywhere or fix us everywhere, tbh
[16:03] <rbasak> nacc: inside snap run, if I run python3, then "import pygit2" fails
[16:03] <rbasak> With ImportError: /snap/git-ubuntu/x8/lib/python3.6/site-packages/../../../usr/lib/x86_64-linux-gnu/../libgcrypt.so.20: symbol gpgrt_get_syscall_clamp, version GPG_ERROR_1.0 not defined in file libgpg-error.so.0 with link time reference
[16:04] <rbasak> /snap/git-ubuntu/x8/lib/python3.6/site-packages/../../../usr/lib/x86_64-linux-gnu/../libgcrypt.so.20 is inside the snap
[16:04] <rbasak> ldd gives me ldd /snap/git-ubuntu/x8/lib/python3.6/site-packages/../../../usr/lib/x86_64-linux-gnu/../libgcrypt.so.20
[16:04] <rbasak> 	libgpg-error.so.0 => /snap/core/current/lib/x86_64-linux-gnu/libgpg-error.so.0 (0x00007f348d9e3000)
[16:05] <nacc> right the last bit is the bug
[16:05] <rbasak> And libgpg-error.so.0 doesn't have that symbol in a broken snap
[16:05] <rbasak> But in my local build, it does
[16:05] <nacc> it's the *wrong* libgpg-error
[16:05] <rbasak> Oh.
[16:05] <rbasak> Staging order?
[16:05] <nacc> not libgcrypt, libgpg-error
[16:05] <nacc> yeah, i *think* so, but that's what my commit was supposed to fix
[16:05] <nacc> so it's somethign else, but heck if i know wht
[16:06] <nacc> i did see this from the launchpad build logs:
[16:06] <nacc> Copying needed target link from the system
[16:06] <nacc>              /lib/x86_64-linux-gnu/libgpg-error.so.0.17.0
[16:06] <nacc> i think "system" here is the core snap, but i'm not sure
[16:07] <nacc> snapcraft does some weird stuff about library resolution, and that was something we had to fix when we reverted snapcraft before
[16:07] <nacc> because it did it "wrong" :)
[16:08] <nacc> rbasak: can you confirm in the broken snap, the libgpg-error.so *in* the git-ubuntu snap does have the symbol?
[16:08] <rbasak> So in the "good" build tree, I don't see the missing symbol in ./parts/git-ubuntu/install/lib/x86_64-linux-gnu/libgpg-error.so.0, but I do see it in ./parts/libgpg-error/install/usr/lib/libgpg-error.so.0.22.0
[16:09] <rbasak> And it's also in ./prime/usr/lib/libgpg-error.so.0.22.0 in the good snap
[16:09] <nacc> hrm, yeah, the 'lib/x86_64-linux-gnu bit should get pruned by our yaml
[16:09] <rbasak> In the bad snap, the final thing doesn't have the symbol, which is presumably equivalent to what was in prime in the build tree
[16:10] <rbasak> So that's the difference I think. Same snapcraft.yaml, same version of snapcraft.
[16:10] <nacc> rbasak: line 395 of the yaml, iirc
[16:12] <rbasak> OK. So if I can get that line working, the problem should be solved
[16:12] <rbasak> lib/x86_64-linux-gnu/libgpg-error.so.0 vs. usr/lib/*/*gpg-error*
[16:12] <rbasak> Perhaps the globbing doesn't catch the multiarch directory
[16:14] <nacc> line 395 should be          - -lib/*/*gpg-error*
[16:15] <nacc> i think it does, because we don't end up with two libgpg-error in the snap
[16:15] <nacc> the problem is that possibly the timing is off as to when snapcraft does it's rpath search
[16:15] <nacc> i think we'd need kyrofa to help with that, he figured this out last time :)
[16:17] <rbasak> If it is ignoring it, then how is it that the "bad" libgpg-error.so.0* ends up in the snap?
[16:18] <rbasak> Or are you saying that it ignores the ignore for the purposes of this rpath search?
[16:18] <nacc> rbasak: afaik, the bad libgpg-error never ends up in the snap
[16:19] <nacc> it's the linktime reference that is bad
[16:19] <nacc> which i believe snapcraft influences by editing the rpath in the library itself, but i'm not sure (they do soe patchelf voodoo)
[16:19] <rbasak> I can tell whether a snap is good or bad by examining the symbols in libgpg-error.so.*. Would you expect that?
[16:19] <nacc> rbasak: *which* libgpg-error.so ?
[16:20] <rbasak> The one in usr/lib in the snap squashfs
[16:21] <rbasak> Perhaps we should move to #snappy
[16:21] <nacc> yeah we can
=== beisner-sick is now known as beisner