[00:00] <cjwatson> infinity: I think actually three busted nodes.  The thing with Cleaning is that if a lot of instance resets are sitting there for minutes failing to boot, then they can starve the instance reset workers of resources.
[00:01]  * infinity nods.
[00:01] <infinity> cjwatson: Has anyone managed to come up with a reproducer that we can throw at IBM and say "fix, pls"?
[00:02] <cjwatson> Not to my knowledge.
[00:02] <infinity> cjwatson: Like some sort of "run qemu in a loop really fast and watch the world burn" thing?
[00:02] <teward> infinity: are autopkgtests taking eons or is it just me?
[00:02] <cjwatson> I believe William tried that kind of thing and failed to come up with a reproducer.
[00:02] <infinity> teward: Everything is taking eons, welcome to autosync.
[00:02] <cjwatson> teward: Look at the queue lengths on http://autopkgtest.ubuntu.com/running.shtml
[00:02] <teward> heheh
[00:03] <infinity> teward: You'd be better off ignoring reports and migrations and such for a couple of days, it's lower stress that way.
[00:03] <teward> cjwatson: eww
[00:03] <cjwatson> Don't expect stuff that lands on the end of it to come out very soon
[00:03] <teward> infinity: ack, thanks for accepting the SRU (with the pending yakkety-proposed -> yakkety migration being stuck)
[00:03] <infinity> teward: yakkety'll get there eventually.  Just stop babysitting and find something more fun to do. :)
[00:03] <cjwatson> I would be astonished if migrations weren't mostly stuck right now. :-)
[00:04] <teward> infinity: was actually just checking is all, though what the running page shows and proposed migration excuses show don't match :p
[00:04]  * teward was just curious
[00:04] <teward> I was more concerned about that SRU than I was yakkety, 'cause Yakkety will land eventually :p
[00:04] <infinity> teward: excuses shows the state as of the last time it was generated.  Which was a while ago.
[00:04] <infinity> (Because everything is slow right now, blah blah)
[00:06] <teward> heheh
[00:06] <teward> indeed
[00:10] <cjwatson> [6~[6~$ curl -s http://people.canonical.com/~ubuntu-archive/auto-sync/2016-04-26/20:00:02.log | grep Updated: | tr -s ' ' | cut -d' ' -f2
[00:10] <cjwatson> 2824
[00:10] <cjwatson> $ curl -s http://people.canonical.com/~ubuntu-archive/auto-sync/2016-04-26/20:00:02.log | grep Updated: | tr -s ' ' | cut -d' ' -f2
[00:10] <cjwatson> 2824
[00:12] <cjwatson> Sigh, sorry for the dup
[01:53] <Laney> xnox: I should say it's because there are NBS packages in yakkety which are bad, for example libboost-context-dev.
[08:41] <mapreri> infinity: ! thank you ♥
[09:18] <xnox> Laney, found it
[09:19] <xnox> doko, pitti - please cleanup NBS of boost-defaults on s390x, as in please remove
[09:19] <xnox> libboost-coroutine-dev 1.58.0.1ubuntu1 s390x
[09:19] <xnox> libboost-context-dev 1.58.0.1ubuntu1 s390x
[09:19] <pitti> xnox: hm, that's not on http://people.canonical.com/~ubuntu-archive/nbs.html ?
[09:20] <xnox> pitti, it's NBS on some arches only. It is built practically everywhere, apart from a few niche arches.
[09:20] <xnox> pitti, e.g. it is valid on all other arches....
[09:20] <pitti> well yes, but the nbs report is supposed to have that
[09:22] <xnox> pitti, there is libboost-doc 1.58.0.1ubuntu1 yakkety all too
[09:22] <xnox> from rmadison
[09:28] <pitti> xnox: removed
[10:12] <xnox> doko, pitti, infinity - could you please promote libboost-random1.60.0 and libboost-regex1.60.0 to main?
[10:12] <xnox> otherwise ceph will not migrate...
[10:12] <xnox> http://people.canonical.com/~ubuntu-archive/proposed-migration/update_excuses.html#ceph
[10:12] <pitti> ah yes, it's on http://people.canonical.com/~ubuntu-archive/component-mismatches-proposed.txt
[10:13] <doko> publisher runs are slooowwww
[10:13] <pitti> err, wait -- libboost-random1.60.0 does not exist at all in rmadison
[10:14] <pitti> sorry, looked in xenial
[10:14] <xnox> >_<
[10:14]  * xnox was like surely the world would be on fire
[10:14] <pitti> yeah, publishers, testing, everything will be a tarpit for the next days
[10:14] <pitti> rmadison is also dog slow, FWIW
[10:15] <pitti> xnox: so why is change-override yelling at me
[10:15] <pitti> $ change-override -s xenial-proposed -c main libboost-random1.60.0 libboost-regex1.60.0 libdatrie1-udeb libthai-data-udeb libthai0-udeb
[10:15] <xnox> pitti, s/xenial/yakkety/ ?
[10:15] <pitti> don't say anything
[10:15] <pitti> xnox: same error, though
[10:15] <pitti> lputils.PackageMissing: Could not find binaries for 'libboost-random1.60.0/None' in yakkety-proposed
[10:15] <pitti> ah, that's already in yakkety
[10:15] <pitti> this is utterly confusing
[10:16] <pitti> promoting some binaries in -proposed, some in -release, and hoping that LP/britney will DTRT on landing -proposed packages
[10:16] <doko> outdated reports?
[10:17] <pitti> xnox: ok, promoted the lot in y-release; I hope that shines through to yakkety-proposed, let me know if it still doesn't work in an hour or so
[10:17] <xnox> doko, not really. thanks to my patches to britney components missmatches must be satisfied before things migrate.
[10:18] <xnox> thus e.g. one has to promote things in -release, for packages to migrate from -proposed to -release if they pull new things into main.
[10:18] <xnox> pitti, ok, thanks.
[11:05] <mdeslaur> infinity: did anyone answer you yesterday? it's scrolled out of my scrollback buffer, what was it for, apt?
[11:05] <mdeslaur> infinity: oh, looks like you did it, never mind
[11:06] <Odd_Bloke> If someone could migrate gce-utils from (partner) xenial-proposed to xenial, it would be much appreciated. :)
[12:44] <xnox> Err:11 http://ddebs.ubuntu.com yakkety-updates Release
[12:44] <xnox>   404  Not Found
[12:44] <xnox> pitti, ^ is that normal? =)
[12:51] <pitti> xnox: I'm getting similar errors from the ddeb-retriever builders since yakkety's creation; I haven't investigated that yet
[12:51] <pitti> i. e. ddeb-retriever cannot find yakkety-updates indexes on the archive (which I don't understand, they are there)
[12:51] <xnox> ack.
[12:52] <xnox> pitti, ceph looks good now, it now progressed to scheduling autopkgtests....
[12:52] <pitti> xnox: so, on my list, but not top prio
[12:52] <xnox> 600 more armhf builds to go, 2k autopkgtests and can start investigating fall out =)
[12:53] <pitti> 600? that was quick
[12:54] <pitti> xnox: ah, it's more like 1900 :)
[12:54] <xnox> 600 builds to go on https://launchpad.net/builders =) 2k autopkgtests pending ;-)
[12:55] <pitti> oh right, sorry
[12:55] <pitti> xnox: yeah, isn't opening week fun :)
[12:55] <pitti> xnox: we had > 5000 tests per arch this morning, I already short-circuited most of the perl ones
[12:56] <xnox> cheat
[12:56] <pitti> well, it wasn't a new upstream release or so, just a debian rev with three relatively small chagnes; so running only some tests was good enough
[12:57] <pitti> otherwise doko would haunt me for not landing anything at all any more :)
[13:40] <Odd_Bloke> Should amd64/i386 be absent from http://cdimage.ubuntu.com/releases/16.04/release/ ?
[13:41] <xnox> yes
[13:41] <xnox> Odd_Bloke, amd64/i386 is on releases.ubuntu.com
[13:41] <Odd_Bloke> xnox: Of course, how silly of me. ;)
[13:41] <xnox> Odd_Bloke, think [archive|ports].ubuntu.com
[13:43] <Odd_Bloke> xnox: Ack, that's helpful, thanks.
[13:50] <cjwatson> pitti: it occurred to me that the ddeb-retriever errors are maybe it complaining about its own archive, not archive.u.c?
[13:50] <cjwatson> pitti: since it is indeed true that /srv/ddebs.ubuntu.com/www/dists/yakkety-updates/ doesn't exist
[14:19] <xnox> cjwatson, pitti - my compaint is about ddebs.ubuntu.com missing yakkety-updates initialisation.
[14:19] <xnox> also xenial-security is missing
[14:19] <pitti> xnox: yes, understood
[14:20] <xnox> huh, -security is missing on all after trusty
[14:20] <pitti> xnox: -security isn't a thing on ddebs
[14:20] <xnox> ok
[14:20] <pitti> we always immediately copy -security to -updates
[14:20] <pitti> for better mirror load
[14:21] <xnox> then just yakkety-updates was not (yet) created on ddebs.
[14:53] <pitti> infinity, slangasek: can you please have a look at my last comment in bug 1547466 ? i. e. do you agree that we should just SRU grep 2.25 wholesale or should I continue monkey-patching this?
[14:55] <stokachu> is getting a MRE for my packages the way to go in order to have newer features (in addition to bug fixes) approved for xenial? packages im referring to are in universe (conjure-up, bigdata, openstack)
[15:59] <infinity> pitti: I'm inclined to agree with the "just update to 2.25" position.
[16:00] <slangasek> stokachu: an MRE is specifically intended for upstreams that do bugfix-only releases.  We do make other exceptions for featureful upstream releases but they are more exceptional
[16:01] <slangasek> pitti: given that we seem to have landed on a lemon of a release with grep 2.24 I'm ok with 2.25 in principle, but how do you propose to regression-test this?
[16:01] <stokachu> slangasek: ok, specifically in conjure-up, openstack case is there something I need to file with the TB to get an exception for new features?
[16:02] <stokachu> they are listed as part of juju's exception process so i wasn't sure if i needed to do anything else
[16:04] <slangasek> stokachu: "listed as part of juju's exception process"?
[16:05] <slangasek> stokachu: the TB has ruled that the SRU team has the authority to make their own decisions about these exceptions without them going to the TB.  However, there is no structure in place within the SRU team to review these.  So I do think the TB is still the best place for now to get these things done
[16:05] <stokachu> slangasek: sorry looks like it was just an SRU that included enhancementshttps://bugs.launchpad.net/ubuntu/+source/juju-core/+bug/1573176
[16:42] <pitti> slangasek: grep has tests itself and for these changes, but of course that doesn't tell us if anyone was relying on the grep 2.2[34] binary behaviour under C by now; although that's rather adademic IMHO
[16:43] <pitti> slangasek: beyond that, I don't have a good idea myself how to define regression tests for that other than the obvious test for fixing that bug (i. e. whether it causes new FTBFS)
[16:44] <pitti> it could just equally be that the regression in .24 (grep -Pz) breaks stuff
[16:44] <pitti> so in summary having the fixes seems better to me than keeping the current broken behaviour
[16:44] <pitti> but yeah, difficult :(
[16:46] <slangasek> pitti: I don't consider an upstream testsuite sufficient regression testing for such a core tool.  Maybe an archive rebuild, yes
[16:46] <pitti> wrt. regression testing backporting that (huge) fix vs. taking the full release doesn't make much difference either
[16:46] <slangasek> and a longer aging?
[16:46] <pitti> i. e. maybe we should step back and rather ask us "do we want to fix this bug at all"
[16:46] <slangasek> I think we do
[16:46] <slangasek> it's a huge and problematic interface change
[16:47] <pitti> I think we should, but saying "no" would be policy compliant too)
[16:47] <slangasek> and if we had C.UTF-8 as default everywhere already then we could ignore it
[16:47] <slangasek> but as it stands this has a big negative impact on the LTS
[16:47] <pitti> yeah, folks upgrading from trusty who then find their usages of grep broken won't be amused
[16:48] <pitti> so back then when we fixed all the ftbfs fallout that actually sounded like a deliberate (although painful) behaviour change, so I'm glad that it wasn't after all
[16:49] <pitti> slangasek: an archive rebuild sounds good (maybe on amd64 only), but we build under C.UTF-8 now
[16:49] <slangasek> it is a misfeature that users' locale is not UTF-8 by default.  infinity had aspirations to fix this for 16.04 but it didn't make the cut
[16:49] <pitti> if we could do a test rebuild under C, that might help
[16:50] <pitti> how do you mean? user's locale for a normal install always ought to be a real UTF-8 one, not C (nor C.UTF-8)
[16:50] <slangasek> pitti: reverting the locale isn't relevant for regression testing, which is what we'd want the archive rebuild for
[16:50] <slangasek> pitti: "for a normal install" - only for a desktop install
[16:51] <slangasek> if you do a C locale install, e.g. on a server image where you select no locale at all, you get ASCII where you should get UTF-8
[16:51] <pitti> for server and cloud too
[16:51] <pitti> ah, does server allow you to choose C? (it's been a while since I did a d-i server install, but I don't remember that)
[16:52] <pitti> sorry, time for basketball, will be back in a few hours
[16:53] <slangasek> not only that, but there are environments (chroots, etc) where by default you're never selecting a locale, period
[16:53] <pitti> wgrant, doko: how difficult would it be to do an archive rebuild (one arch only) against xenial plus grep from xenial-proposed?
[16:53] <pitti> yes, sure
[16:53] <slangasek> (and schroot's handling of LANG has been making me mad for some time, besides)
[16:53] <pitti> I'm not saying it doesn't happen, I just disagree that it can be considered "our normal default"
[16:53] <pitti> in the end stuff needs to work in both
[16:54] <slangasek> When you don't configure a locale, the default is C.ASCII.  That we have installers which guide you to choose a locale doesn't change this
[16:55] <cjwatson> pitti: copy it to a PPA and then it's easy
[16:55] <pitti> slangasek: right (not sure what we're actually debating now)
[16:55] <pitti> cjwatson: ah, that's even easier
[16:55] <slangasek> pitti: the meaning of the word "default" ;)
[16:55] <slangasek> pitti: enjoy basketball :)
[16:55] <pitti> ah :)
[16:55] <cjwatson> pitti: though I'd like to wait until the current librarian expiry situation is sorted out a bit better, if that's possible
[16:55] <pitti> cjwatson: oh, absolultely
[16:56] <slangasek> but in fact, maybe this highlights misunderstanding between infinity and myself about how this should be fixed... he may be assuming that it's a sufficient fix to set C.UTF-8 in all environments by default, and I may have just argued that this is not sufficient
[16:56] <pitti> ok, will copy to a PPA when I'm back
[17:17] <infinity> slangasek: If you're arguing for redefining the libc C/POSIX as C.UTF-8, that could be rather surprising to people who actually expect it to be ASCII.
[17:18] <infinity> slangasek: I think the best we can do is attempt to make all system/user login environments default to C.UTF-8 (when not otherwise specified) and call it good enough.
[17:19] <slangasek> infinity: I'm arguing that when no locale env vars are set, C.UTF-8 should be used instead of C
[17:19] <infinity> slangasek: But POSIX says otherwise.
[17:20] <infinity> slangasek: If you mean that when nothing is specified, a login session should SET C.UTF-8 (ie: mangle pam_env to do this), I could get behind that.
[17:20] <infinity> slangasek: But when you explicitly unset it all after that, it should come out POSIX.
[17:20] <slangasek> pfft
[17:20] <xnox> slangasek, given that C.UTF-8 is a superset of POSIX....
[17:20] <xnox> infinity, that is ^
[17:20] <infinity> xnox: Yes and no.
[17:21] <slangasek> infinity: can you point me to the bit of POSIX that specifies this?
[17:21] <slangasek> I'd like to spend the next half hour of my life rules-lawyering you
[17:21] <xnox> =)
[17:21] <infinity> xnox: C.UTF-8 contains the same characters at the beginning of the codeset as C, yes, but UTF-8 still isn't ASCII, and pretending it is is surprising if you expected ASCII.
[17:24] <cjwatson> POSIX actually contradicts infinity, although I think I still agree with infinity that keeping it ASCII is a safer choice
[17:24] <cjwatson> http://pubs.opengroup.org/onlinepubs/9699919799/basedefs/V1_chap07.html#tag_07_02
[17:24] <cjwatson> "This default locale can be the POSIX locale or any other implementation-defined locale"
[17:25] <mdeslaur> heh, was about to paste that :)
[17:25]  * mdeslaur loves butting into other people's arguments
[17:25] <infinity> "For C-language programs, the POSIX locale shall be the default locale when the setlocale() function is not called."
[17:25] <infinity> cjwatson: It seems to contradict itself. ;)
[17:25] <cjwatson> Ooh, now that's subtly different
[17:26] <infinity> Or, we're arguing what "default" means again.
[17:26] <cjwatson> setlocale() not called != setlocale() called but no locale env vars present
[17:26] <cjwatson> I would certainly argue that it is less confusing for those two situations to result in the same thing
[17:26] <infinity> Indeed.
[17:27] <slangasek> I would argue that if you're calling setlocale(), you're i18nized and expecting to deal with non-asciiness at some level, and if you're not calling setlocale() you're not, so a difference in behavior there is actually o
[17:27] <slangasek> k
[17:28] <infinity> slangasek: I agree that it's "ok", but it's still confusing from a user perspective.
[17:28] <slangasek> especially when the only difference we're actually talking about is whether or not to choke on utf-8
[17:30] <infinity> Anyhow, we could indeed change the setlocale() with empty vars behaviour.  I've never been super comfy about doing that because C.UTF-8 isn't actually a builtin (ie: it requires an external locale def), but we try really hard to make sure it's always on disk and built.
[17:34] <slangasek> infinity: supposing we don't change the default locale under setlocale().  can you tell me how to make the behavior of my schroot not suck?  Because pam_env on the host already sets my desired locale for the host, which is not C.UTF-8; that is /not/ the correct locale for use inside the chroots, where I should not have to install langpacks or have matching locales configuration; and I believe
[17:34] <slangasek> schroot currently deals with this by unhelpfully nuking any locale env vars.
[17:36] <infinity> slangasek: schroot invokes pam, it could be pam_enving as well.
[17:36] <infinity> slangasek: But you bring up a valid point for bare chroot(1) perhaps.
[17:36] <infinity> (Though, it's never bugged me that those are POSIX)
[17:36] <slangasek> infinity: it invokes pam on the host. pam_env already sets *the correct locale for the host*.
[17:37] <infinity> Oh, I was just grepping code.  It doesn't call into pam in the chroot?  Check.
[17:37] <infinity> I guess that would be a nonsensical thing to do if you're not "logging in" to the chroot.
[17:38] <infinity> slangasek: Anyhow, for individual tools like schroot, the answer could as easily be "set LANG=C.UTF-8".
[17:38] <slangasek> yeah; if it did call pam in the chroot, I would expect that to be busted in various other ways
[17:38] <infinity> slangasek: But the larger question is if we should have to, and I get that.
[17:38]  * slangasek nods
[17:41] <infinity> slangasek: Though, arguably, schroot should be doing something clever to read /etc/environment and /etc/default/locale (ie: emulating pam_env, if it can't call it).   It's meant to be a much smarter tool than dumb ol' chroot(1).
[17:41] <infinity> slangasek: So the real question is if the behaviour of chroot(1) is correct, and I'm sort of on the fence there.
[17:42] <slangasek> infinity: if I have to configure /etc/environment and /etc/default/locale inside each schroot chroot to get sane behavior, that is also full of fail
[17:43] <slangasek> schroot setting LANG=C.UTF-8 by default is saner, but still has the problem that we have to individually touch each tool whose behavior we want to make sane
[17:44] <infinity> slangasek: Why?  I treat my schroots as individual "systems", not unlike a container or a VM.  Those are configured via /etc/default/locale
[17:44] <infinity> chroot(1), on the other hand, just gets spillover from your environment and no guarantee of cleanliness, which is entirely correct, if gross.
[17:44] <slangasek> infinity: the whole point is that this should *not* be "configuration". This should be "don't be stupid by default".
[17:45] <apw> isn't that down to making the chroots sane when we deliver them
[17:45] <xnox> slangasek, i thought glibc doesn't default to C.UTF-8 because it was not compiled-in locale into libc.so|a binary itself, unlike posix. the point of posix/c locale was that it doesn't need any external files. has that been fixed now, and c.utf-8 is compiled into libc.a ?
[17:45] <infinity> slangasek: Okay, sure.  schroot should set C.UTF-8 if there's no existing config, but I'd argue it should also be honoring /etc/* in case we have an /etc/default/locale that says otherwise.
[17:45] <infinity> xnox: c.utf-8 is still external.
[17:46] <infinity> xnox: Even in our talks about integrating it upstream, no one's yet gone down the scary road of trying to make it a built-in.
[17:46] <xnox> right, then it can't be universally default.
[17:47] <xnox> infinity, i tried and failed when working on clearlinux
[17:48] <infinity> We do always ship it pre-compiled with libc.so, so for the dynamic case, it's basically as good as builtin (though, we'd need to make sure to pull it into the initrd, etc).
[17:48] <infinity> But for static, it's a bit more of a crapshoot.
[17:48] <infinity> Especially with craptastic Built-Using tracking, which we need to fix.
[17:49] <infinity> Since the compiled version on disk may not match what the setlocale() you compiled into your binary expects.
[17:49] <infinity> Well, also means the concept of a portable 100% static binary dies.
[17:49] <infinity> But hey, libnss broke that more than a decade ago.
[17:53] <infinity> Ideally, I'd like to figure out how to make the upstream build system let you pick a --setlocale-default and build that locale into libc.{so,a} at build time.
[17:53] <infinity> That would make this less scary.
[17:56] <infinity> That would also let embedded vendors build, say, a C+ru_RU system without the overhead of external locales.
[17:57] <infinity> Which would be neat.
[19:13] <doko> pitti, pretty easy. is this urgent? I'm planning to do one with an updated gcc too, but not yet
[20:47] <pitti> doko: not "this week" urgent for sure, the buildds are aching enough as they are
[20:47] <pitti> doko: perhaps next week or so, when the dust settles
[20:47] <pitti> for now, just discussing options how we could verify the new grep
[20:48] <pitti> slangasek: if doko wants to do a test rebuild with gcc-6+the new grep, would you consider this a close enough approximation? doing two rebuilds at the same time seems a bit wasteful
[20:49] <pitti> (grep 2.25 is in yakkety already)
[20:50] <pitti> ♩ ♪ 5.500 tests on the wall, 5.500 tests -- break one cloud, upload one perl, 8.800 tests on the wall ♫ ♬
[20:54] <slangasek> pitti: yes, that's fine by me
[20:56] <slangasek> tests> heh
[20:56] <slangasek> queue under 2k!  fancy
[20:57] <pitti> slangasek: btw, there are 8 extra armhf workers on scalingstack now
[20:58] <slangasek> pitti: excellent :)
[20:58] <pitti> slangasek: of course I get a watchdog email every 15 minutes or at least 2 hours which hard-reboots them, so I'm not entirely sure how much load they actually take off
[20:58] <pitti> as each hang causes the client side to hang for a fair while too (until commands time out)
[20:59] <pitti> but I think the net result between taking off load and introducing hangs is still positive, armhf was a lot furhter behind yesterday (relative to x86)
[21:00] <pitti> slangasek: can you exercise your magic management powahs to make cking available to me for half a day or so on our sprint? :-)
[21:00] <pitti> he's a miracle on figuring out such stuff, and I don't have a clue how to debug this any further :(
[21:04] <pitti> speaking of breaking clouds -- wgrant, cjwatson: https://launchpad.net/builders → quo vadis, lcy01 buildds?
[21:05] <pitti> FWIW, on my side I keep running into "ERROR" states recently again, but it's by and large running
[21:05] <slangasek> pitti: cking isn't mine to give you, put something on the sprint agenda and ask ogasawara ? :)
[21:06] <cking> i am sure that is doable
[21:07] <pitti> ah, there's not a list, but a timetable; I'll pick some random time and let ogasawara move it around or slap my fingers then :)
[21:07] <pitti> cking: that'd be great, thanks
[21:08] <pitti> ah no, there's a link to "kernel agenda", found it
[21:08] <pitti> r/o
[22:45] <slangasek> infinity: apparently we've had cloop as a "supported" "desktop" package in main since 2008 to support mounting livecds.  But I'm pretty sure that's not compatible with any format we're currently using. ;)  Should we drop that from the seed?
[22:47] <infinity> slangasek: Oh, it was manually seeded?  Oops.  Should have dropped out when we dropped the dep from livecd-rootfs.
[22:47] <infinity> slangasek: Kill mit fire.
[22:47] <slangasek> infinity: gekillt und gepusht