michirobru: Question for you…01:37
robrumichi: hey what's up?01:37
michiJust trying to understand how things work.01:37
michiWe have some silos stuck on testing because their dependencies can’t be satisfied.01:38
michiSo britney isn’t happy.01:38
robrumichi: where?01:38
michiOnce the dependencies appear, how do we know? Does it re-test automatically once the dependencies are OK?01:38
michisilo 37.01:38
michiSee the excuses for xenial01:38
michiscopes-shell won’t test because qtcreator-plugin-ubuntu is out of date, I think.01:39
michiI’m just trying to understand how to recover from this situation once the dependencies are OK again.01:39
michiDo we need to manually re-trigger a build?01:39
michiOr does Britney automatically wake up to the fact that it now makes sense to try again?01:39
cjwatsondon't retrigger a build, people with appropriate privs can retry01:39
michicjwatson: thanks01:40
cjwatsonit will ... sometimes automatically retry, it depends on what changed01:40
michiso we just wait until things are fixed magically?01:40
cjwatsontoo many levels down and it won't notice01:40
robrucjwatson: michi is talking about autopkgtests, not depwaits in ppas01:40
cjwatsonyes I know01:41
cjwatsonit's still true01:41
cjwatsonif an immediate dependency changes then that will in itself trigger new tests01:41
cjwatsonbut a change to a dependency of a dependency won't01:41
michiSo, we basically just sit and do nothing?01:41
michidobey mentioned that britney is struggling a bit right now.01:42
michiWhere can I look at it?01:42
cjwatsonthat looks vaguely like the known jsoncpp ABI thing?01:42
robrumichi: my 'right' was aimed at cjwatson01:42
cjwatsonmichi: that's not britney struggling, that's the autopkgtest workers (different bit of the system, former uses the latter).  http://autopkgtest.ubuntu.com/running.shtml01:42
robrumichi: britney logs are here: https://requests.ci-train.ubuntu.com/static/britney/last-run.txt it's currently taking an hour to do a run. this is caused by high load, but it's not actually causing any problems other than slowness01:43
michiAh, cool01:43
michicjwatson: Where do you see the json problem?01:43
jameshmaybe things would run faster if we had less tests?01:44
cjwatsonI mean, robru knows more about how britney is set up in bileto, but dobey was specifically talking about autopkgtest queues earlier and just used the wrong name01:44
cjwatsonmichi: people talking about it today01:44
cjwatsonand context and stuff01:44
michiYes, I know, but I don’t actually know what happened with json01:44
michiSome ABI break01:44
michiBut in detail?01:44
robrumichi: yeah there's two different things going on here. your tests run in the autopkgtest system, britney polls it for results in regular intervals. currently that interval is "every 65 minutes" due to high load. in the past it was 25 minutes.01:45
cjwatsonmichi: I don't see it named in this particular test, but I know it's causing installability problems of this kind.  for detail I'm afraid you'll have to hunt around for more, I'm just stopping by briefly here.01:45
michicjwatson: Cool, thanks.01:45
robrumichi: really who you want to talk to about autopkgtest issues is pitti01:45
michirobru: The queue for britney looks ten miles long.01:46
cjwatsona lot of those are relatively short, but it's probably a day's worth of queue :-/01:46
robrumichi: yeah I haven't had time to dig in, theoretically there's 17ish silos to process and they take 4ish minutes each, in order (can't be parallelized due to staggering memory usage)01:46
michiOK. So the upshot of it all is that we do nothing and just wait until silo 37 is ready.01:46
michiSo be it :)01:47
robrumichi: I dunno, if it still says 'regression' after a day and the queue died down a bit I'd poke somebody to retry them01:47
cjwatsonin principle a lot of that has got to be parallelisable because most of the data structures should be based on the primary archive and therefore shared, in practice that's very hard to set up ...01:47
michirobru: OK. But I don’t want to needlessly get on people’s nerves.01:48
michiSo, is there any way for me to check when it makes sense to talk to someone?01:48
michiAnd whom?01:48
robrumichi: yeah pitti is the guy for digging into autopkgtests01:48
robrumichi: I dunno how to tell when really, i'm not really familiar with that 'test dependency' error, and I think it's suspect that it all passed in vivid but failed in xenial.01:49
michithe problem is that a whole bunch of stuff is stuck behind scopes-api which got broken by an update to cap’n proto, which in turn broke abi-compliance-checker.01:49
michiNow everything is piling up because we can’t get silo 37 landed, and nothing that uses scopes can land either.01:49
michirobru: right. Vivid is fine01:50
robrumichi: yeah I'd get pitti on it if this is blocking a lot of other stuff for you01:50
cjwatsonI would suggest retrying when https://launchpad.net/ubuntu/+source/ubuntu-touch-meta/1.258 hits the release pocket + half an hour01:50
michido you know his TZ?01:50
cjwatsonpitti is in .de01:50
cjwatsonbut the early side01:50
michiOK, thanks01:50
michiSo late afternoom my time.01:50
cjwatsonhilariously ubuntu-touch-meta is blocked on autopkgtest failures due to basically the thing it fixes.  *that* probably needs pitti to disentangle01:52
robrucjwatson: heh, was just going to say01:52
robrumichi: http://people.canonical.com/~ubuntu-archive/proposed-migration/xenial/update_excuses.html#ubuntu-touch-meta01:52
michiHow does something like this *ever* land with that many dependencies?01:54
robrumichi: "that many"? what are you looking at?01:54
michiThere are tons of autopkg tests that ubuntu-touch-meta lists in its excuses01:55
michiDon’t they all have to pass before this can land?01:55
jameshmichi: the only thing mentioned under ubuntu-touch-meta is the click scope tests01:56
michiAh, I get it now01:56
robrumichi: http://i.imgur.com/hUnVrbx.jpg yeah01:56
michiYep. I was under the mis-apprehension that everything the whole list had to pass.01:57
michiyou can tell that I don’t know much about the packaging machinery :(01:57
robrumichi: anyway follow that up with pitti, he can poke that through if that's the appropriate thing to do.01:58
michiYes, thanks!01:58
robruyou're welcome01:58
cjwatsonAs jamesh alluded to earlier, this was all a lot easier when we just uploaded straight to the archive without any automated tests running on the built packages. :-P  (But I think on the whole people prefer that we don't do that any more ...)02:14
jameshit's a shame libjsoncpp could be uploaded without fixes to all the packages that sit on top of it then ...02:15
cjwatsonWell, it hasn't reached xenial yet, it's only in xenial-proposed.02:15
cjwatsonUnfortunately ubuntu-touch-meta has explicit dependencies on a particular ABI, which is anomalous for metapackages.02:16
cjwatsonThere are reasons for it, but it does tend to exacerbate this kind of problem.02:16
robrucjwatson: yeah kind of a funny situation, good that the change didn't break xenial, but seeing as how everything builds against -proposed anyway, it's sort of ground everything else to a halt02:18
jameshyeah.  We wouldn't have been able to get something like that into -proposed for one of our own packages with problems like that02:19
robrucjwatson: TODO: make all debian imports go through silos ;-)02:21
cjwatsonrobru: hahahaha02:24
cjwatsonrobru: next step, work out how they interleave02:24
jameshit looks like the autopkgtest queue length for armhf is steadily going down, but ppc64el is stuck at 115703:27
jameshis something broken there?03:27
robrumichi: don't worry about that "version mismatch" message, i think you just lost a race condition, should sort itself out within 15 mins.05:31
michirobru: Aha.05:31
michiOK, thanks.05:31
robruyou're welcome05:31
Mirvrobru: \o/ yes, 023 got automated signoff now. thanks, train rules!05:40
robruMirv: haha thanks. We've come a long way!05:40
jameshrobru: I don't suppose you're in a position to kick the ppc64el autopkgtests?05:52
robrujamesh: nope i think you need pitti for that05:53
jameshthought that might be the case :(05:53
Mirvrobru: and now... another corner case! https://requests.ci-train.ubuntu.com/static/britney/xenial/landing-032/excuses.html - removed binary, will be removed from archives but britney would need to run...05:54
Mirvjamesh: I may be able, there is now a button for those with upload rights. link?05:55
jameshMirv: I've just been watching http://autopkgtest.ubuntu.com/running.shtml05:55
robruMirv: I've seen that message before, i think that can be fixed without bileto changes05:55
jameshMirv: the armhf queue length has steadily been decreasing from about 700 this morning05:55
jameshMirv: the ppc64el queue was stuck at 1157, and now seems to be up to 116005:56
robruMirv: he doesn't want a retry, he wants the entire machinery kicked because it's stuck05:56
jameshit doesn't seem to be doing anything05:56
robruLike physically kicked05:56
Mirvjamesh: ah kicking, not rerunning... yes you need pitti05:57
robruMirv: try deleting old superseded source packages from your ppa. The version being complained about doesn't seem to exist in Ubuntu so i think it's just complaining about ppa contents06:01
Mirvrobru: but how? (fixed) the normal fix is to remove the obsolete binary from archives. although I'm not sure why it's saying that already there, since usually it's just a reason stated in addition to the tests run06:01
Mirvrobru: sorry, reading your msg :)06:01
Mirvrobru: ah, yes it seemed really weird. trying to delete superseded packages then, makes sense (in the weird PPA way)06:01
Mirvrobru: indeed! I've seen that before - in such a situation it will not get auto-deleted in PPA even if superseded. ok, let's wait 12h for the new results :)06:02
robruMirv: you should see new results in 1 to 2 hours, but then it will trigger autopkgtests of course after that06:03
Mirvrobru: yeah, I was assuming how long time it will take until getting all reverse deps of qtbase finished06:04
Mirvanyway, no hurry, I put it churning those britney results early precisely for that reason. I've tested the silo but that's really a silo that can't see too much testing so I'll just continue.06:06
michijamesh: could you look at this one? https://code.launchpad.net/~michihenning/thumbnailer/fix-admin-relative-paths/+merge/28419207:19
michiIt’s fairly simple.07:19
michijamesh: Whoa, something’s happening with ppc queue.07:26
jameshmichi: commented on the MP.07:34
michiI just saw, thanks!07:34
michiIf a relative path is passed to vs-thumb, it bombs.07:35
michiThat’s why the canonicalization on the client side.07:35
jameshright, but you're canonicalising twice.  Maybe just checking for absolute paths in one of those cases would do.07:36
michiYes, just looking at that.07:36
michiBut the client could be broken too and pass a relative path to the qt lib07:37
michiSo, I guess it would be sufficient to do it in the qt lib, and not in vs-thumb07:37
michiOr not… Thinking...07:38
michiSight, it’s been a while.07:38
jameshnothing in that branch touches vs-thumb07:38
michiSo, originally, I stumbled across this because thumbnailer-admin get didn’t work with relative paths.07:38
jameshthere is already a canonical() call inside thumbnailer-service, so vs-thumb should be protected.07:39
michithumbnailer-admin goes through the client lib.07:39
michiSo, canonicalising in the client lib is correct.07:39
michiAnd, when I use vs-thumb for testing manually, I’d like it to work too.07:39
michicanonicalising in thumbnailer-service is pointless because that has the wrong directory07:39
michicurrent directory07:39
jameshin your branch, thumbnailer-admin is canonicalising the path, and then passing that canonicalised pathname to a method that also canonicalises it07:40
jameshcanonicalising in thumbnailer-service is not pointless, since it is in a different security context07:40
jameshit is doing so for a different reason07:40
michiIf the client sends a relative to the service, the service will end up with a completely different canonicalized path.07:42
michiThat’s why the change in thumbnailer.cpp07:42
michiaround line 103 of the diff07:42
michiBut, yes, the service side still canonicalises the path to not get confused with different keys for the same file.07:44
michiSo, the service rejects non-absolute path names now, which is what we want I think.07:44
jameshmichi: my understanding is that the canonical() call already in thumbnailer-service was to improve cache hits in case the client passed a symlink file name in07:44
michiNo other change there.07:44
michiYes, right.07:44
michiDon’t disagree with that.07:44
jameshmichi: adding a requirement for is_absolute() there is fine by me too.07:44
michiBut it was possbile previously to send it a relative path, which turned into garbage on the service side.07:44
michiRight, cool :)07:44
jameshbut I wouldn't remove anything on that side, because it shouldn't rely on any work done on the untrusted client side.07:45
jameshmy comment on the MP was that on the client side you were doing canonical() twice07:45
michiSo, I think what *can* be removed is the check in thumbnailer-admin.07:45
michiBecause the lib does that too.07:46
michiAre we on the same wavelength now?07:46
jamesheither remove the canonical() call in thumbnailer-admin, or make the client lib just throw on !is_absolute()07:46
michiI think the safer option is to leave the lib as is and remove it from thumbnailer-admin.07:47
michiThat way, if someone talks directly to the lib, we avoid a round-trip to the service.07:47
sil2100pstolowski, michi, dobey: hey! I see that the acc-disabling silo is failing on autopkgtests, are you guys looking into that?10:19
michisil2100: Been looking at it all day :(10:19
michiIt’s a break in qt-something.10:19
michiThe autopkg tests weren’t making progress for almost the entire day.10:20
michiBecause the PPC test machine died.10:20
michiSo nothing got through.10:20
michisilo 37 is what we really would like to get unblocked.10:20
michisil2100: Been tinkering with abigail most of the day.10:22
michiMaking progress. Found a bug today. The main dev at RedHat is looking at it.10:22
pstolowski"badpkg: Test dependencies are unsatisfiable."10:25
pstolowskiBroken libscope-harness2:amd64 Depends on libunity-scopes1.0 [ amd64 ] < none -> 1.0.3+16.04.20160209-0ubuntu1 > ( libs )10:26
pstolowski  Considering libunity-scopes1.0:amd64 2 as a solution to libscope-harness2:amd64 010:26
pstolowski  Holding Back libscope-harness2:amd64 rather than change libunity-scopes1.0:amd6410:26
sil2100hm, I'm not an expert in autopkgtests10:34
sil2100Mirv: you seem to have more experience, could you help me parse that ^ ? :)10:35
Saviqsil2100, can you please press ♻ on the scope click regression in http://people.canonical.com/~ubuntu-archive/proposed-migration/xenial/update_excuses.html#unity810:50
* Saviq finds it quite weird this is limited to core-devs10:52
sil2100Yeah, it shouldn't10:55
Mirvsil2100: I'm also not too expert in finding the actual problem from the logs.. experimenting in chroot helps, like with the python problem11:18
sil2100pete-woods: hey!11:18
sil2100pete-woods: do you know if we still use unity-voice anywhere? Is that project maintained and necessary?11:19
MirvSaviq: more exactly it's limited to "those with upload rights to the package". there needs to be some limitations, but maybe additional rights could be considered at some points. pitti would know the rationale.11:20
pete-woodssil2100: we don't use it anywhere, no11:21
MirvI'll try to look more into the failures in a bit11:21
pete-woodssil2100: it used to be used by HUD, but at request we removed it, as the functionality was only available on the phone11:22
pete-woodsand HUD was also removed from the phone11:22
sil2100Ok, I wonder if we could remove it from the archive11:22
pete-woodsif could certainly at least be bumped into universe from main11:23
pete-woodsbut I'd rather not remove it completely, in-case suddenly HUD needs to do voice recognition again11:23
cjwatsonpete-woods: unity-voice is already in universe.11:26
LaneyThe problem is that some APIs it uses have changed and so it needs maintaining11:27
LaneyYou could ~easily bring it back if required11:27
sil2100I would say it's no use having it in the archive just for the sake of having it11:33
sil2100The lp project can stay so as Laney said, it can be re-released whenever needed11:33
pete-woodsokay, fair enough, feel free to nuke it then11:36
Mirvsil2100: michi's silo 037 problem is because of the libjson transition. not sure what to do, maybe agree that vivid autopkgtests show green and override to get it into QA queue anywya.12:19
michiMirv: Are you sure it’s because of json?12:19
Mirvsil2100: there's a clash with the two sides of things that the builds are with proposed and autopkgtests without proposed but with the PPA. that's intended and most of the time provides the best results.12:19
michiI thought I saw an excuse about qt-something12:19
Mirvmichi: the PPA depends on libjson1 which is only available in -proposed, but the autopkgtests run without proposed so that there wouldn't be failures coming from outside the silo.12:20
Mirvmichi: can you point me to it? the autopkgtest logs however have a lot of things in them that seem suspicious but are simply observations and not a cause for a failure.12:20
michiLet me see if I can find it.12:20
Mirvsil2100: but the libjson transition should simply be finished, that's the real fix12:21
michiMirv: I’m looking at https://requests.ci-train.ubuntu.com/static/britney/xenial/landing-037/excuses.html12:22
michiScroll to the bottom.12:22
michiThen look at the i386 regression for scopes-shell: https://objectstorage.prodstack4-5.canonical.com/v1/AUTH_77e2ada1e7a84929a74ba3b87153c0ac/autopkgtest-xenial-ci-train-ppa-service-landing-037/xenial/i386/q/qtcreator-plugin-ubuntu/20160210_000636@/log.gz12:22
michiRight at the bottom of that, it bitches about qt-something12:22
sil2100Mirv: the scopes silo is the only thing left for the libjsoncpp transition12:23
sil2100Mirv: the only packages left with the old libjsoncpp0v5 dependencies are in this silo and this silo was supposed to fix it12:24
sil2100(as we needed them to be re-built)12:24
Mirvsil2100: ah. in that case I think we might simply stare at this beauty https://requests.ci-train.ubuntu.com/static/britney/vivid/landing-037/excuses.html and ask QA to get it into their queue manually12:24
sil2100Mirv: hah ;) Yeah, I just couldn't understand why this is failing, since all the required leftover packages that needed a rebuild were in this silo, so I expected autopkgtests to install all that are in the silo12:25
Mirvmichi: the fact that it runs something (certain autopkgtests) is not that the failure in those running would be related to the failing package. _all_ reverse dependencies of unity-scopes-shell that have autopkgtests are always executed, and that includes qtcreator-plugin-ubuntu (SDK install some scope package) and unity8.12:25
michiMirv: cjwatson mentioned earlier today that, when something won’t work because a dependency of a dependency isn’t available, it needs manual intervention.12:25
Mirvsil2100: because the libjson itself is not in the silo, but only in -proposed12:26
Mirvsil2100: and autopkgtests run without proposed12:26
sil2100Aaaaaaah, ok, I keep forgetting about the -proposed bits12:26
michiMirv: If we are buliding against proposed, the bloody autopkg tests had better run against proposed too, otherwise this is a thoroughly pointless exercise!12:27
sil2100Yeah, I don't know why I forgot about it, I got bitten by it last week already12:27
Mirvsil2100: well it changed yesterday evening so ... :D12:27
Mirvsil2100: so after consulting with pitti we agreed to disable proposed from autopkgtests, because that causes a lot of non-PPA-related failures (failures that need to be fixed in -proposed before the other things migrate from there). but the counterside is this that if something from proposed is actually required...12:28
michiI really think it doesn’t make sense to run autopkg against xenial, when the code was built with xenial-proposed.12:28
Mirvsil2100: so this implementation made it possible for my silo 23 to get automatic signoff after 1,5 weeks of related train fixes :)12:28
michiBoth success and failure of a test are pretty much meaningless in that case.12:29
sil2100jibel, davmor2, rvr: hey guys! Regarding silo 37 - could you guys manually shove it into the QA queue? The failing xenial tests are caused by us not using what's in -proposed12:29
sil2100jibel, davmor2, rvr: it's all part of a transition in xenial12:29
davmor2sil2100: take it the fix that was meant to land still didin't then12:30
davmor2didn't fix it that is12:30
michiAm I missing something here?12:30
cjwatsonMirv: Surely all this is libjsoncpp, not libjson?12:30
michiWe are testing with old packages for code that built with newer packages and expect to get meaningful answers?12:30
Mirvcjwatson: sure, yes12:30
cjwatsonmichi: The point here is to ensure that integrating new code doesn't break stuff already in the archive.  Of course that's important and expected to be meaningful.12:31
cjwatsonmichi: In cases where we're integrating a block of multiple packages then as a general rule (modulo bugs) we try to test the newer ones, though.12:31
michiThis feels dubious to me.12:31
Mirvsil2100: the proposed thing was bug #154133412:31
ubot5`bug 1541334 in autopkgtest (Ubuntu) "Do not run silo tests against all of -proposed" [Undecided,Fix released] https://launchpad.net/bugs/154133412:31
michiI mean, pretty much anything could happen.12:31
cjwatsonYes.  It's software.12:32
michiIt’s essentially undefined what answers come back from a test.12:32
michiThe code tests with packages it wasn’t built against.12:32
cjwatsonWe can investigate and override cases where mistakes happen.12:32
michiThat doubleplusungood.12:32
michiHow so?12:32
cjwatsonIf the packages you're fearful of need to be rebuilt, then that's a thing you need to know.12:33
cjwatsonIf tests fail, then it may well indicate that your landing is incomplete.12:33
michiSo, I call into libx at v112:33
cjwatsonSimply ignoring that on the basis that it's "essentially undefined" is deeply dubious practice.12:33
michiI write tests, blah, blah.12:33
michiThen something goes and runs my tests built against v1 with autopkg tests from v012:34
michiSorry, with autopkg tests that call into into v012:34
michiSo: bool f()12:34
michireturns false in v0, and true in v112:34
cjwatsonIn most cases, these things are integration tests.  They're not testing specific functions like that, they're testing that the package as a whole is still functional.12:35
michiHow can I make any QA assurance now?12:35
michiYes, I know.12:35
michiBut this entire approach has landmines all over it.12:35
cjwatsonThere are some edge cases where one needs to change the autopkgtests to take account of interface changes, and those cases we can override.12:35
cjwatsonBut you still want to know about them.12:35
michiWell, for the record, I think I can safely predict that this will blow up in unpredictable ways some of the time.12:36
cjwatsonIf the package containing these autopkgtests has changed, then typically the newer tests will be run.12:36
cjwatsonThat's nice.12:36
michiI think that’s a thoroughly bad idea.12:36
michiYes, newer tests are better than older ones.12:36
cjwatsonWhat are you actually proposing, in concrete terms?12:36
michiBut if they call into something that is now essentially undefined because it is unexpected, anything can happen.12:36
cjwatsonThat's not helpful.12:36
cjwatsonWhat are you actually proposing, in concrete terms?12:36
michiIf something was built with v1, it needs to test with v112:37
michithis approach effectively switches dependencies underneath autopkg tests, unless I’m misunderstanding something.12:37
cjwatsonIn general that will happen if there was a documented interface change (i.e. soname change or whatever).12:37
michiWe change version all the time, with minor behavioral changes in an API.12:37
michiThese are definitely not soname changes.12:38
cjwatsonMost of which has zero effect on autopkgtests.12:38
michiYes, because most people write sloppy autopkg tests that pay lip service to what an autopkg test should actually be doing, namely, to test the full functionality of a package.12:38
cjwatsonThat's kind of a retcon.12:39
michiI count myself among the sinners...12:39
michiretcon? What’s that? I don’t know the term.12:39
cjwatsonRetroactive continuity12:39
michiAh :)12:39
cjwatsonAs in, declaring something to have always been the case after the fact.12:39
michiYes I get it :)12:39
cjwatsonThe real point of autopkgtests was to test packages in their as-installed state, rather than just in the build tree.12:40
michiSo the main point is to test that installation worked, rather than the code, which should have been thorougly tested much earlier.12:40
michiI get that.12:40
michiExcept that software has its ways.12:41
Saviqsil2100, no dice, try again? http://people.canonical.com/~ubuntu-archive/proposed-migration/xenial/update_excuses.html#unity812:41
Saviqat least there's no queue now12:41
michiTo blow up in the most unbelievable ways, due to chains of impossible coincidences.12:41
michiAnyway, I got my five cents worth in. Thanks for listening! :)12:41
cjwatsonThe point here is that if your package doesn't require a newer version of the thing that depends on it to migrate, then you *do* need to make sure that the older version still works12:42
cjwatsonBecause otherwise you can end up letting integration failures through12:42
michiCorrect. But these dependencies get switched underneath me without any warning too.12:42
cjwatsonAnd in general it's even less likely that the newer version of autopkgtests for libx will work against an older version of libx12:42
michiWe get broken all the time by packages we call into that do something different suddenly.12:43
cjwatsonSo you have to run the older test code12:43
michiIt is *highly* likely that v2 of one of our dependencies will break us when v1 is still working.12:43
michiHappened to us just yesterday.12:43
michiAnd we never asked for v212:43
cjwatsonGreat, so hopefully that's a test failure.12:43
sil2100Saviq: done12:43
michiNope, it isn't12:43
* sil2100 off to prepare lunch12:44
cjwatsonOr insufficient tests.12:44
cjwatsonNow, if your landing is part of a set with a newer version of libx, then we'd want to run the newer tests for libx and newer code for libx12:44
michiabi-compliance-checker crashes because of totally legitimate and correct and backward compatible changes in another package that abi-compliance-checker doesn’t not even know exists.12:44
michiAnd, as a result, our unit tests fail.12:44
cjwatson"nope, it isn't [a test failure]", "and as a result our unit tests fail"12:45
cjwatsonwhich is it?12:45
michiSee, we have a new version of cap’n proto in xenial-proposed.12:45
michithat’s a perfectly good and legitimate and compatible update.12:45
michiAnd acc has a but that causes it to barf when it compiles the new capn proto headrs.12:45
michiAnd, because we are diligent, we run ABI compliance checks as part of our unit tests.12:46
michiAnd, suddenly, we fail in xenial-proposed.12:46
cjwatsonthat's not autopkgtests though12:46
michiAnd we didn’t even know that capn proto had changed, or that, suddenly, someone was building our code against xenial-proposed.12:46
michiBut it’s the same thing.12:46
cjwatsonno, it's really really not12:46
cjwatsonone of the things that autopkgtests are really good for that entirely didn't exist in any form previously is to allow packages to make assertions about the behaviour of their dependencies and have those stick12:47
michiIt’s spooky action at a distance.12:47
cjwatsonif capnproto were making assertions about the behaviour of acc in an autopkgtest, then failures there would prevent a newer version of autopkgtest from migrating to xenial12:47
michiYes, I understand the reasoning. And I’m totally supportive of autopkg testins.12:47
michicapnproto does not know acc exists, and vice versa.12:47
cjwatsonAnyway, I'm getting very confused by the way you're shifting between different facets of the argument here, so I'm going to go and do something more useful ...12:48
michiacc call gcc to compile the capn proto header files.12:48
michiAnd that fails.12:48
michiUltimately, it’s a bug in acc.12:48
michiI’m not trying to make trouble. Just pointing out that this is dangerous.12:48
michiNow, one thing that would really be useful:12:48
michiHave Jenkins start building on the side for <next-adjective>-proposed, but without hard failures.12:49
cjwatsonthe problem is that you're pointing out lots of different things that are not actually technically the same thing, and using them in support of each other12:49
michiThen we’d get an early heads up when packages in -proposed cause trouble.12:49
cjwatsonit's nearly impossible to follow an argument structured like this12:49
michiAll I know is that we only found out on Monday that there was a problem.12:49
michiBecause things started building in -proposed.12:49
michiSo, the problem is that we find out late.12:49
michiIf we could have some CI support, it would alert us to any problems much sooner.12:50
cjwatsonthat seems reasonable12:50
michiFor example, the capn proto chane has been sitting in proposed for months.12:51
michiBut we didn’t even know that this was coming.12:51
michiThen, we get all surprised when, suddenly, nothing passes tests anymore, and whole ton of people can’t land stuff because our package fails its tests due to an unknown change to a dependency.12:51
cjwatsonAny reason the capnproto change hasn't landed properly?  It looks like it breaks several phone-related packages.12:52
michiI’m trying to catch that sort of thing earlier, so we can be proactive.12:52
michiI have no idea.12:52
cjwatsonOne thing that would help is to not let things languish in -proposed.12:52
cjwatson    * amd64: camera-app-autopilot, gallery-app-autopilot, indicator-network-autopilot, indicators-client, libscope-harness-dev, libscope-harness2, libunity-scopes-cli, libunity-scopes-dev, libunity-scopes-qt-dev, libunity-scopes-qt0.2, libunity-scopes1.0, python3-scope-harness, qtcreator-plugin-ubuntu, qtcreator-plugin-ubuntu-autopilot, ubuntu-experience-tests, ubuntu-pocket-desktop, ubuntu-push-autopilot, ubuntu-sdk, ...12:52
cjwatson... ubuntu-sdk-libs-dev, ubuntu-touch, ubuntu-touch-session, unity-plugin-scopes, unity-scope-click, unity-scope-click-autopilot, unity-scope-mediascanner2, unity-scope-scopes, unity-scope-snappy, unity-scope-tool, unity8, unity8-autopilot, unity8-common, unity8-desktop-session-mir12:52
michisil2100 wasn’t totally sure whether the guy wo pushed the change actually tested all the reverse deps.12:52
cjwatsonthat's the uninstallable list from the new capnproto12:52
cjwatsonnot only are they untested, they're uninstallable12:52
cjwatsonvery likely because of the soname change12:52
michiWe just disable one unit test, and that “fixed” it.12:53
cjwatsonneeds a rebuild of libunity-scopes1.012:53
michiAt the cost of no longer running abi compliance checks.12:53
michiWe pushed a branch for that yesterday. silo 3712:53
cjwatsona thing I generally advocate for is keeping -proposed as empty as possible so that it's more obvious when this sort of thing is languishing12:55
michithat would be good, yes.12:55
michiIt’s not going to happen without some rules and some process to enforce it though.12:56
michito me, it’s all about catching problems early.12:56
cjwatsonwell, that was what I was trying to do with the whole +1 maintenance effort12:56
michiIf we find out three days before a release freeze when something fails in a silo, that’s no good.12:56
cjwatsonbut it also requires getting people to care about things outside their little bubble12:57
michiSends everyone into headless chicken mode.12:57
michiI have no simple answers :(12:57
michiBut I know from bitter experience that, if we catch something early, that’s exponentially cheaper than catching it late.12:58
=== alecu-natholiday is now known as alecu
michiLooking at the list of reverse deps for capn proto, quite a few of those are simply because scopes-api uses capn proto, I think.12:59
michiBut, getting focus...13:00
michiIs there a way to get silo 37 unblocked?13:00
michithe test failure is fixed now.13:00
michiSo, whatever is holding up silo 37, it ain’t us, to the best of my knowledge.13:00
cjwatsonlet's see13:00
cjwatsonI'll see what happens if I pick a random one of those tests and rerun it13:01
cjwatsonbut we still have a new ubuntu-touch-meta only in -proposed, so I'm a little doubtful13:01
michiThe regression tests for scopes-shell fail for xenial in silo 13 because of this, I think: https://objectstorage.prodstack4-5.canonical.com/v1/AUTH_77e2ada1e7a84929a74ba3b87153c0ac/autopkgtest-xenial-ci-train-ppa-service-landing-037/xenial/i386/q/qtcreator-plugin-ubuntu/20160210_000636@/log.gz13:03
michiRight at th end of the log.13:03
cjwatsonI think that's ubuntu-sdk-libs being uninstallable which is basically due to the same thing13:04
cjwatsonWe probably just need to force the tests to be run with a slightly larger trigger set13:04
michiIs that the same problem as for ubuntu-touch-meta?13:04
michiI’m not aware of all the dependencies/groupings :(13:04
michiSo there is nothing we can do for the moment.13:07
michiSomehow, qtcreator-plugin-ubuntu needs to pass first.13:08
michiI have no idea how to accomplish that.13:08
* cjwatson tries running unity-scope-click tests with unity-scopes-api added to the trigger13:15
cjwatsonmichi: (now waiting to be able to see the output of the test I triggered)13:37
michicjwatson: We just decided to unbundle scopes-api from silo 37.13:37
ogra_hmm, i'm running the ubuntu-pd image on my N7 now ... are the Xorg app launchers supposed to do anything ?13:37
michiWe are going to land the scopes-api fix separately, which should unblock a whole bunch of other packages.13:37
ChrisTownsendubuntu-qa: Hi, any ETA on when testing will commence for https://requests.ci-train.ubuntu.com/#/ticket/982 ?13:38
ogra_(i definitely dont even get feedback clicking something like gedit or firefox)13:38
michicjwatson: What test did you trigger?13:38
michicjwatson: Ah: tries running unity-scope-click tests with unity-scopes-api added to the trigger13:39
michiI don’t even know what a trigger is in this context :(13:40
cjwatsonmichi: can you explain why you're unbundling?  I really think that will make matters strictly worse13:40
* cjwatson checks13:40
michiThe list of reverse deps you pasted earlier had a bunch of packages in it that are failing only because of the inability of scopes-api to make progress.13:40
jibelChrisTownsend, is it for OTA9.5 or 10? there is no milestone13:40
michisilo 37 bundles the scopes-api change with a bunch of other fixes.13:41
cjwatsonmichi: but it isn't going to be possible for unity-scopes-api to make independent progress13:41
cjwatsonmichi: please don't unbundle it13:41
michisilo 37 doesn’t make progress because there is a problem with one of scopes-shell’s dependencies.13:41
jibelour plate is already full but if it's urgent we can allocate some time.13:41
jibelChrisTownsend, ^13:41
cjwatsonmichi: it's one of the things that has to be rebuilt against libjsoncpp1; therefore it has to go together with the other things being rebuilt against libjsoncpp113:41
michicjwatson: The *only* change for scopes api is that we have disabled the failing test.13:41
michiOh Christ… :(13:42
ChrisTownsendjibel: Umm, who decides that?  I guess OTA9.5 as it's needed asap.13:42
michiAre you sure that *scopes-api* is breaking because of json? I have seen no evidence of that.13:42
cjwatsonmichi: it's also a rebuild against libjsoncpp1, perhaps incidentally but necessarily13:42
michiAs best as I know, scopes-api works as well with the new json as with the old one.13:42
cjwatsonmichi: it's not that it's breaking, but promoting the libjsoncpp change requires everything to be rebuilt in sync13:43
michiOh my God :(13:43
michipstolowski: ^13:43
cjwatsonmichi: if you unbundle unity-scopes-api, not only will it make little progress on its own but it will also make it strictly more difficult to make progress on unity-scope-click13:43
michiI don’t know how to help anymore.13:43
cjwatsonso please don't13:43
cjwatsonjust leave it, I'm working on your silo13:43
michiI really tried all day to move this ahead somehow, but I keep running into dead ends :(13:44
michipstolowski: Please co-ordinate with cjwatson about this.13:44
michiI’m about to sign off.13:44
cjwatsonthe trigger in this case is what the autopkgtest system considers to be the cause of the test attempts; in this case it's used to construct apt pins for packages to pull from later than xenial13:44
cjwatsonadding a trigger is how we arrange for multiple things to be tested in combination13:44
michicjwatson: You just threw more Chinese at me than I can read. I’m way out of my depth here...13:44
cjwatsonbut that gets way harder if they're in separate silos, we'd basically have to forcibly ignore test results13:45
michiOk, re-reading this, I sort of get the gist of it.13:45
cjwatsonin this case they're necessarily combined in terms of the dependency graph, so keep them together13:45
michithe autopkg tests for scopes-api will pass in the silo.13:45
michiWe actually run them as part of our unit tests.13:45
cjwatsonyeah, they should do as soon as I figure out the correct trigger to pass13:46
michiI’m virtually certain that our autopkg test will pass even with proposed.13:46
cjwatsonI'm just being slowed down by not being able to see results immediately13:46
michiBecause I strongly suspect that the autopkg test for scopes API won’t touch any code path related to json.13:46
cjwatsonthat seems likely13:46
michiSo, we *might* just get away with a separate scopes-api landing from silo 24.13:47
cjwatsonseriously, no13:47
michiI’ll leave it up to you and pstolowski to decide.13:47
cjwatsonplease please please do not muck with this13:47
michiI’m about to go to bed. It’s nearly midnight here. I won’t physically be able to muck with anything for the next seven hours.13:48
cjwatsonthat was a collective plea :)13:48
michiI’m out of my depth here, so I’m happy to leave it to you.13:48
cjwatsonI'm reasonably sure I can get 37 sorted out, it will just take a little time13:48
cjwatsonand I will get back to you/pstolowski if I run into a problem that isn't just infrastructural13:49
michiJust one thought to keep in mind for the next sprint, maybe. The complexity of what has just happened tells me that we must be doing something wrong. I thing the root problem might be late detection of issues caused by new packages in proposed. If we can get to and test with proposed early, I suspect we’ll mitigate the problems.13:49
michicjwatson: Thanks!13:49
cjwatsonI think the basic thing that went wrong here is allowing too many separate transitions to become entwined due to inattention to the size of -proposed13:50
cjwatson(possibly collective)13:50
michiAnd us not even knowning about the oncoming train, such as capn proto and acc.13:50
michiSo, we were in blissful ignorance all along, until yesterday, even though the actual problem has been there for months.13:51
cjwatsonthe other thing is, if this weren't phone, I'm pretty sure somebody would've just fired off rebuilds ages ago and been done with it13:52
cjwatsonbut because phone is all super-careful about everything, the people who try to generally care about -proposed backlog are often averse to dealing with it13:52
cjwatsonand as a result this can end up backing up in a way that wouldn't have been a problem if dealt with earlier :-/13:52
jibelChrisTownsend, someone will verify it today.13:53
ChrisTownsendjibel: Ok, thanks.13:54
dobeywell i guess the queue is emptied now13:55
ChrisTownsendjibel: Just for my info, is there a freeze going on that would normally cause this to be delayed?13:55
ChrisTownsendjibel: ie, if there wasn't a critical bug being fixed?13:55
jibelChrisTownsend, there is no freeze, just a busy week.13:57
ChrisTownsendjibel: Ok, I understand13:57
dobeycjwatson: so you understand these pkgProblemResolver errors? i'm a little confused about them14:04
dobeyhmm, what a mess14:09
cjwatsondobey: more or less.  excessive isolation for things that need to be tested in combination14:10
dobeycjwatson: ah. and you're doing some work to hopefully squeeze things through?14:11
dobey(sorry if this is repetitive, but i just got on line and trying to catch up :)14:11
cjwatsondobey: that is my hope14:17
dobeycjwatson: great. thanks!14:23
AlbertAtrainguards: could someone retrigger the xenial/amd64 build in this silo ppa: https://launchpad.net/~ci-train-ppa-service/+archive/ubuntu/landing-064/+packages14:30
robruAlbertA: done14:31
AlbertArobru: thanks!14:31
robruAlbertA: you're welcome14:31
pstolowskicjwatson, hey, i see you and michi had a lengthy conversation... the conclusion is not to try to land unity-scopes-api with a separate silo right?14:42
cjwatsonpstolowski: indeed, I think that will only make matters worse14:43
cjwatsonpstolowski: doing that won't decouple it, it will just mean we have to figure out how to do cross-silo autopkgtesting and argh14:44
pstolowskicjwatson, fair enough.. i fully trust your judgement14:44
davmor3ping davmor214:45
davmor2pong davmor314:46
dobeyoh no, the machines are taking over14:48
davmor3ChrisTownsend, ping well this is xchat mako \o/14:53
ogra_neat !14:53
ChrisTownsenddavmor3:  Awesome!  Thanks for confirming!14:53
davmor3jibel, so looks like this is working for the ever \o/ now to test the rest :)14:54
cjwatsonrobru: So, I need to copy some packages from -proposed into a silo in order that they can all be autopkgtested together.  Do I need to add them to the source packages list in the request, or does that not matter if they're already in -proposed anyway?15:15
jibelrvr, didn't you approve silo 46 (click) 2 days ago?15:16
robrucjwatson: the source packages list in the request will be auto-filled, just copy the packages15:16
cjwatsonrobru: thanks, will try it15:17
dobeyoh wow that's kind of awful15:27
dobeycjwatson, robru: does that mean the CI machinery is going to want to have the new capnproto and jsoncpp in vivid-overlay too?15:28
robrudobey: only if you've pushed packages to vivid overlay that depend on them? I dunno15:29
dobeyrobru: well i see the status says "Ready to build" in vivid for those packages now since cjwatson copied the xenial-proposed binaries in15:30
robrudobey: what request are you even talking about. i don't see it15:31
cjwatsonis it possible it doesn't understand that those source packages are only needed in xenial and not vivid?15:31
cjwatsonrobru: 97515:31
dobeyrobru: 97515:31
dobeycjwatson: right, that's what i'm thinking15:32
robrucjwatson: dobey: well you guys have the silo set as a dual silo, so that means it wants everything in xenial and vivid15:32
cjwatsonthere's another approach available for this, but this would be the simplest one15:32
cjwatsonrobru: no way to make an exception for a few packages then?15:33
dobeyreally the autopkgtests should be running against proposed15:33
robrucjwatson: no, a dual silo is a dual silo. but if those are manual packages you can just copy the vivid versions into the silo and it'll be happy with that.15:33
cjwatsondobey: well, I'd like to avoid pulling in random other stuff I don't know about15:34
cjwatsonand pitti tells me silos in general deliberately don't because that causes other problems15:34
dobeycjwatson: right, but i mean, i don't see how this hasn't been a problem already, given the PPAs build against -proposed.15:34
cjwatsondobey: maybe it has occasionally and has been worked around in similar ways15:34
cjwatsonrobru: does that seem reasonable to you?  it potentially results in duplicate packages in the overlay15:35
dobeyrobru: will it be happy if the same versions are already in the target?15:35
dobeyi guess it will be ok15:35
cjwatsonit'll be fine in xenial, I just don't want to create cruft in vivid15:35
robrucjwatson: what do you mean duplicate? if they're packages that are already in the overlay just copy what's in the overlay. train will just see it as a successfully published package rather than something needing to be built15:35
dobeycjwatson: i guess they won't be duplicate in the overlay. it'll just treat them as already copied15:35
cjwatsonrobru: I suspect these are only in the base archive, not the overlay15:36
cjwatsonno particular reason they would've been overlaid at any point15:36
cjwatsonwell, not all of them anyway15:36
dobeywell yeah15:36
dobey-meta will be in the overlay15:36
dobeybut jsoncpp and capnproto probably not15:36
robrucjwatson: ok, my recommendation is just copy whatever from vivid archive and then delete from overlay ppa after publishing the silo.15:36
dobeyi think we do just need to fix these autopkgtests to always run against proposed though15:37
cjwatsonpitti has just finished telling me how the effects of that were worse15:38
cjwatson15:13 <pitti> as that is in many cases unintended and led to too many blockings15:38
cjwatsonso I mean you can argue it out with him, but he seems to have taken a decision based on data ...15:38
dobeywell, that blockage is going to happen when things get published to -proposed anyway; but sure, it will maybe make things block a bit more in silos15:39
davmor2ChrisTownsend: approved libertine15:45
ChrisTownsenddavmor2: Thank you very much!15:45
dobeyhmm, i guess it's not entirely happy about that15:58
cjwatsonI don't know if the "diff missing" stuff actually matters16:01
cjwatsonhad to copy in cmake too, so there'll be another cycle here16:01
cjwatsonbut at least unity-scopes-api and qtcreator-plugin-ubuntu are now passing16:06
cjwatson(not green on excuses yet but they should be when it next updates)16:06
TrevinhoLaney: can you ACK https://requests.ci-train.ubuntu.com/#/ticket/873 ?16:13
dobeycjwatson: ok, cool, hopefully it is happy now then16:25
cjwatsonrobru: "Diff missing" is going to prevent this getting anywhere, isn't it?16:26
robrucjwatson: define "getting anywhere"16:26
cjwatsonrobru: reaching the "Successfully built" state16:27
dobeyrobru: "qa: ready"16:27
robrucjwatson: it's not going to stop britney running on it16:27
robrucjwatson: also trivially fixed by running the build job with "DIFF_ONLY" set16:27
* cjwatson will do that16:28
LaneyTrevinho: what is:16:43
Laney-rm -f debian/tmp/usr/share/compiz/networkarearegion.xml16:43
Laney-rm -f debian/tmp//usr/lib/compiz/libnetworkarearegion.so16:43
TrevinhoLaney: it was a plugin that we didn't use in ubuntu anymore since long time.. It was still built, though. So I just disabled it from CMake instead of removing the built bits16:46
LaneyTrevinho: I should publish it?16:52
dobeyyay, got past installation now16:56
cjwatsondobey,pstolowski: can you have a look at https://objectstorage.prodstack4-5.canonical.com/v1/AUTH_77e2ada1e7a84929a74ba3b87153c0ac/autopkgtest-xenial-ci-train-ppa-service-landing-037/xenial/armhf/u/unity-scope-click/20160210_165255@/log.gz ?17:03
pstolowskicjwatson, yes, that's the test flakiness that is going to be disabled with https://code.launchpad.net/~dobey/unity-scope-click/temp-no-int-tests/+merge/28507917:05
dobeycjwatson: ah, i can fix that. i need to disable it in two places17:05
cjwatsonwell, the test is retrying anyway for reasons, if it's just flaky then maybe it will pass next time17:06
dobeycjwatson: well, the MP in that silo is supposed to disable the tests for now. but i forgot we were actually running them against the installed scope in the autopkgtests too, so need to update the branch to disable there and rebuild unity-scope-click in that silo17:06
cjwatsondobey: ok17:08
dobeycjwatson: and just started the rebuild. hopefully it will be quick17:08
cjwatsonit'll retrigger unity8 autopkgtests, so that'll take a while17:09
cjwatsonbut with any luck this should all go through without my intervention now17:09
cjwatsonand hopefully this can get quickish QA17:10
dobeyok, i'm going to go get lunch.17:15
robrualright! fun times. anybody need the train urgently for the next 2 hours? I'm gonna do a big rollout...17:37
robru(britney/autopkgtests/ppas unaffected, just source builds might be a bit delayed)17:38
TrevinhoLaney: yeah, if you can...17:58
=== alan_g is now known as alan_g|EOD
TrevinhoLaney: why that? ^18:00
Laneyit's working now18:02
dobeyrobru: ugh, your timing sucks ;)18:12
dobeyi think i found a bug in the autopkgtests running in silos, but i'm not quite sure how/where to report it20:02
dobeyor it's not a bug and things are slow enough to just be annoying/confusing20:03
dobeyi wonder if the currently "running" stuff has passed yet20:11
dobeyok, who can i bug from qa right now20:15
dobeyalesage: can you be our qa savior?20:16
alesagedobey, not sure yet, more info pls20:18
dobeyalesage: xenial -proposed is highly blocked at the moment, and silo 37 fixes it. we've been trying to get everything sorted so qa can review it (it's a dual landing silo), and it's about ready. but we want to get it through asap to get things unblocked20:21
dobeyso there are some silos that can't build on xenial at the moment as a result of this blockage, which means they can't land until it's unblocked20:22
alesagedobey, not sure what you're asking for20:24
dobeyalesage: to get https://requests.ci-train.ubuntu.com/static/britney/xenial/landing-037/excuses.html onto the qa trello, tested, and passed, so we can publish it and unblock everyone20:25
dobeynot the excuses20:25
alesagedobey, unlikely we'd get to it until someone better qualified can answer :) , also not able to parse test failures20:27
dobeyalesage: the one failed on that excuses page is the older unity-scope-click build; the newer one would pass.20:27
dobey(and did pass)20:28
dobeyalesage: whom would be better qualified to answer, that would be awake in our TZs?20:38
dobeywell, awake and working, as opposed to awake and at the pub20:38
alesagedobey, we're all busy with a release atm, unlikely to find an ear until EU AM20:40
cjwatsondobey: I just poked it22:53
cjwatsondobey: (I only bothered with armhf)22:54
cjwatsondobey: https://requests.ci-train.ubuntu.com/static/britney/xenial/landing-037/excuses.html \o/23:46
cjwatsonI guess the train will notice in a bit23:46
cjwatsonmichi: 37 passes autopkgtests now.  What I eventually did was copy various other packages into the silo from -proposed which had previously needed to be rebuilt against libjsoncpp1, in order that the system knows to test them all together.23:47
cjwatsonThe copy back to -proposed when you publish should be a no-op.23:47

Generated by irclog2html.py 2.7 by Marius Gedminas - find it at mg.pov.lt!