| === vrruiz_ is now known as rvr | ||
| balloons | so pitti I was speaking with elopio yesterday about autopkgtest and popey's experiences. There doesn't seem to be a good way to test old images; and in popey's case, he had actually flashed a new image, but presumably the archive had updated anyway. So he was unable to use autopkgtest to run his tests. I'm concerned about running into scenarios where autopkgtest cannot work | 15:17 |
|---|---|---|
| pitti | balloons: I talked to popey this morning; he flashed an n7 image from the release channel, which presumably hasn't gotten a promotion in ages | 15:18 |
| pitti | I didn't hear about his experience with devel-proposed any more | 15:18 |
| pitti | balloons: this isn't specific to autopkgtest -- even with ro-apt-update he got uninstallable test dependencies | 15:18 |
| balloons | pitti, ahh, I thought he flashed -proposed.. Still, the basis of having a more or less untestable image remains | 15:19 |
| pitti | balloons: because the archive was just too far ahead (library transitions, Breaks: etc | 15:19 |
| pitti | correct, such an image is untestable | 15:19 |
| pitti | as tests rely on uninstalled test dependencies which just simply don't exist any more | 15:19 |
| pitti | if we want to be able to do this, we need to keep those old test deps (matching the time when the image was built) and corresponding Packages.gz indexes on archive.u.c. | 15:20 |
| balloons | pitti, so that's kind of a problem . . . I guess the old way of running tests would still work to the same extent on such an old image, but I'm concerned we don't have an answer for this | 15:21 |
| pitti | balloons: no, it wouldn't | 15:21 |
| pitti | if apt says "your test deps are uninstallable" then it doesn't matter whether adt-run or phablet-test-setup requested those packages | 15:21 |
| pitti | unless you run something like dist-upgrade, but that would then entirely fail the point | 15:21 |
| balloons | pitti, well sure.. but let's say your test has no dependencies beyond autopilot. It seems today you still cannot run it on old images as autopilot has been removed from the image | 15:22 |
| pitti | balloons: as long as current vivid AP is installable, things are fine | 15:23 |
| pitti | but e. g. if unity8-autopilot from the current archive wants dependencies which are incompatible to what is on the image, you lose | 15:23 |
| balloons | pitti, well right.. ubuntu-ui-toolkit-autopilot as well | 15:24 |
| pitti | it also wouldn't be the unity8-autopilot which would actually match the installed unity8 (but that's yet another problem) | 15:24 |
| pitti | right | 15:24 |
| pitti | balloons: FYI, http://paste.ubuntu.com/10386645/ is what balloons got from the n7 devel channel | 15:24 |
| balloons | so.. I am happy test dependencies are gone from the images (yay!), but we've actually made the situation worse since we can't test old images | 15:24 |
| balloons | and ofc, I don't see these issues. It still works for me on RTM and -proposed ;-) | 15:25 |
| pitti | right :) | 15:26 |
| pitti | RTM doesn't change much any more, and -proposed images are (usually) new enough | 15:26 |
| elopio | good morning | 15:26 |
| balloons | so pitti apart from cloning the archive and keeping old versions of packages, what can we do in these cases? | 15:27 |
| balloons | I guess I'm concerned we have test results we can't repeat in the future | 15:27 |
| * elopio gets in line for pitti's attention. | 15:32 | |
| elopio | pitti: when we use --click com.ubuntu.camera as an argument to adt-run, it will install the deps, install the camera tests, set up the app armor rules and run the tests, right? | 15:32 |
| elopio | is there a way to do only the setup part without running the tests? | 15:32 |
| pitti | elopio, balloons: in meeting, bbl | 15:33 |
| balloons | elopio, why would you want to do that? (as it's all gone when the test is done) | 15:34 |
| elopio | balloons: I want the camera helpers, but I don't want to run the camera tests. I have a test that takes a picture and puts it into a contact. | 15:34 |
| elopio | and I see that adt-run does some things I don't understand for the app armor rules, when running in click. I think we need them in order to introspect the camera during this test. | 15:35 |
| elopio | actually, this is like making a debian package depend on a click package. | 15:36 |
| elopio | fgimenez: how late is it for you? | 15:36 |
| elopio | do you have time to do some pair programming? | 15:36 |
| balloons | elopio, but you need it's setup to stick around while you run other tests then right? | 15:46 |
| elopio | balloons: the tests we are running are defined as deb autopkgtests | 15:48 |
| elopio | so what we need is that at the same time it's setting up the dependencies for those tests, it sets up the camera. | 15:49 |
| fgimenez | elopio: it's fine for me now, i have still one hour | 15:51 |
| pitti | balloons: well, snapshotting at least the part of the archive that we need to run tests is pretty much the only thing that's guaranteed to fix this problem | 15:56 |
| pitti | balloons: aside from the, I think --setup-commands ro-apt-update is the next best hack, which sometimes works | 15:56 |
| pitti | elopio: no, not at the moment; we have --shell to interrupt the process *after* a test run, but not in between setup and start of the test | 15:57 |
| pitti | elopio: so you want something like a --shell-before-the-test-starts? | 15:58 |
| balloons | pitti, does bundling depends in a click make sense? | 15:59 |
| balloons | not the application click . . . | 15:59 |
| elopio | pitti: no, what I really want is to make a deb test depend on a click package. Install the dependencies of the click package and set up the apparmor rules it needs. | 16:00 |
| pitti | balloons: ah, like building a click with all the test debs included, at image build time? yeah, could be (although this would be fairly big) | 16:01 |
| elopio | as a workaround, I saw that if I added --click com.ubuntu.camera, everything I wanted was done. | 16:01 |
| elopio | fgimenez: I've just taken this card: https://code.launchpad.net/~canonical-platform-qa/ubuntu-sanity-tests/adding-installing-application-test/+merge/244395 | 16:01 |
| elopio | letss do together the part for waiting for the dialog. Lets say, in 10 minutes, so I can read the code first. | 16:02 |
| balloons | pitti, yes exactly. I suppose we could build a click with autopilot and the helpers as a base. The other depends for tests generally are just python packages which should be fine under the current setup | 16:02 |
| pitti | or imagemagic etc., but indeed they are less prone to getting out of sync | 16:03 |
| balloons | right, less prone to getting out of sync, but also if you must use the apt-get update hack; they are not likely to pull depends that break things | 16:04 |
| fgimenez | elopio: ok | 16:04 |
| pitti | balloons: right, that's what I meant | 16:04 |
| === roadmr is now known as roadmr_afk | ||
| elopio | fgimenez: https://plus.google.com/hangouts/_/canonical.com/qa | 16:14 |
| elopio | for when you are ready. | 16:14 |
| elopio | wait, no, that one will be needed, sorry. | 16:14 |
| elopio | fgimenez: https://plus.google.com/hangouts/_/canonical.com/qa-pair | 16:14 |
| balloons | pitti, ok.. I know when I first ran into the archive out of sync issue we spoke about this but I'll ask again. Can we automagically fallback to trying ro-apt-update when things fail from the archive? It's very confusing for folks running tests. | 16:15 |
| balloons | I would also suggest we give a better message in the event of these failures so people understand | 16:16 |
| elfy | afternoon balloons | 16:17 |
| balloons | afternoon elfy | 16:19 |
| balloons | pitti, ahh I guess you would advocate for just using the flag then which is fair enough. Any thoughts on making a better error message should it occur though? | 16:38 |
| === roadmr_afk is now known as roadmr | ||
| pitti | balloons: I really don't like doing it automatically; if you are at that point, then things are already wrong (you don't get the unity8-autopilot that matches the installed unity8) | 16:45 |
| pitti | this would just cause more so-called "flaky" tests | 16:45 |
| balloons | pitti, right I'm agreeing with you. Really running it is a hack that most folks shouldn't even perform. If it doesn't work, use a newer image. But to that end, could we make a nice error message? | 16:46 |
| pitti | balloons: hm, I find apt's "package not found" error message clear enough, but I've known it for so long that my definition of "clear" might not be very universal :) | 16:48 |
| pitti | so if you have a suggestion, bug report appreciated | 16:48 |
| pitti | balloons: I documented the ro-apt-update bit on https://wiki.ubuntu.com/Touch/Testing now, but that's about as far as I would go TBH | 16:48 |
| balloons | pitti, at the bottom of popey's log he gets: adt-run [10:15:31]: ERROR: unexpected error: test dependencies are unsatisfiable | 16:49 |
| pitti | i. e. not using it automatically, and for sure we must not use it in CI | 16:49 |
| balloons | while understandably that makes sense to you, it's meaningless to him. I'm suggesting tweaking the line to say something like "adt-run [10:15:31]: ERROR: unexpected error: test dependencies are unsatisfiable. blah blah, use a newer image " | 16:50 |
| balloons | give them a suggestion on how to make it work | 16:50 |
| pitti | ah, tricky | 16:51 |
| pitti | "image" doesn't make any sense with the null, chroot, ssh, or LXC runners, and often not even with QEMU | 16:51 |
| pitti | but maybe something similar | 16:51 |
| balloons | ohh so true | 16:51 |
| pitti | "upgrade to the lastest version" or so | 16:51 |
| pitti | "upgrade your system"? | 16:52 |
| * pitti needs to run now, sorry | 16:52 | |
| balloons | pitti, sure, I'll file a bug. It's not huge but maybe there is something generic enough we can include in there | 16:52 |
| balloons | thanks for talking it through | 16:52 |
| === roadmr is now known as roadmr_afk | ||
| === roadmr_afk is now known as roadmr | ||
| elopio | veebers: lets go into the same hangout in 35 minutes. | 19:55 |
| veebers | elopio: sounds good | 19:56 |
| veebers | elopio: I'm just reflashing now and will try the updated version of adt-run | 19:56 |
| veebers | (as well as changing out of this wet top, stupid cup) | 19:56 |
| elopio | :) ok. | 19:56 |
| === roadmr is now known as roadmr_afk | ||
| balloons | Letozaf_, would you like to have a look at calendar again? | 20:13 |
| Letozaf_ | balloons, yes sure | 20:14 |
| Letozaf_ | balloons, to be honest I was already f idling with it: https://code.launchpad.net/~carla-sella/ubuntu-calendar-app/agenda-view-test | 20:15 |
| balloons | or perhaps doug5 as well? It needs some love. They've added lots of new features and the coverage is lacking. In general the app is much different than before and could use some re-writes to test each of the pages | 20:15 |
| balloons | Letozaf_, nice! | 20:15 |
| Letozaf_ | balloons, :) still have a little issue I hope to fix | 20:16 |
| balloons | Letozaf_, so I'll work on filing bugs to direct what I'm after | 20:18 |
| Letozaf_ | balloons, oh yes, sure, that would be great | 20:19 |
| Letozaf_ | balloons, you file bugs and I go after them :) | 20:19 |
| Letozaf_ | balloons, just one question on calendar app, am I the only one with this issue: https://dl.dropboxusercontent.com/u/10083981/Ubuntu%20errors/Screenshot%20from%202015-02-24%2021%3A24%3A36.png | 20:26 |
| Letozaf_ | balloons, I get this when launching autopilot3 launch -i Qt qmlscene ../../calendar.qm | 20:26 |
| Letozaf_ | balloons, I am wondering if I have something broken on my PC | 20:27 |
| balloons | Letozaf_, that's odd.. I was playing with calendar last week and I didn't see that | 20:27 |
| * dobey wonders how to submit branches to autopkgtest | 20:27 | |
| Letozaf_ | balloons, this happened to me yesterday | 20:27 |
| Letozaf_ | balloons, but I will try to get it form trunk now and see if it's the same | 20:28 |
| * balloons points dobey to git | 20:28 | |
| dobey | balloons: i know it's in git. i don't know the process pitti uses to manage it though | 20:29 |
| Letozaf_ | balloons, calendar app from trunk has the same issue | 20:32 |
| balloons | dobey, afaik you need an account on alioth; then you can push to master the same as pitti | 20:32 |
| dobey | yeah, i don't have that. | 20:33 |
| balloons | right, so I would clone the repo and mail him | 20:33 |
| dobey | question is, how annoying will it be if i push a branch to lp and make an MP there | 20:33 |
| dobey | is it more or less annoying than this snow here right now | 20:35 |
| balloons | snow? oh my! | 20:42 |
| Letozaf_ | balloons, dobey it's also snowing here in Italy :P | 20:42 |
| dobey | but i guess you're not in sicily | 20:43 |
| elfy | any snow here has melted before it gets far outside of the clouds - we've had some of the melted stuff :p | 20:44 |
| Letozaf_ | dobey, no northern Italy, near Venice, more or less | 20:44 |
| Letozaf_ | elfy, :) | 20:45 |
| veebers | elopio: you still there? I think ho froze | 20:47 |
| balloons | it's spring here.. no more cold think | 20:47 |
| elfy | it's the 1st March soon - and as far as I am concerned Spring starts then | 20:48 |
| Letozaf_ | balloons, here in Italy it will still be cold till the end of March, more or less | 20:48 |
| elfy | wxl: but that is only a guess - I had a quick try at doing a policy for LSC - but that didn't work | 20:49 |
| wxl | thanks elfy | 20:49 |
| elfy | certainly seems to be what it points at | 20:49 |
| elfy | *shrug* | 20:49 |
| balloons | elfy, I'm still shocked by home warm the UK is during winter.. You are in the same zone as me! | 20:52 |
| balloons | actually a bit higher :-) | 20:52 |
| elfy | balloons: funnily enough - we're at the same latitude as Newfoundland I think | 20:53 |
| knome | home warm? | 20:53 |
| elfy | Gulf Stream \o/ | 20:53 |
| balloons | yes.. something like 2k km or more difference and yet.. you have a warmer zone rating | 21:11 |
| flexiondotorg | elfy, Hey | 21:56 |
| elfy | flexiondotorg: so - go to tracker and login - you should then get admin rights | 21:57 |
| elfy | you might need to check details when you login to make sure you do so with mate release team rights | 21:57 |
| flexiondotorg | elfy, Before I start. cjwatson enabled PowerPC earlier. Does that need adding to the tracker? | 21:57 |
| elfy | probably - lets do one thing at a time :) | 21:57 |
| flexiondotorg | elfy, OK. | 21:58 |
| flexiondotorg | Logged in. | 21:58 |
| elfy | ok - so find the Mate products | 21:58 |
| flexiondotorg | Yep. | 21:58 |
| === roadmr_afk is now known as roadmr | ||
| elfy | click in the boxes (or the main one) | 21:59 |
| elfy | should select your 2 products | 21:59 |
| flexiondotorg | Under the Builds tab? | 21:59 |
| elfy | mmm - tab? | 21:59 |
| elfy | http://iso.qa.ubuntu.com/qatracker/milestones/334/builds | 21:59 |
| elfy | there | 22:00 |
| flexiondotorg | http://iso.qa.ubuntu.com/admin/config/services/qatracker/builds | 22:00 |
| elfy | nope - not in admin this bit :) | 22:00 |
| elfy | then once you've clicked on mate on that link - at the bottom - request a rebuild - update rebuild status | 22:01 |
| flexiondotorg | OK, checked. | 22:01 |
| elfy | there you go - rebuilding :) | 22:01 |
| flexiondotorg | And rebuild selected. | 22:01 |
| flexiondotorg | Ooooh. The power. | 22:02 |
| flexiondotorg | elfy, Thanks. | 22:02 |
| flexiondotorg | elfy, How do I know where the build job are? | 22:02 |
| flexiondotorg | Ah, links have updated. | 22:03 |
| flexiondotorg | elfy, How do I raise a bug against a QA tracker build? | 22:05 |
| elfy | not sure I understand | 22:05 |
| flexiondotorg | I see some flavour have bugs listed against the beta 1 builds. | 22:05 |
| elfy | you seriously doing PPC for Mate ? | 22:05 |
| flexiondotorg | How do you do that? | 22:05 |
| elfy | oh right | 22:06 |
| flexiondotorg | elfy, Yes. Have a team all lined up. | 22:06 |
| elfy | mmm | 22:06 |
| elfy | so when your people are testing something - they have to login to tracker | 22:06 |
| flexiondotorg | From within the live session? | 22:07 |
| elfy | flexiondotorg: ok - there are 2 sides to this | 22:07 |
| elfy | 1st - finding a bug while running the session - you need to report the bug from there - say ubuntu-bug pulseaudio | 22:08 |
| elfy | do the whole reporting thing - then you end up with bug # | 22:08 |
| flexiondotorg | OK | 22:09 |
| elfy | you can just write this down if you want to - I tend to do so many tests - that logging into ubuntu one is just a pain | 22:09 |
| elfy | then go to tracker - example 64bit gets you to the 4 tests you have http://iso.qa.ubuntu.com/qatracker/milestones/334/builds/89669/testcases | 22:09 |
| flexiondotorg | OK | 22:10 |
| elfy | once there - live session gets you to http://iso.qa.ubuntu.com/qatracker/milestones/334/builds/89669/testcases/1303/results | 22:10 |
| elfy | that is where you report the bug against the build - go to bottom - fail/pass/in progress | 22:10 |
| elfy | then critical bugs - ie - this fails completely example | 22:11 |
| elfy | real live stuff for Xubuntu 64bit manual partitioning http://iso.qa.ubuntu.com/qatracker/milestones/334/builds/89654/testcases/1302/results | 22:11 |
| flexiondotorg | Thanks. Got it | 22:13 |
| elfy | make more sense now ? | 22:13 |
| flexiondotorg | Yep. | 22:13 |
| elfy | as you've got rights for Mate - if someone has for instance failed something they perhaps shouldn't have - pencil on the far right allows you to edit that result | 22:14 |
| flexiondotorg | When I request a rebuild can I see the build log? | 22:14 |
| elfy | I guess so - I never bothered - I just wait for it to appear :p | 22:14 |
| flexiondotorg | OK. | 22:16 |
| flexiondotorg | And I see the other flavour have different test suites. | 22:16 |
| flexiondotorg | Can I add some of those to Ubuntu MATE? | 22:16 |
| elfy | you should be able to yep | 22:17 |
| elfy | real life example? | 22:17 |
| flexiondotorg | OEMConfig | 22:17 |
| elfy | ok - now you DO you want to hit admin :) | 22:18 |
| flexiondotorg | OK | 22:19 |
| elfy | testsuites | 22:19 |
| elfy | mate is right at the top - on the right - Edit | 22:19 |
| flexiondotorg | OK, think I see what I need. | 22:20 |
| elfy | you want to add a testcase | 22:20 |
| flexiondotorg | elfy, OK | 22:20 |
| flexiondotorg | In Admin I see 4 tabs | 22:21 |
| flexiondotorg | Summary Products Series Build | 22:21 |
| elfy | sorry - was ahead of you | 22:21 |
| elfy | oh | 22:21 |
| elfy | mmm | 22:21 |
| elfy | you don't see Testcases Testsuites ? | 22:21 |
| flexiondotorg | Not in the tabs. | 22:21 |
| elfy | balloons: ^^ bit of a problem here :) | 22:22 |
| flexiondotorg | Under Products I have entries for MATE 64bit and 32Bit with links to "Linked Testcases." | 22:22 |
| elfy | yea | 22:22 |
| balloons | flexiondotorg is an admin yes? | 22:22 |
| balloons | I see the release team now, he should see it | 22:22 |
| elfy | balloons: should be yep - just not seeing 2 tabs in there | 22:23 |
| flexiondotorg | Just see "Summary Products Series Build" | 22:23 |
| balloons | not everyone gets all the tabs; to be fair I'm not sure how reduced your tab load is | 22:23 |
| elfy | I'm in that release team too temporarily - I see 6tabs | 22:23 |
| balloons | sure, but you likely see more normally or no? | 22:24 |
| elfy | he obviously wants to be able to to deal with adding testcases to testsuites | 22:24 |
| elfy | balloons: that's as many as I always see | 22:24 |
| balloons | right.. I'm saying his perms should be on par with yours | 22:25 |
| elfy | yea | 22:25 |
| elfy | I guess a job for stgraber again :) | 22:25 |
| flexiondotorg | elfy, OK. We can request my uplift in perms as and when. | 22:27 |
| flexiondotorg | But I'd like to add a few more test case to Ubuntu MATE. | 22:27 |
| elfy | flexiondotorg: so what testcase number is this OEM config? | 22:27 |
| elfy | flexiondotorg: yea - but till you have the perms you can't :) | 22:27 |
| flexiondotorg | Hah 😉 | 22:27 |
| elfy | flexiondotorg: try also to bear in mind that in 2 days time I will be on your back about marking the beta's ready ;) | 22:28 |
| elfy | I just wonder if perhaps you'd be better to get what you can right now - there'll be another beta :) | 22:28 |
| flexiondotorg | elfy, That's why I just issues a rebuild so I can test 😃 | 22:29 |
| flexiondotorg | Also, I've taken Thursday off work | 22:29 |
| elfy | yea | 22:29 |
| elfy | I'm just saying if you add foo tests to the suite - you need to test them all :) | 22:29 |
| flexiondotorg | elfy, Yep. I get that. I only want to add one additional for now. But slowly increase the coverage. | 22:30 |
| elfy | who has an OEM Config test? | 22:30 |
| elfy | same one that kubuntu use? | 22:31 |
| * flexiondotorg goes to get the answer. | 22:31 | |
| flexiondotorg | elfy, Yes. | 22:32 |
| elfy | mandatory test? run once? optional? | 22:32 |
| flexiondotorg | So, I gues those options mean. | 22:33 |
| flexiondotorg | Mandatory = You have to do it for each architecture? | 22:33 |
| flexiondotorg | run once = Do it once for any architecture? | 22:33 |
| flexiondotorg | Optional = Don't have to do it? | 22:33 |
| elfy | flexiondotorg: how it works is you add the testcase to the testsuite | 22:33 |
| elfy | then both arch get the same testsuite | 22:34 |
| elfy | if you look at http://iso.qa.ubuntu.com/qatracker/milestones/334/builds/89657/testcases | 22:35 |
| flexiondotorg | Looking now. | 22:35 |
| elfy | kubuntu have to do 6/6 for the mandatory stuff and 1 run once then this http://iso.qa.ubuntu.com/qatracker/milestones/334/builds | 22:36 |
| elfy | assuming no bugs would be 6/6 1/1 and maybe 0/1 or 1/1 for the optional | 22:36 |
| flexiondotorg | OK | 22:36 |
| flexiondotorg | So, all these tests are performed manually. None of them are automated? | 22:37 |
| elfy | I'd make it mandatory tbh - you can change it later once you've perms | 22:37 |
| elfy | and even if there are fails if *you* are happy to mark it ready for release you can regardless of fails and passes | 22:38 |
| elfy | oh - all manual :) | 22:38 |
| flexiondotorg | Understood. | 22:38 |
| elfy | so - I can add this for you now if you want - or wait to see if you get you perms sorted out | 22:39 |
| elfy | but I'm running out of awake :D | 22:39 |
| flexiondotorg | elfy, Let add it when I've actually confirmed other stuff works 😉 | 22:39 |
| elfy | okey doke :) | 22:40 |
| flexiondotorg | elfy, Thanks for your help. | 22:40 |
| flexiondotorg | Sleep well 😃 | 22:40 |
| elfy | I'll try :D | 22:40 |
| elfy | I'm not sure how you'd add PPC - not sure I would have rights to do that | 22:40 |
| elfy | balloons can help you I'm sure | 22:41 |
| elfy | night all :) | 22:41 |
| flexiondotorg | balloons, cjwatson enable PowerPC for Ubuntu MATE earlier. | 22:47 |
| flexiondotorg | balloons, The build infrastructure has had PowerPC support for Ubuntu MATE from the outset. | 22:47 |
| flexiondotorg | balloons, Is there something that needs doing to add PowerPC to the QA tracker? | 22:48 |
Generated by irclog2html.py 2.7 by Marius Gedminas - find it at mg.pov.lt!
