[03:10] <pitti> Good morning
[05:43] <DanChapman> Good Morning all :-)
[07:09] <jibel> good morning
[13:06] <elfy> balloons: is it possible to interrogate the iso.qa areas to find out how much testing has been done?
[13:08] <knome> elfy, you mean a report across different milestones?
[13:24] <Saviq> asac, ping
[13:26] <elfy> knome: yea
[13:27] <knome> elfy, i don't think there's a tool like that but i'm all ears for balloons' answer ;)
[13:27] <elfy> or anything that I can get hold :)
[13:27] <knome> elfy, you should be able to see testing numbers per milestone by simply going under those
[13:27] <elfy> lol - balloons hates me :p
[13:28] <elfy> knome: you mean like the 0/5 mandatory 0/1 run once type things?
[13:29] <knome> yes
[13:29] <elfy> k - knew about those - I mean more like this milestone had these results, this one these results etc - maybe see all since saucy was released
[13:29] <knome> mhm
[13:30] <asac> Saviq: hey how is it going
[13:30] <asac> what can i do?
[13:30] <Saviq> asac, just wanted to let you know that enabling indicators_client tests is probably not needed
[13:31] <Saviq> asac, as we're building it *into* lp:unity8 soon (as in it's merging today)
[13:31] <Saviq> asac, and lp:indicators-client will get deprecated
[13:31] <Saviq> asac, we have some general problems with unity8 tests on the device, though, that we're trying to resolve now
[13:32] <asac> Saviq: thanks for getting back to me. unity8 is afaict scheduled for next week
[13:33] <asac> so you still have some time :)
[13:33] <asac> i anticipated that that test suite will be one of the least easy to fix ones
[13:35] <Saviq> asac, actually it was targeted at device runs from the get go, we just have some generic input issue (that we might've solved already, but need to verify)
[17:05] <DanChapman> balloons, hey mate. You busy?
[17:05] <balloons> hey DanChapman
[17:06] <balloons> trying to finish up video :-)
[17:06] <DanChapman> :-) what on?
[17:09] <balloons> autopilot of course!
[17:09] <DanChapman> sweet!
[17:13] <DanChapman> balloons, when you get a chance I have 2 MP's waiting on your review. But really i want your thoughts on the firefox one
[17:13] <balloons> DanChapman, ohh, right.. I reviewed the ff one, unless you have another
[17:14] <balloons> mmm.. I see :-)
[17:14] <DanChapman> I have just proposed it couple mins ago
[17:15] <balloons> first glance these both look nice
[17:15] <balloons> on the ff stuff, it's always going to be a bit of a pain because we can't introspect it properly
[17:15] <balloons> trying to do something useful while minimizing errors is always the key
[17:15] <balloons> I like the idea of keeping it local and opening a file
[17:17] <balloons> I don't see where you killed the second window stuff -- did you/
[17:19] <DanChapman> Running the current test many times today on a few occasions it opened a second tab while it was typing. Have no idea how though, but it was giving me the same output as the failed test on jenkins
[17:21] <balloons> well we do open a new tab
[17:21] <balloons> timing is everything.. can we add asserts in there to time things better?
[17:24] <balloons> I know we're so limited without introspection
[17:24] <balloons> have to be creative
[17:24] <balloons> to some extent we're sending keystrokes blind
[17:33] <DanChapman> balloons, sorry had screaming child to deal with. Errm well i think asserting we have gone back to a single tab will probably help. AT the mo it closes it without 'checking' that it has gone back to one tab. I'll see where else i can slow it down a bit
[17:33] <balloons> DanChapman, sounds like a plan :-)
[17:43] <DanChapman> balloons, have slowed that test down. let me no what you think
[18:46] <WebbyIT> balloons: tonight there is Automated Testing Workshop? Can it be a good tutorial? :)
[18:46] <balloons> WebbyIT, there is a workshop tonight
[18:46] <balloons> I think in about 4 hours
[18:47] <balloons> if i remeber right :-)
[18:47] <balloons> If you want a jumpstart, read over the tutorials and workshop intro and FAQ..
[18:47] <balloons> :-)
[18:47] <balloons> https://wiki.ubuntu.com/Touch/CoreApps/Testing
[18:48] <balloons> WebbyIT, first things first if you've not yet gone through it is http://developer.ubuntu.com/resources/cookbook/mobile/how-to-write-autopilot-tests/
[18:50] <WebbyIT> balloons: yes, I red and did it, but I miss that leap to start working in the real :)
[18:50] <balloons> WebbyIT, well then can you make it this evening?
[18:50] <WebbyIT> balloons: sure
[18:50] <balloons> I'm also very close to finishing the video walking through a real contribution
[18:51] <balloons> here's the text version: https://wiki.ubuntu.com/Touch/CoreApps/Testing/ContributeAutopilotTestcase
[18:51] <balloons> I know it's not quite the same, but it's there
[18:51] <WebbyIT> balloons: thanks you :)
[18:54] <balloons> WebbyIT, your most welcome!
[20:44] <adegoodyer> Hey all, hows it going this evening?
[20:45] <balloons> adegoodyer, hello!
[20:45] <adegoodyer> balloons, Hey, how are things?
[20:45] <balloons> busy, but good :-) doing video edits and takes
[20:46] <balloons> so can't really chat, but :-)
[20:47] <adegoodyer> balloons, No worries, I certainly don't envy you this evening! Good luck and sure all will be bafta worthy :oD
[20:47] <balloons> how's it coming for you?
[20:47] <balloons> how's the test?
[20:49] <adegoodyer> balloons, Don't ask lol. I can click the play button to start, but then have no idea how to test for game start.... well I have but the boxes are generated at runtime so don;t even think it's possible to select them for introspection by autopilot.
[20:51] <adegoodyer> balloons, I have committed what I have written so far, so hopefully a good base for someone to further if nothing else.
[20:52] <balloons> adegoodyer, well good.. a start is a start
[20:52] <balloons> do you propose a merge?
[20:52] <balloons> I can probably help with the boxes.. but later :-)
[20:55] <adegoodyer> balloons, No I haven't proposed a merge yet, shall I? The code is pretty much what you executed last night. Not a problem at all, I'm already conscious  of taking loads of your time all the time lol
[20:58] <balloons> adegoodyer, yes of course
[20:58] <balloons> if what you have is a good base, commit it and get it merged
[20:58] <balloons> at the moment there is nothing for dropping letters
[20:58] <balloons> it's best to merge small pieces at  atime
[20:58] <balloons> don't want to write everytestcase and then propose it :-)
[20:58] <balloons> ok.. bbl everyone
[20:59] <adegoodyer> balloons, agreed - ok will do and will leave you alone for a while :oP
[22:07] <balloons> ok whew.. back :-)
[22:07] <balloons> upload time :-)
[22:34] <darran> hi all
[22:34] <balloons> hello darran
[22:44] <darran> is the ubuntu touch testing presentation going to be recorded?
[22:47] <balloons> hey darran if we do any livestreams those will be recorded
[22:47] <balloons> but in addition, I'm uploading a big walkthrough video now to youtube
[22:48] <darran> is the presentation today on livestream or irc?
[22:48] <balloons> So in about another hour or so that will be available to view also, which will help alot if needed.
[22:48] <balloons> darran, a bit of both. I'll be on IRC, with a small intro and taking questions. If someone has a good question, I'll turn on the livestream and answer it
[22:49] <sak> hi guys
[22:49] <balloons> hey sak! good to see you!
[22:49] <sak> same here. Am I in-time for the tutorials?
[22:50] <balloons> you are indeed!
[22:50] <sak> Great.
[22:51] <balloons> we'll start in about 10 mins. I was mentioning to darran that there's also a video walkthrough of everything I'm uploading now, so very soon it'll be on youtube :-)
[22:51] <darran> where are you based out of balloons?
[22:51] <balloons> it's long but in-depth.. you get to follow me through writing 2 tests
[22:51] <balloons> I'm east coast us
[22:54] <sak> East cost here as well
[22:58] <balloons> I tried to host this at different times for everyone.. this is the latest one of them all :-)
[22:59] <balloons> in UTC time that is
[23:00] <balloons> alrighty, let's get this show on the road :-)
[23:00] <sak> nice
[23:00] <balloons> So welcome to everyone who's here for the testing workshop
[23:01] <Noskcaj> i here, but busy with debian stuff
[23:01] <sak> here
[23:01] <balloons> So the goal here is for me to give a little background introduction, then help anyone with specific questions they may have
[23:01] <sak> ok
[23:02] <balloons> if we get into some details or people are interested, we'll do a livestream or hangout
[23:02] <balloons> I know not everyone wants to be on camera, so we'll stick with IRC for the moment. First I'll give a little introduction on what autopilot is, and what we're doing.
[23:02] <balloons> In a nutsheel, Autopilot itself is a functional testing tool allowing us to interact with an application the same way a user would. it can click, swipe, touch, and type in an application by simulating a user
[23:02] <balloons> So we can use the tool to test and ensure our apps work as expected from a user perspective
[23:03] <balloons> This includes apps like the core apps project, whom we've been focusing on to get tests written for
[23:03] <balloons> The core apps are written by community developers and represent the core applications for the ubuntu touch platform.
[23:03] <balloons> So, in order to help contribute tests there are a couple things you'll need:
[23:03] <balloons> the first is an installation of ubuntu saucy or ubuntu raring. It can be in a VM or installed on physical hardware
[23:04] <balloons> it's important to have raring or saucy for the sdk and autopilot stuff..
[23:04] <balloons> the second is what we're here for.. an understanding of autopilot :-)
[23:04] <balloons> So, first things first
[23:04] <balloons> For anyone who hasn't yet gone through the tutorial on developer.ubuntu.com, http://developer.ubuntu.com/resources/cookbook/mobile/how-to-write-autopilot-tests/, please do so now
[23:05] <balloons> it's a nice intro to autopilot and gives you an example app to play around with
[23:05] <sak> how long do we have to go over the tutorial of the link you gave us?
[23:05] <balloons> sak, if you've not yet gone over it.. do it now :-) I'll field questions as you do so
[23:05] <balloons> For those who have gone through the tutorial, let's talk about what to do next
[23:06] <balloons> The basic steps are:
[23:06] <balloons> 1) pick an app and test you want to write
[23:06] <balloons> 2) contribute it
[23:06] <balloons> 3) Profit! (there is no step 3 :-) )
[23:06] <balloons> so how do you know what's needed? there's a handy list found on this page: https://wiki.ubuntu.com/Touch/CoreApps/Testing
[23:06] <knome> isn't the step 3 "repeat steps 1 and 2" ?
[23:07] <balloons> knome, indeed that's the true step 3
[23:07] <darran> Is there someone that should approve out tests?
[23:07] <balloons> aka, "repeat"
[23:07] <darran> I'm curious who developed autopilot
[23:07] <balloons> darran, yes, the development teams and others will review the tests you write
[23:07] <balloons> just as a normal code contribution
[23:07] <balloons> darran, autopilot itself was developed by a group of folks within the unity team a few years ago
[23:08] <balloons> originally they wrote it to test unity. However since then it's morphed into a general testing tool for use by many applications
[23:08] <Noskcaj> Is it not better to do the writing on Ubuntu itself, rather than the various flavors.
[23:08] <Noskcaj> ?
[23:08] <balloons> you can interact with the folks who wrote the tool directly on #ubuntu-autopilot
[23:09] <Noskcaj> It also tests ubiquity
[23:09] <balloons> Noskcaj, I'm running unity, but you don't have to in order to develop tests
[23:09] <balloons> these core apps should run on other flavors.. I don't know why they wouldn't
[23:10] <balloons> Noskcaj, yes your correct. Besides the core apps the quality team here maintains a project of gtk testcases, including tests for automating the installer. DanChapman was instrumental in getting those written ;-)
[23:10] <balloons> the gtk tests can be found here: https://launchpad.net/ubuntu-autopilot-tests
[23:11] <sak> So just to be clear, autopilot is used to test an application from a user's prospective?
[23:11] <balloons> sak, yes that's correct
[23:11] <balloons> as such it's important to use the tool where it makes sense
[23:12] <balloons> test user facing application pieces, not things that are better suited for a unit test.. For example verifying a function handles race conditions, etc
[23:13] <balloons> So, going back to the contributing part for core apps.. There's a handy list of what needs doing.. aka outstanding tests
[23:13] <balloons> the big list is here: https://bugs.launchpad.net/ubuntu-phone-coreapps/+bugs?field.tag=needs-autopilot-test
[23:14] <darran> is this the place to come if we have questions?
[23:14] <balloons> plenty of places to get invovled. So pick an app, pick a test.. Then your ready to get crackin' on solving it!
[23:14] <sak> Can we use any flavor of ubuntu for autopilot tests, or does it have to be the original ubuntu flavor, unity?
[23:14] <balloons> darran, yes you can ask questions here, in #ubuntu-autopilot, #ubuntu-touch and the mailing lists
[23:16] <balloons> autopilot hackers are in -autopilot, folks like us hang out in -quality, and the devs themselves generally are in -touch
[23:17] <balloons> sak, yes you can write autopilot tests using any flavor, and for any gtk or qt/qml application
[23:18] <balloons> so, if you've picked an app, picked some tests to write, what's next?
[23:18] <balloons> the first step is to run through the tests yourself
[23:19] <balloons> fire up the application and go through the test you want to write
[23:19] <balloons> in english (or your native language if you wish :-) ), write down the steps you take to perform the test
[23:20] <balloons> the basic idea here is to note everything you do as an action
[23:20] <balloons> then after you perform an action, note what happens
[23:20] <balloons> in other words, what you expected
[23:20] <balloons> thus your basis for a testcase look something like this
[23:20] <balloons> click on X button
[23:20] <balloons> the screen changes in Y way
[23:20] <balloons> aka
[23:21] <balloons> perform an action
[23:21] <balloons> expect a result
[23:21] <balloons> all of these actions steps can be simulated by autopilot
[23:21] <balloons> you can have autopilot click, type, etc
[23:22] <balloons> then, for the expected results you've written done, you can write assert statements
[23:22] <balloons> these statements will verify what you expected actually happened
[23:22] <balloons> this is the basis of a good testcase
[23:23] <balloons> So, let's quickly look at the developer tutorial for an example.. I like real world examples :-)
[23:23] <balloons> http://developer.ubuntu.com/resources/tutorials/quality/how-to-write-autopilot-tests/
[23:23] <balloons> if you scroll down to the part "Testing the clear button" you can see a function displayed
[23:24] <balloons> this is a very simple test in the exact format we described
[23:24] <balloons> first we perform an action
[23:24] <balloons> self.pointing_device.click_object(clearButton)
[23:24] <balloons> we then issue some asserts to confirm our results
[23:24] <balloons>   #confirm fields have been wiped
[23:24] <balloons>     self.assertThat(fromField.text, Eventually(Equals('0.0')))
[23:24] <balloons>     self.assertThat(toField.text, Eventually(Equals('0.0')))
[23:25] <balloons> Does that make sense? If you understand that basis for the testcase, we can build your knowledge of syntax and tricks from there :-)
[23:25] <balloons> So I'll stop yapping for a moment and see what questions you all may have, and if anyone has something specific they would like help on
[23:27] <WebbyIT> balloons: where can we find all option to do something? E.g: if i want to swype something, how can i do? And where can I find how can i do? :)
[23:28] <balloons> WebbyIT, great question
[23:28] <balloons> http://unity.ubuntu.com/autopilot/ is the documentation for autopilot
[23:29] <balloons> I know everyone loves reading docs, but there's really just a page or two you should look at to get started
[23:29] <balloons> http://unity.ubuntu.com/autopilot/api/input.html
[23:29] <balloons> this covers the input section; so in your example you want to swipe something
[23:30] <balloons> on that page you'll find the drag command which will do exactly what your looking for
[23:31] <balloons> the other handy page is this one: http://unity.ubuntu.com/autopilot/api/introspection.html
[23:31] <balloons> it covers select_single and select_many
[23:31] <balloons> these are the introspection tools you have available in autopilot
[23:32] <balloons> armed with an understanding of those couple things and you can be dangerous and write some tests :-0
[23:32] <balloons> WebbyIT, does that answer your question?
[23:32] <WebbyIT> balloons: yes, perfectly, thanks :)
[23:39] <balloons> sak, darran how goes it? looking through the tutorial on developer.ubuntu.com?
[23:40] <WebbyIT> balloons: I would have another question :) How can we verify if something is on the display? Eg: on calculator-app I've to swipe to delete a calculation. Now, thank you, I have undestand how to swipe, but how can I check if swype works?
[23:41] <balloons> WebbyIT, we don't directly read the screen in autopilot. This is for various reasons, but basically boils down to it's not sustainable to have a test read pixels (they change too much!)
[23:41] <balloons> so instead, you view the application state
[23:42] <balloons> if you swiped to delete a calculation, you can check and make sure it's been deleted by examining the application.. see if it's stored by the app still, and see if the display data still shows the calculation or not
[23:43] <balloons> I trust this makes sense ;-) You can read the values of say a label or textfield and thus confirm it's no longer there
[23:43] <sak> Completing the installation and setup process
[23:44] <balloons> sak, gotcha :-) ping if you encounter any questions.. Youtube says 10 more mins and the video tutorial will be done.. you'll be able to catch it here: http://youtu.be/qD_e_xqlBbg
[23:44] <WebbyIT> balloons: so, I have first to do a calculation with autopilot, check his ID (every calculation has his ID) or something that identify the calculation, swipe away the result and check if his ID is not longer there?
[23:52] <balloons> WebbyIT, yes that sound correct :-)
[23:52] <balloons> so I would:
[23:52] <balloons> perform an operation and finish it so it hits the history
[23:52] <balloons> grab that calc id and verify it's contents
[23:52] <balloons> swipe it
[23:52] <balloons> verify the id is gone / contents are wiped
[23:53] <WebbyIT> balloons: cool! So, tomorrow I'll try to do my first autopilot test on this :)
[23:54] <balloons> WebbyIT, excellent!
[23:55] <balloons> anyone else have questions?
[23:55] <balloons> WebbyIT, you know where to find me as needed.. I think you've got it down
[23:58] <WebbyIT> balloons: yes, sure :) it was an evening very informative, thanks a lot! :)
[23:58] <WebbyIT> balloons: If nothing else I would go to bed, it's very late here in Italy
[23:59] <balloons> WebbyIT, indeed!
[23:59] <balloons> good night!
[23:59] <balloons> and to everyone else, thank you as well for coming
[23:59] <balloons> dinner time here for me, but do keep in touch if you need help!