[09:40] <jamespage> psivaa, around? have a query about the server tests excuting with UTAH
[09:41] <psivaa> jamespage, sure
[09:42] <jamespage> psivaa, so its more a fundamental UTAH question - is the success or failure of a test measured by the return code of the script?
[09:44] <psivaa> jamespage, yes, ideally jcollado should be able to answer more on this
[09:45] <jamespage> psivaa, OK - I was just checking
[09:45] <jcollado> jamespage: Yes, that's the way it works for now
[09:47] <jamespage> jcollado, great - thanks for the confirmation
[09:47] <jamespage> psivaa, FYI I think we really need to separate the server tests to be task centric as they are at the moment
[09:48] <jamespage> but I'm spending some time on the next few weeks on UTAH for server so I can work on that
[09:59] <psivaa> jamespage, ack, we were going to split the tests and the job in jenkins is just an example. and thanks for doing it :)
[10:49] <jamespage> jcollado, if I'm using my own preseeds how much stuff do I get for free from utah?
[10:49] <jamespage> for example will it inject the utah proxy I configured in /etc/utah?
[10:49] <jamespage> and the required packages for testing?
[10:51] <jcollado> jamespage: utah will add the lines required to install the test client, but not any packages required run specific tests
[10:51] <jamespage> jcollado, ack - just trying to figure out how much I need in the preseed
[10:51] <jcollado> jamespage: If you need specific packages for a test suite, it's probably better to have that in the setup scripts for that test suite
[10:52] <jcollado> jamespage: Aside from that, in the latecommand the ssh key for the utah user will be added to the authorized keys in the VM so that you can send/receive files
[10:53] <jcollado> jamespage: One thing I've found looking at your test cases is that you assume that the user name will be ubuntu while the preseed in utah assumes it's utah
[10:53] <jcollado> jamespage: To avoid problems, I think it's better to set utah in the test cases as well
[10:53] <jamespage> jcollado, I could just drop it altogether and let utah add it right?
[10:54] <jcollado> jamespage: With regard to the preseed, yes, that should be fine.
[10:54] <jcollado> jamespage: But you need to take that into account for the test case scripts
[10:55] <jamespage> jcollado, yep - got that
[10:56] <jcollado> jamespage: Looking at the code, I see that the username will be indeed updated, but it doesn't look it will be added if you haven't set one in the preseed.
[11:00] <jamespage> so I just discovered
[11:05] <jamespage> jcollado, no auto proxy add AFAICT either
[11:08] <jcollado> jamespage: What would be the use case for that?
[11:08] <jamespage> jcollado, /etc/utah/uqt-vm-tools.conf allows setting of a proxy for the vm
[11:09] <jamespage> the use case is when I'm trying to test stuff at home with limited internet bandwidth its x1000 faster to get deb's etc from a local squid-deb-proxy cache rather than from the archive everytime
[11:09] <jamespage> at least as soon as the cache is hot anyway
[11:10] <jamespage> its the proxy for apt rather than the proxy generally
[11:10] <jamespage> "d-i mirror/http/proxy string  http://192.168.1.1:8000" for example
[11:11] <jcollado> jamespage: I see. Certainly, when I run tests I've to download the same packages over and over for different VMs
[11:11] <jamespage> for me its the difference between 300K/s and 30MB/s
[11:12] <jamespage> jcollado, its less import when testing the full ISO - but I'm just using the mini ISO ATM
[11:14] <jcollado> jamespage: That would be interesting to have. I've written it down, so that I remember to discuss this in the utah meeting we've got this afternoon.
[11:15] <jamespage> jcollado, please - its an essential feature for me - I can workaround in my test preseed for the moment but its not portable
[11:21] <jcollado> jamespage: Opened: https://bugs.launchpad.net/utah/+bug/1043249
[11:21] <ubot5> Ubuntu bug 1043249 in UTAH "apt proxy added automatically to the preseed" [Undecided,New]
[11:21] <jcollado> jamespage: Please review and comment if something is missing
[11:22] <jamespage> jcollado, LGMT
[11:22] <jamespage> LGTM even
[11:22] <jamespage> :-)
[11:29] <jamespage> jcollado, is it possible to use utah to execute a runlist locally so I don't have to provision a VM to test it?
[11:30] <jcollado> jamespage: Yes, you can use `utah -r <runlist>`
[11:37] <jamespage> jcollado, hmm - I get this error with the example one in utah
[11:37] <jamespage> http://paste.ubuntu.com/1173672/
[11:38] <gema> jcollado, jamespage afaik nuclearbob was working on the proxy stuff, I'd check with him
[11:39] <gema> I will ask him on today's catch up
[11:39] <gema> jamespage: if you are going to be working in utah for the next couple of weeks, you may want to come to our weekly meetings for a while
[11:39] <gema> jamespage: we have a hangout on wed afternoons
[11:40] <jamespage> gema, what time?
[11:40] <gema> 3pm
[11:41] <jamespage> gema, OK - I'll try to make it today - I have a 14:30 which should be done in time hopefully
[11:41] <gema> jamespage: ack
[11:42] <gema> jamespage: there is no problem in moving it half an hour later as far as I can tell, so if you find it useful today we could try that
[11:42] <jamespage> OK - lets see
[11:42] <jcollado> jamespage: I've seen validation problems in the past, but right now I cannot reproduce it. Which package version are you using?
[11:42] <jamespage> jcollado, 0.4ubuntu57-r661~quantal1
[11:46] <jcollado> jamespage: Are you using the example runlist or another one?
[11:46] <jamespage> jcollado, I get it with the example and my own
[11:47] <sagaci> the desktop interface looks a lot better in virtualbox since the removal of unity-2d but performance is excruciating
[11:49] <jcollado> jamespage: That's strange. I don't get any validation error with this one: http://pastebin.ubuntu.com/1173679/ (which should be the same example one you have)
[11:50] <jamespage> jcollado, it is - I'm using quantal rather than precise - could that make a difference?
[11:51] <jcollado> jamespage: Oh, yes, now I realize
[11:52] <jcollado> jamespage: The thing is that we use an newer version of python-jsonschema that is available through the PPA: https://launchpad.net/~utah/+archive/stable
[11:52] <jcollado> jamespage: Probably you have version 0.2 when you should have 0.5
[11:52] <jamespage> jcollado, I am using 0.2 yes
[11:53] <jamespage> jcollado, OK - the docs probably need updating then
[11:54] <jamespage> jcollado, how does that get into the VM being tested ten?
[11:54] <jamespage> then
[11:55] <jcollado> jamespage: The PPA is enabled in the preseed
[12:12] <jamespage> jcollado, it tried to by add-apt-repository was not installed...
[12:15] <jcollado> jamespage: The add-apt-repository call failed or after adding the PPA to your sources you still don't get python-jsonschema 0.5?
[12:19] <jamespage> jcollado, I patch it all up after provisioning and then it works OK
[12:20] <jamespage> jcollado, "add-apt-repository -y ppa:utah/stable" - command not found
[12:20] <jamespage> I've added the required packages to my preseed to fixup
=== _salem is now known as salem_
=== sagaci_ is now known as sagaci
[14:27] <smartboyhw> Hi, is there any people who develop checkbox?
[14:28] <sagaci> looks like there is
[14:29] <smartboyhw> Hi, is there any checkbox dev here?
[14:35] <roadmr> smartboyhw: sure, ask away, we may be a bit slow, I for one am on a phone conference
[14:35] <smartboyhw> Er, I don't know why, the audio tests passed, but then still it shows a popup showing the tests had failed and I need to report a bug. Thanks roadmr
[14:37] <roadmr> smartboyhw: I suggest you say "no" to the bug report request and then run the rest of the tests, the final report will tell you why it thinks the audio test failed
[14:37] <smartboyhw> I know.
[14:37] <smartboyhw> where can I find the report then?
[14:37] <roadmr> smartboyhw: if you like you could put the submission.xml somewhere for me to look at and figure out what happened
[14:38] <roadmr> smartboyhw: in .cache/checkbox/submission.xml, you can open it with firefox
[14:38] <smartboyhw> OK
[14:38] <roadmr> smartboyhw: note it only gets generated when the test run ends
[14:38] <smartboyhw> OK
[14:39] <smartboyhw> OK, where to put the submission.xml?
[14:39] <roadmr> smartboyhw: you could pastebin it
[14:40] <smartboyhw> http://paste.ubuntu.com/1173908/
[14:40] <roadmr> smartboyhw: awesome thanks
[14:41] <smartboyhw> )
[14:41] <smartboyhw> :)
[14:59] <roadmr> smartboyhw: strange because this submission doesn't show any failed tests :/
[15:00] <smartboyhw> :)
[15:01] <roadmr> smartboyhw: so the window asking you to report a bug, it had a test name on it, which was it?
[15:01] <smartboyhw> All the audio ones
[15:04] <roadmr> oh ok.. let's see
[15:04] <roadmr> smartboyhw: which version of checkbox are you running?
[15:06] <jamespage> jcollado, gema, bug 1043382
[15:06] <ubot5> Launchpad bug 1043382 in UTAH "Reporting of test failures" [Undecided,New] https://launchpad.net/bugs/1043382
[15:06] <jcollado> jamespage: Thanks
=== yofel_ is now known as yofel
[15:56] <jamespage> jcollado, erm - stupid question but can I make UTAH run my tests as root?
[15:57] <jcollado> jamespage: There's a "run_as" field in the test suite runlist for that.
[15:57] <jcollado> jamespage: http://utah.readthedocs.org/en/latest/development.html#runlists-hierarchy
[15:59] <jamespage> jcollado, and that can escalate to root? I'll try it
[15:59] <jcollado> josepht: ^
[16:06] <josepht> jamespage: by default they run as 'root', 'run_as' will allow dropping privileges for testcases that require it.
[16:12] <jamespage> josepht, OK - lemme give it a swing - was seeing some odd issues....
[16:48] <jamespage> josepht, gema: OK - so I rejigged all of the server test cases and hacked out the preseed templating code
[16:48] <jamespage> https://code.launchpad.net/~james-page/+junk/server-tests
[16:49] <jamespage> still struggling a bit to get them to execute through UTAH tho
[19:33] <xnox> I see a surge of bugs against ubiquity, when really they are bugs in live-desktop, i.e. bugs in other packages.
[19:33] <xnox> Does the iso tracker encourage filing bugs against ubiquity for some reason?
[19:33] <balloons> xnox, yes..
[19:33] <xnox> Or is it just the random sample of latest ubiquity bugs which just happen to be anything but ubiquity?
[19:33] <xnox> balloons: that's bad.
[19:33] <balloons> in general, bugs encountered during installation are filed against ubiquity
[19:34] <xnox> bug 1043488
[19:34] <ubot5> Launchpad bug 1043488 in ubiquity (Ubuntu) "Scroll with a scroll wheel doesn't work on a web page" [Undecided,New] https://launchpad.net/bugs/1043488
[19:34] <xnox> bug 1043403 was also filed against ubiquity
[19:34] <ubot5> Launchpad bug 1043403 in linux (Ubuntu) "Wireless not active during installation or after boot" [Undecided,Confirmed] https://launchpad.net/bugs/1043403
[19:35] <xnox> (it also, in the very first description) said "even after I boot into installed system there is no wifi"
[19:35] <xnox> balloons: I do not scale to reassign them to different packages =/
[19:35] <balloons> xnox, heh, no I'm sure you don't
[19:36] <xnox> I am ok with ubiquity crashing & filing bug against ubiquity, even if it is a particular package install problem. But making ubiquity yet another bucket "ubuntu (no package)" is counter productive =/
[19:36] <balloons> hmm.. no I don't believe that's the intent here
[19:36] <balloons> part of the issue  I think is stemming from the post-installation testing rolled into the iso-testing
[19:36] <xnox> balloons: so can you show me where the iso tracker / test cases encourage to file bugs specifically against "ubiquity"
[19:36] <xnox> ?
[19:37]  * xnox doesn't use iso tracker / test cases that often. Or am I blind and just don't see such encourangment.
[19:37] <xnox> post-installation hmmm....
[19:37] <xnox> to be honest anything can be install or a package issue.
[19:37] <balloons> meaning since we're now rebooting into the installation and running tests, those bugs should be reported against the package in question
[19:37] <balloons> http://iso.qa.ubuntu.com/qatracker/milestones/219/builds/22057/buginstructions
[19:38] <balloons> we can change what the instruction are obviously :-)
[19:38] <balloons> but what I meant was there is no further information given about reporting a bug that didn't occurr in the installr
[19:41] <xnox> balloons: I would say for "Graphics problems file against: xorg";  "For all other hardware file against: linux"; "For installer bug, crashes, or installation failure: ubiquity"; "For particular applications during live session or after the install: against that application"; "If not sure: ubuntu (no package)"
[19:41] <xnox> "Fail to boot after install: grub2"
[19:41] <xnox> or uboot on arm.
[19:41] <balloons> ok.. we can customize the info by arch, so
[19:42] <xnox> awesome! =)
[19:42] <xnox> Mention Xorg only on Desktopy Cds.
[19:42] <xnox> And ubiquity on the desktopy CDs; but debian-installer on others
[19:43] <xnox> balloons: does iso tracker/testcases have a launchpad bugtracker?! I think I should just file a bug about it.
[19:43] <xnox> ?
[19:43] <balloons> xnox, indeed
[19:43] <balloons> check the bottom of the page
[19:44] <balloons> debian installer / ubiquity is split properly atm
[19:53] <xnox> balloons: bug 1043503
[19:53] <ubot5> Launchpad bug 1043503 in Ubuntu QA Website "bugreporting instructions need improvement" [Undecided,New] https://launchpad.net/bugs/1043503
[19:54] <balloons> xnox, I'll have a look and comment on it
[19:54] <balloons> this is something we can change on the fly, so assuming we all agree, we'll update it
[20:01] <balloons> xnox, so bugs filed against ubuntu are triaged, or intended to be triaged?
[20:01] <xnox> for some reason I do get bug mail about "Ubuntu (no package)" And I do notice that many/different/random people assign them to correct packages
[20:01] <xnox> very quickly
[20:02] <xnox> so somebody is taking care of those =)
[20:02] <balloons> xnox, alright
[20:02] <xnox> I have no clue who though =)
[20:02] <xnox> bugsquad? brian murray's army of bots?
[20:02] <balloons> I'm assuming the bugsquad :-)
[20:02] <balloons> the bot army!
[20:02] <balloons> I love it
[20:05] <xnox> "(we don't always watch that area, though, 24/7)" --- reply on #ubuntu-bugs
[20:07] <balloons> xnox, ok, I will try and make the suggested changes this week and mention it to everyone to try and file bugs accordingly. Often they don't know where to file, but if ubutu (no package) is being triaged, then that's the place to go.. The trouble is I believe historically all isotesting bugs were triaged by jibel et la on QA
[20:07] <balloons> so having them land in a couple places probably worked out well for that
[20:08] <xnox> balloons: redirecting all to one place, does not work.
[20:08] <xnox> balloons: either keep it, as it is. Or stream it into multiple buckets.
[20:08] <balloons> xnox, what do you mean?
[20:08] <xnox> balloons: to be honest "linux" and "xorg" are big buckets as it is. But by landing more-or-less correct bugs gets more people to look at them quicker.
[20:08] <balloons> there will be cases of peopl egetting it wrong.. but even more so for the random post-install bugs, people simply won't know where to assign it
[20:09] <xnox> balloons: yeah, so ubuntu (no package) should be a last fallback.
[20:09] <xnox> but not "the only encouraged location to file bugs"
[20:10] <balloons> should we encourage folks to file against the application they are having issues in, even if it turns out to be something like a compiz bug?
[20:10] <xnox> yes.
[20:10] <balloons> is that better or worse than filing against ubuntu?
[20:10] <xnox> better - cause filing in multiple locations, triggers more/different people to "sheppard/triage" bugs
[20:11] <xnox> and compiz is a corner case - by the time a bug becomes a compiz bug it usually already has been in linux/xorg packages =)
[20:11] <xnox> or even unity
[20:11] <balloons> it's just important that we note that in the bug reporting instructions
[20:12] <balloons> we don't want a novel, but :-)
[20:12] <xnox> balloons: if somebody is using ISO testing website I think they are capable to separate: installer (ubiquity), hardware support (linux), or other bugs =)
[20:12] <xnox> balloons: remember they managed to download & burn an ISO =))))))))))
[20:13] <balloons> hehe
[20:52] <jibel> balloons, maybe we should add some requirements before people can join ubuntu-qa/ubuntu-testing ? like read and understood the bugsquad and bug reporting documentation
[20:53] <balloons> jibel, asking them to read things in advance and understand isn't a bad idea.. but if we do it, we need to help get them knowledgable
[20:53] <balloons> in general I don't think it's too bad
[20:53] <balloons> the bugs xnox mentioned came from someone I was helping over email actually..
[20:53] <balloons> they are new.. clearly saying 'file a bug' isn't enough detail
[20:54] <xnox> balloons: is that so!
[20:54]  * xnox phht! it's all balloons fault =)))))
[20:54] <xnox> *kidding*
[20:55] <jibel> balloons, people doing testing *will* file bugs, IMO reading documents like https://help.ubuntu.com/community/ReportingBugs/ or https://wiki.ubuntu.com/Bugs/BestPractices should be mandatory
[20:55] <balloons> xnox, I can affirm one of them anyway
[20:55] <balloons> :-)
[20:56] <balloons> jibel, yep.. but further we should make sure they are filing good reports
[20:56] <balloons> both in where they land, and what they say
[20:58] <jibel> testers make the effort to download iso, install them and reporting bugs. We can expect that is there are requirements like reading some documentation to join the team they will follow the guidelines
[20:58] <jibel> s/is there/if there
[20:58] <jibel> then filing good reports comes with experience and time.
[21:11] <leadsled> ISO tracker question why has the Xubuntu 12.10 daily iso been updated since 8-22-2012?
[21:11] <balloons> leadsled, gtk2 indicator errors
[21:13] <leadsled> how long will it take for this to be resolved?
[21:15] <balloons> leadsled, unknown to me.., however, I expect it to be fixed this week.. the images are in a state of flux do to feature freeze occuring
[21:15] <balloons> next week is beta 1 testing.. we will need an iamge ;-)
[21:21] <leadsled> Why has the decision been made to end Ubuntu 12.10 alternate when there is only a month and half until the final release?
[21:22] <balloons> leadsled, the idea was discussed at the last UDS, as a target for this cycle. Now, it was agreed the alternate images could be considered to be dropped if ubiquity was able to meet all the features of the alternate installer
[21:22] <balloons> Now, as you read in the email, feedback is being solicitied for dropping the alternate, even though all features are not yet contained in ubuiquity
=== salem_ is now known as _salem
[21:32] <leadsled> should the daily isos at this point have beta instead alpha on the readme.diskdefines?
[21:43] <balloons> leadsled, now.. there not yet beta :-)
[21:43] <balloons> *no
[21:47] <leadsled> so beta starts tomorrow
=== Ursinha` is now known as Ursinha