| === sfeole` is now known as sfeole | ||
| Mirv | cihelp intel AP machine having continued problems http://q-jenkins.ubuntu-ci:8080/job/autopilot-trusty-daily_release/label=qa-intel-4000/948/console | 06:57 |
|---|---|---|
| sil2100 | Morning! | 07:48 |
| Mirv | hi sil | 07:58 |
| Mirv | I reported to ci_help the intel AP machine issue | 07:58 |
| vila | Mirv: there is kernel crash in that console file that I think you should report upstream, it is a genuine test failure | 08:00 |
| vila | Mirv: the fact that it brings down a host in the ci infra *is* a ci bug we are painfully aware of but that shouldn't block reproting the test failures upstream | 08:01 |
| vila | Mirv: note that I did revert to aprevious kernel on Friday hoping to survive a bit longer to this issue... obviously that wasn't enough :-/ | 08:02 |
| vila | Mirv: just checked, qa-intel-4000 is using the previous kernel :-/ (Just checked that this change wasn't reverted by mistake) | 08:03 |
| sil2100 | Mirv, vila: I remember times when we also been blocked by intel kernel crashes... | 08:05 |
| vila | sil2100: yes, that the pain I'm referring to | 08:05 |
| sil2100 | So, actually a failing test causes it you say? | 08:05 |
| Mirv | vila: what was again the specs of that machine? | 08:05 |
| vila | sil2100: yes, look at http://q-jenkins.ubuntu-ci:8080/job/autopilot-trusty-daily_release/ *some* tests don't trigger it | 08:05 |
| Mirv | vila: ie. which intel CPU/GPU? | 08:05 |
| vila | Mirv: qa-intel-4000 so I think that indeed refers to the intel GPU integrated graphic chipset 4000 | 08:06 |
| vila | sil2100, Mirv: I'm torn between 2 solutions | 08:06 |
| vila | sil2100, Mirv: 1) you abort the jobs that crash qa-intel-4000 hoping that it's rare enough to still gather successful runs | 08:07 |
| Mirv | vila: and the kernel versions that have been now in use showing the problem both? (I'm filing the bug report) | 08:07 |
| vila | sil2100, Mirv : 2) we de-provision qa-intel-4000 keeping only autopilot-nvidia | 08:08 |
| vila | Mirv: $ uname -a | 08:08 |
| vila | Linux qa-intel-4000 3.12.0-5-generic #13-Ubuntu SMP Mon Dec 2 18:19:58 UTC 2013 i686 i686 i686 GNU/Linux | 08:08 |
| vila | is the current (reverted) one | 08:08 |
| Mirv | thanks | 08:09 |
| Mirv | only nvidia seems a bit too low, but I fear 1) doesn't really work well either. | 08:09 |
| vila | Mirv: Linux 3.12.0-7-generic did crash in the ~same way on Friday | 08:10 |
| vila | Mirv: there is the radeon one too but I can't remember the details about why it wasn't re-provisioned ? | 08:10 |
| vila | Mirv: and thanks for filing the bug ! | 08:10 |
| vila | Mirv: wasn't there a bug for the radeon ? (Damn memory !!) | 08:11 |
| Mirv | vila: the only example of intel crashing on Friday I see is this http://q-jenkins.ubuntu-ci:8080/job/autopilot-trusty-daily_release/940/label=qa-intel-4000/consoleText - but I don't see any kernel crash trace there either? | 08:12 |
| Mirv | vila: yes there's a bug for radeon https://bugs.launchpad.net/ubuntu/+source/glamor-egl/+bug/1253974 | 08:12 |
| vila | Mirv: also https://wiki.canonical.com/UbuntuEngineering/CI/Playbook/Otto says: 'autopilot-ati doesn't seem to be used but is morally owned by cu2d ' so may be we can use it ? | 08:12 |
| ubot5 | Ubuntu bug 1253974 in glamor-egl (Ubuntu) "glamor-egl 0.5.1-0ubuntu6 crashes when running autopilot tests" [Undecided,New] | 08:12 |
| vila | Mirv: ha great ! Thanks | 08:12 |
| vila | Mirv: that one didn't get a lot of attention apparently ? | 08:13 |
| vila | :-/ | 08:13 |
| vila | Mirv: nope, #940 crash is caused by the reboot after updating the host, "expected" failure :-/ Let me find the one I'm thinking about | 08:14 |
| Mirv | found, http://q-jenkins.ubuntu-ci:8080/job/autopilot-trusty-daily_release/936/label=qa-intel-4000/consoleText | 08:15 |
| vila | Mirv: https://wiki.canonical.com/UbuntuEngineering/CI/IncidentLog/2013-12-13-qa-intel-4000-kernel-crash | 08:15 |
| vila | Mirv: yup, that one | 08:16 |
| Mirv | filed bug #1261308 against kernel now | 08:17 |
| ubot5 | bug 1261308 in linux (Ubuntu) "Intel ivy bridge crash on a CI machine executing autopilot tests" [Undecided,New] https://launchpad.net/bugs/1261308 | 08:17 |
| Mirv | vila: https://bugs.launchpad.net/ubuntu/+bugs?field.tag=ci | 08:18 |
| Mirv | anyhow, an older radeon machine for running AP test could be a nice option | 08:19 |
| Mirv | one that doesn't require glamor | 08:19 |
| sil2100 | vila, Mirv: as I still didn't have time to look myself, how frequently these intel crashes happen? | 08:19 |
| vila | sil2100: you mean in general or for that particular one ? | 08:20 |
| sil2100 | vila: this one I guess, like... how frequently we would see intel dying if we left it like that | 08:20 |
| vila | sil2100: well, I think the one Mirv caught is the first blocking the line since Friday no ? | 08:21 |
| Mirv | we didn't have much check jobs running before the fixes last week, so there is not that much data | 08:22 |
| vila | http://q-jenkins.ubuntu-ci:8080/computer/qa-intel-4000/builds | 08:22 |
| vila | indeed, not much | 08:23 |
| vila | :-/ | 08:23 |
| vila | but still some success since #940, even #944 and #947 are non-crashing (so valid) results | 08:25 |
| sil2100 | For now let's not de-provision intel, let's keep a lookout for crashes and try to get as much results as possible, aborting when needed | 08:26 |
| vila | Mirv, sil2100: It's paradoxical that the crashes are useful to diagnose the kernel issue while also being a pain for by invalidating whole runs :-/ | 08:26 |
| sil2100 | That's what I would think | 08:26 |
| sil2100 | If it gets too troublesome, we can de-provision then, and stay only with nvidia for a while | 08:27 |
| vila | sil2100: indeed, I think that's the way to go, if that becomes too painful, de-provisioning can still be done | 08:27 |
| vila | hehe | 08:27 |
| vila | sorry, crossed on the wire ;) | 08:27 |
| vila | sil2100: ^ | 08:27 |
| sil2100 | ;) | 08:28 |
| vila | sil2100, Mirv : at least the hack I added to shut down the container whatever happens during a job properly stopped the container so further jobs can be attempted | 08:29 |
| vila | sil2100, Mirv: if it doesn't, you'll need cihelp to reboot qa-intel-4000 | 08:31 |
| vila | sil2100, Mirv : I've seen at least one such case on Friday as mentioned in the incident log | 08:31 |
| vila | Mirv: How can we know if we have an "older radeon machine for running AP test that doesn't require glamor" ? | 08:33 |
| sil2100 | vila: ok, I remember in the past we were able to reboot the machines electrically ourselves by using some UI interface - is that no longer the case? | 08:35 |
| vila | sil2100: ha right, well, my understanding is that we now want to implement a policy where you shouldn't have to do that but the cihelp should do it for you :-/ retoaded sent an email I received this morning about contacting him for exceptions, see with ev, I think this could b arranged at least for the time being | 08:37 |
| vila | sil2100, Mirv : The email title is "Upcoming changes to the CI Lab" | 08:38 |
| Mirv | yes no access to those anymore | 08:38 |
| Mirv | I read it in the morning | 08:39 |
| sil2100 | vila: I guess it would be nice if we could have some temporary credentials to reboot the machines until intel is not 100% reliable again, but I suppose we can also use cihelp | 08:45 |
| vila | sil2100: yup, especially for Mirv that is otherwise blocked until someone answer cihelp early in the morning | 09:05 |
| sil2100 | vila: is CI covering every timezone, or are there some spots when no-one is around? | 09:07 |
| === ev changed the topic of #ubuntu-ci-eng to: Ubuntu CI Engineering Team | Vanguard: ev | Landing instructions: http://paste.ubuntu.com/6292280/ | Known issues: - | ||
| ev | sil2100: there are some spots where no one is around (from about 9-10pm UTC onwards) | 09:08 |
| vila | sil2100: I think we are closer to 16/5 than to 24/7 for the foreseeable future | 09:08 |
| vila | as ev said ;) | 09:08 |
| Mirv | vila: ev: not really sure if it should be different, but it seems that the stopping of container when aborting job does not work: http://q-jenkins.ubuntu-ci:8080/job/autopilot-trusty-daily_release/label=qa-intel-4000/951/console | 09:34 |
| Mirv | so abort has been clicked once for that since the kernel crashed again. second click would abort that aborting but I thought if you'd like to take a look. | 09:34 |
| vila | Mirv: yup, I'm working on switching to Little Boy to Fat Man as a way to nuke that container | 09:34 |
| vila | *from Lillte Boy | 09:35 |
| vila | bah | 09:35 |
| vila | you get my meaning | 09:35 |
| Mirv | yes :) | 09:35 |
| ev | sil2100: for what it's worth, this is to make the CI engineering team more responsible for the software they're operating. When actions need to be taken on the machines, when things go wrong, and so on, the finger is only pointed at one group: us. | 09:38 |
| popey | sil2100: 68 is good for me on mako | 09:43 |
| asac | sil2100: so i also sent a mail now to qa ML | 09:45 |
| asac | sil2100: lets see... if nothing happens i will send a text to jfunk and davmor, but only later as i dont think its urgent enough to text them during their night :) | 09:46 |
| asac | (e.g. we have the image in the bank kind of and can test it at our leisure) | 09:46 |
| sil2100 | asac: thanks :) | 09:54 |
| vila | sil2100, Mirv: So, http://q-jenkins.ubuntu-ci:8080/job/autopilot-trusty-daily_release/ will now reboot the otto node if a container is still running, no need to ping cihelp to do it manually | 10:06 |
| vila | sil2100, Mirv: keep an eye on how those jobs end, they will probably have a log showing the lost connection from jenkins caused by the reboot, the real error (leading to the container left running) should be earlier in the same log | 10:07 |
| vila | sil2100, Mirv: that's far from ideal and I'll keep investigating for a better handling but at least for the current qa-intel-4000 issue this should give us some... non-interrupted service | 10:08 |
| Mirv | vila: great!! let's see. | 10:09 |
| sil2100 | vila: thanks! That's a good workaround I guess, thanks for taking care of it :) | 10:10 |
| vila | sil2100, Mirv : thanks to you for tracking the bugs identified there, that's also an important part of the larger picture ! | 10:11 |
| === vrruiz_ is now known as rvr | ||
| === josepht changed the topic of #ubuntu-ci-eng to: Ubuntu CI Engineering Team | Vanguard: josepht | Landing instructions: http://paste.ubuntu.com/6292280/ | Known issues: - | ||
| === dednick is now known as dednick|lunch | ||
| asac | sil2100: so i found out | 13:54 |
| asac | its omer | 13:54 |
| asac | so lets get him | 13:54 |
| asac | sil2100: sent him a mail | 13:55 |
| sil2100 | asac: awesome, thanks ;) | 13:56 |
| asac | sil2100: he is now on | 13:57 |
| asac | sil2100: will test | 13:57 |
| asac | tell him what :) | 13:57 |
| asac | sil2100: will we do a checkpoint image? | 14:04 |
| asac | sometimes during afdternoon? | 14:04 |
| asac | to keep capturing potential regressions coming in through archive upload that is | 14:04 |
| asac | sil2100: do you know where to build the image? | 14:04 |
| asac | thats a button in the iso tracker now | 14:04 |
| asac | maybe check | 14:04 |
| cwayne | plars, hi, any update on the new mako running custom? | 14:06 |
| sil2100 | asac: I don't have the power to start builds so I don't remember where the building bits are now | 14:06 |
| sil2100 | I tried finding it now but hm, can't remember | 14:07 |
| sil2100 | asac: we could do an image soon - not much new things coming in in overall, but I guess nothing bad will happen if we spin one | 14:07 |
| sil2100 | Mirv: hey, you still around? | 14:08 |
| sil2100 | Mirv: I just noticed that the settings check job was never ran - I wonder if I missed anything when adding the check bits there? | 14:09 |
| asac | sil2100: right. idea is to spin to get a checkpoint ... so good if nothing happens :) | 14:09 |
| asac | thanks | 14:09 |
| asac | just schedule it at your convenience | 14:09 |
| === josepht_ is now known as josepht | ||
| === dednick|lunch is now known as dednick | ||
| lool | sil2100: do you want me to start one? | 14:41 |
| lool | sil2100: (image build) | 14:42 |
| lool | sil2100: generally it's a good idea to build a couple per day even if we end up not promoting them | 14:42 |
| asac | lool: hi... thanks for helping out. yes. we have a button in isotracker now | 14:42 |
| lool | just to have something to compare against | 14:42 |
| asac | so you might want to try that feature | 14:42 |
| ogra_ | lool++ | 14:42 |
| lool | sil2100: you have to tick the checkbox near the image in the ISO tracker to request a build | 14:42 |
| asac | lool: right. lets do one and lets discuss cut off times tomorrow in meeting | 14:42 |
| asac | lool: you didnt show up today :)? | 14:42 |
| lool | asac: No :-( I was at work but had forgotten to add the invite to my agenda | 14:43 |
| lool | asac: I added it for tomorrow | 14:43 |
| ogra_ | lool, only members of ubuntu-touch-release can trigger builds currentlyy | 14:43 |
| asac | lool: want me to inveit you? | 14:43 |
| lool | asac: I did it from the UE calendar already | 14:43 |
| asac | kk | 14:43 |
| lool | ogra_: aha | 14:43 |
| lool | ogra_: and sil2100 isn't one? | 14:43 |
| ogra_ | (i dont think sil2100 is in that team since ubuntu-core-dev membership is a req. i was told) | 14:43 |
| lool | that sounds familiar | 14:44 |
| rsalveti | I'm landing libhybris today as well, and want to trigger a new image after it's in | 14:45 |
| rsalveti | it's already in proposed, waiting to be migrated | 14:46 |
| lool | sil2100, ogra_: So just in case you two haven't used it and might use it in the future, you go to Trusty Daily milestone on iso.qa.ubuntu.com which is at http://iso.qa.ubuntu.com/qatracker/milestones/308/builds then you tick the checkbox next to Ubuntu Touch armhf then at the bottom you hit "Update rebuild status" next to "Request a rebuild" | 14:47 |
| ogra_ | ok | 14:47 |
| * ogra_ tends to just log in to nusakan :P | 14:48 | |
| ogra_ | but i know didier used ot for the last week and it worked fine | 14:48 |
| lool | ogra_: yeah, Didier had to (no access to nusakan) but it's a good idea that we all use the same path IMO, so I'll try to use web UI | 14:57 |
| lool | problem for Didier is that he couldn't tell whether any build was in progress | 14:57 |
| ogra_ | well, i dont think there is an easy way apart from checking the livefs builder directly | 14:59 |
| ogra_ | (or the processlist on nusakan for the system-image processing) | 14:59 |
| cjwatson | I'm pretty sure nusakan sends something to iso.qa when builds are running | 14:59 |
| cjwatson | there should be some hint there | 14:59 |
| ogra_ | ah | 15:00 |
| cjwatson | like a build being struckthrough or similar | 15:00 |
| * ogra_ honestly never used the UI before :) ) | 15:00 | |
| lool | Yeah I didn't try it myself yet, just mentioning what I believe Didier said here | 15:04 |
| lool | rsalveti: android-platform-headers | 0.1.0+git20130606+c5d897a-0ubuntu37 | trusty | all | 15:05 |
| lool | rsalveti: out of date on powerpc: android-platform-headers (from 0.1.0+git20130606+c5d897a-0ubuntu37) | 15:05 |
| lool | and a couple of other arches | 15:05 |
| lool | rsalveti: We might have to force libhybris in | 15:05 |
| lool | cjwatson: ^ | 15:05 |
| lool | moving this to #ubuntu-release | 15:06 |
| lool | hmm actually no, it seems to have gone in | 15:06 |
| lool | robru: were you handling account-plugin-evernote? | 15:12 |
| lool | robru: there's a note to NOT put it into the image | 15:12 |
| lool | sil2100: mind poking when you come around? | 15:20 |
| sil2100 | lool: hi! I'm in the middle of preparing lunch, let me get back to you in a moment | 15:21 |
| lool | sure | 15:21 |
| lool | The ALL CAPS in the spreadsheet are a bit annoying; it's not clear who wrote them | 15:21 |
| lool | also it SOUNDS LIKE SHOUTING | 15:22 |
| ogra_ | SOMEONE WITH A BROKEN CAPS KEY I SUPPOSE | 15:23 |
| ogra_ | :) | 15:23 |
| lool | ogra_: you mean :_) | 15:30 |
| ogra_ | heh | 15:31 |
| lool | ah too bad CAPS + - doesnt deliver a _ | 15:31 |
| lool | what's the Projeted Landing Slot column for? | 15:31 |
| lool | it's empty | 15:31 |
| sil2100 | lool: ok, I have a moment now while food is preparing | 15:40 |
| sil2100 | lool: let me backlog | 15:42 |
| sil2100 | lool: ok, so if you could, could you kick a new image? I don't have the creds to do that, I would have to wait for cyphermox or kenvandine to appear for that | 15:42 |
| sil2100 | lool: as for libhybris, I have been waiting for rsalveti to get an update about that | 15:43 |
| rsalveti | sil2100: it's already in proposed, waiting to be migrated | 15:44 |
| sil2100 | lool: and for robru to get some info on account-plugin-evernote - but this one wasn't probably preNEWed yet, so it's no problem | 15:44 |
| popey | sil2100: please put account-plugin-evernote on hold | 15:46 |
| popey | (I put a CAPITAL LETTERS) note in the landing asks sheet about that | 15:46 |
| popey | we need to figure out some details first | 15:46 |
| sil2100 | popey: noted it down as blocked in status, and will poke robru once he's up | 15:46 |
| sil2100 | Ok | 15:46 |
| popey | ta | 15:46 |
| lool | sil2100: ok | 15:53 |
| cyphermox | sil2100: already around btw... | 15:53 |
| lool | sil2100: Yes I can kick images whenever you like | 15:53 |
| lool | sil2100: libhybris will appear in archive in some time, it was stuck but we've unblocked it | 15:54 |
| sil2100 | lool: awesome, so maybe we'll kick a build once it's in the archive? | 15:57 |
| sil2100 | cyphermox: \o/ | 15:57 |
| === Saviq_ is now known as Saviq | ||
| lool | sil2100: Ok | 15:58 |
| === ev_ is now known as ev | ||
| lool | so it seems to be in | 16:10 |
| lool | kicking a build | 16:10 |
| lool | right now this says "Ubuntu Touch armhf (re-building)" | 16:11 |
| lool | I see it got started now (at :15) | 16:15 |
| lool | cron is */5 | 16:15 |
| lool | I dont see any difference on the web UI between time of request and time of start and the cdimage code doens't seem to send any request to ISO tracker | 16:16 |
| lool | it just does a tracker.qatracker.get_rebuilds("Requested") AFAICT | 16:17 |
| lool | the other use is when a build is finished, the image gets posted to the tracker | 16:17 |
| asac | did we build an image? | 16:17 |
| lool | asac: it's building | 16:17 |
| asac | why did we wait another 3 hours ? | 16:17 |
| asac | :) | 16:17 |
| === balloons_ is now known as balloons | ||
| asac | guess it was more 2h :) | 16:18 |
| asac | after we decided to build an image | 16:18 |
| asac | heh | 16:18 |
| === gatox is now known as gatox_lunch | ||
| lool | hehe | 16:18 |
| lool | +1 on building more images | 16:18 |
| lool | yeah; and at least both persons with an opinion on what to include in it said libhybris and we got it in, so it's ok | 16:19 |
| lool | but tomorrow we should build one earlier in the day | 16:19 |
| lool | and one EOD | 16:19 |
| sil2100 | That's a good plan | 16:19 |
| lool | Sounds like the A-team | 16:20 |
| === Saviq_ is now known as Saviq | ||
| rsalveti | sil2100: lool: we can trigger a new image already | 16:24 |
| asac | lool: what landed ? | 16:25 |
| asac | :) | 16:25 |
| asac | err | 16:25 |
| asac | rsalveti: ^ | 16:26 |
| lool | rsalveti: it's already under way | 16:26 |
| rsalveti | hybris | 16:26 |
| lool | 17:10 < lool> so it seems to be in | 16:26 |
| lool | 17:10 < lool> kicking a build | 16:26 |
| asac | ok that was about hybris | 16:26 |
| asac | good | 16:26 |
| sergiusens | lool, http://www.thetriathloncoach.com/wp-content/uploads/2012/08/I-love-it-when-a-plan-comes-together.jpeg | 16:27 |
| === balloons_ is now known as balloons | ||
| lool | sergiusens: exactly what I had in mind :-) | 16:33 |
| === josepht changed the topic of #ubuntu-ci-eng to: Ubuntu CI Engineering Team | Vanguard: cihelp | Landing instructions: http://paste.ubuntu.com/6292280/ | Known issues: - | ||
| sil2100 | robru: meeting! | 17:01 |
| === ken_ is now known as kenvandine | ||
| lool | images are being computed | 17:04 |
| Laney | splines reticulated? | 17:06 |
| lool | it's just bits flowing at light speed! | 17:07 |
| lool | hmm oddly this is still using xz and not pxz, not sure why | 17:07 |
| lool | ah it's decompress | 17:10 |
| lool | right and pxz supports it's, but it's single threaded | 17:12 |
| asac | Bluetooth has no write permission to setup config | 17:18 |
| asac | https://bugs.launchpad.net/ubuntu/+source/lxc-android-config/+bug/1234361 | 17:18 |
| asac | Fix for this landed Friday | 17:18 |
| ubot5 | Ubuntu bug 1234361 in lxc-android-config (Ubuntu) "Bluetooth panel not getting scan results" [High,Fix released] | 17:18 |
| asac | was this in the image today? | 17:18 |
| asac | lool: ? | 17:18 |
| asac | popey: ? | 17:19 |
| asac | :) | 17:19 |
| asac | can you confirm that this is good? | 17:19 |
| asac | sil2100: if it is really fixed, the above might be good to mention as a bug fixed | 17:19 |
| popey | I didnt test bluetooth | 17:19 |
| lool | promoting | 17:19 |
| popey | (I have no bluetooth devices) | 17:19 |
| cyphermox | popey: hold on | 17:19 |
| popey | maybe you need lool to hold on? | 17:20 |
| cyphermox | nah | 17:20 |
| cyphermox | all good | 17:20 |
| sil2100 | hoho | 17:20 |
| cyphermox | popey: I'll ship you or otherwise get you a bt device. | 17:20 |
| lool | asac: yes this fix is on | 17:20 |
| lool | is in | 17:20 |
| sil2100 | asac: right! I'll try to get some news on that one, I know it's in but not sure if it's really working ;) | 17:20 |
| popey | cyphermox: thanks | 17:20 |
| lool | popey, cyphermox: The change there is that your bluetooth pairings will survive reboots | 17:21 |
| popey | cyphermox: well, technically I _have_ some bluetooth devices (Ouya Controllers) but no headset | 17:21 |
| cyphermox | popey: yeah | 17:21 |
| cyphermox | I have a spare headset I never use in the office | 17:21 |
| cyphermox | I'll bring it to the core sprint? | 17:21 |
| lool | (image promoted) | 17:21 |
| * popey spies #68 coming down the pipe on his other phone | 17:21 | |
| lool | #68 that is | 17:21 |
| popey | \o/ | 17:21 |
| popey | hah, and #69 on the other one | 17:22 |
| lool | also #69 published just before or over the hangout | 17:22 |
| lool | right | 17:22 |
| lool | it was published when I started promoting | 17:22 |
| lool | I'll check around later in case there's any need of me, but otherwise I expect to kick another image tomorrow morning :-) | 17:22 |
| popey | cyphermox: ok | 17:22 |
| popey | thanks | 17:22 |
| === gatox_lunch is now known as gatox | ||
| sil2100 | asac: e-mail ready | 17:55 |
| sil2100 | asac: should I send it to you? | 17:56 |
| sil2100 | asac: sent it to you now, just take a quickie look and tell me if I can press the button! | 17:57 |
| sil2100 | Busy busy | 17:57 |
| sil2100 | asac: I think I'll send it out now, since I still have many things to do - I guess we can always throw an update e-mail later on, as a follow-up | 18:03 |
| ogra_ | ooooh ... i might get a 24h clock for christmas ! | 18:13 |
| ogra_ | awesome :) | 18:13 |
| sil2100 | ...;p | 18:14 |
| asac | sil2100: thx | 18:28 |
| asac | sil2100: good | 18:29 |
| === fginther` changed the topic of #ubuntu-ci-eng to: Ubuntu CI Engineering Team | Vanguard: fginther | Landing instructions: http://paste.ubuntu.com/6292280/ | Known issues: - | ||
| fginther` | sergiusens, I have a question regarding the ubuntu-touch-image job when you have some time | 18:48 |
| === fginther` is now known as fginther | ||
| sergiusens | fginther, shoot | 18:48 |
| fginther | sergiusens, the ~/phablet-build-scripts/ubuntu-touch-build script tries to bzr pull a set of 'vendor/*' and 'ubuntu/platform-api' sub directories, but they're not bzr repos. | 18:50 |
| fginther | sergiusens, as far as I can tell they were created by repo init | 18:50 |
| fginther | sergiusens, have these changed over time or do these directories need to be modified to point to a bzr branch? | 18:51 |
| fginther | sergiusens, http://s-jenkins.ubuntu-ci:8080/job/ubuntu-touch-image/94/console | 18:52 |
| ChrisTownsend | Hey Guys, Unity CI is being massively blocked by an armhf build failure. It *looks* to be a possible issue in Mesa 10, but I'm not 100% sure. I've entered a bug, but it's not getting anywhere fast: https://bugs.launchpad.net/ubuntu/+source/mesa/+bug/1260068 | 18:58 |
| ubot5 | Ubuntu bug 1260068 in mesa (Ubuntu) "Unity continuous-integration fails in armhf due possibly to a libegl issue" [Critical,Incomplete] | 18:58 |
| ChrisTownsend | Any suggestions on how we can debug this to know for sure what the issue is? In an email, Timo Aaltonen suggested to revert http://anonscm.debian.org/gitweb/?p=pkg-xorg/lib/mesa.git;a=commitdiff;h=5667b9f6871a9a04acb7da05d305851ba32461aa and try to see if the issue goes away. He suspected llvmpipe may be the issue in armhf. | 18:59 |
| sergiusens | fginther, the vendor stuff is in a rocket scientists branch | 19:01 |
| sergiusens | fginther, but they are private since non redistributable | 19:01 |
| fginther | sergiusens, do you remember how the job was originally setup? | 19:02 |
| fginther | ChrisTownsend, thinking... | 19:02 |
| ChrisTownsend | fginther: Ok, thanks | 19:02 |
| sergiusens | fginther, actually, it might be in the canonical wikis | 19:03 |
| fginther | sergiusens, thanks, I may have found it | 19:13 |
| fginther | ChrisTownsend, I think I can force a build with "LIBGL_ALWAYS_SOFTWARE=1" as mentioned in the bug | 19:14 |
| ChrisTownsend | fginther: Ok, that sounds good. | 19:14 |
| ChrisTownsend | fginther: I'm pretty sure it's not a Unity issue as unrelated merge proposals are getting the same issue on armhf. | 19:15 |
| fginther | ChrisTownsend, if you can find a PPA to build an newer armhf mesa with the specified commit reverted, we could build with that version of mesa too | 19:15 |
| fginther | at least as an experiment | 19:15 |
| ChrisTownsend | fginther: Could you use my PPA if I can get a package up? | 19:16 |
| fginther | ChrisTownsend, yes, any ppa should work, as long as it can build for armhf (I'm not sure if this is default behavior) | 19:17 |
| ChrisTownsend | fginther: Hmm, oh yeah, armhf... | 19:17 |
| sergiusens | fginther, branches are in lp:aal+ , I forgot to paste :-/ | 19:19 |
| bregma | ChrisTownsend, fginther, since none of the Unity MPs have passed CI since the Mesa change went in, it should be enough to build Unity head against the new Mesa to show it's not Unity causing the problem, right? | 19:28 |
| fginther | bregma, I wont disagree with that | 19:33 |
| thomi | Hi CI Team - someone's emailed me about AP tests failing (at merge time) on the mako because tha pp crashes. I'm wondering how hard it would be to get a retraced crash dump off the device? | 19:40 |
| thomi | Do you guys automatically retrace crash files? If not, would that be a good idea? | 19:40 |
| thomi | It would certainly help us figure out what's causing the app to crash. | 19:40 |
| ChrisTownsend | bregma: amd64 and i386 builds are fine and actually Unity builds fine on armhf, it's just when the tests begin to run when it seg faults. | 19:42 |
| bregma | ChrisTownsend, right, because the tests run a captive X server, and evidently the new Mesa causes the X server to fail to run on armhf ... it doesn't really seem fair to assume unity is at fault if an X server won't run on a newer version of a rendering back end | 19:44 |
| ChrisTownsend | bregma: Exactly. I think this coincides with the enablement of llvmpipe in armhf, but the onus is on one of us to prove that. | 19:45 |
| fginther | thomi, one moment | 19:46 |
| fginther | thomi, first, we do not automatically retrace the crashes. | 19:56 |
| fginther | thomi, I remember some discussion about this in the past and there being some technical reason why this was not a good idea to do on the devices, but can't remember any specifics or find any notes to back up my fuzzy memory | 19:56 |
| thomi | fginther: oh, OK. | 19:57 |
| thomi | I wonder why | 19:57 |
| thomi | maybe just a resource issue? | 19:57 |
| thomi | fginther: anyway, now I know :) | 19:57 |
| fginther | thomi, I do have an email chain with a discussion on how this can be done on your desktop, might help solve the problem temporarily until I can remember why this is or get it added. | 19:57 |
| thomi | I'll flash my phone manually and try and reproduce | 19:57 |
| thomi | fginther: if you could forward that to me, that'd be great | 19:58 |
| fginther | thomi, sure | 19:58 |
| thomi | thanks | 19:58 |
| doanac | thomi: question on subunit. I've got a simple test case I was running via subunit and piping through a custom filter i wrote. the filter extend testtools.StreamResult | 20:34 |
| doanac | all was okay, until i added a "print" statement to my test case. now the filter fails | 20:35 |
| doanac | any ideas? | 20:35 |
| fginther | ChrisTownsend, my build of lp:unity is complaining about a deprecated symbol: http://s-jenkins.ubuntu-ci:8080/job/unity-trusty-amd64-ci/59/console | 20:36 |
| thomi | doanac: send me the code maybe? I'll take a look | 20:36 |
| ChrisTownsend | fginther: Yep, I have a MP to fix that, but it can't get merged due to the armhf failure:-( | 20:37 |
| ChrisTownsend | Unless, I just manually merge the danged thing. | 20:37 |
| fginther | ChrisTownsend, can you send me the MP, I can use that | 20:37 |
| doanac | thomi: my test: http://paste.ubuntu.com/6585423/ and my filter: http://paste.ubuntu.com/6585425/ | 20:38 |
| ChrisTownsend | fginther: Just a sec... | 20:38 |
| ChrisTownsend | fginther: https://code.launchpad.net/~townsend/unity/fix-gtk-build-error/+merge/198611 | 20:39 |
| fginther | ChrisTownsend, thx | 20:40 |
| ChrisTownsend | fginther: np! | 20:40 |
| doanac | thomi: i figured it out. i was constructing the converter wrong. it needs a parameter "non_subunit_name='stdout'" | 20:49 |
| thomi | doanac: ahh yes, sorry, I didn;t get to look at it yet | 21:09 |
| thomi | doanac: if you're going to embed non-subunit data in the stream, then yeah, you need to tell the byte converter how to handle it | 21:09 |
| fginther | sergiusens, does an image build require an attach touch device? | 21:13 |
| sergiusens | nope | 21:17 |
| fginther | sergiusens, ok | 21:17 |
| === Ursinha-afk is now known as Ursinha | ||
| tedg | alesage, Do you think this'll do what I think it'll do? https://code.launchpad.net/~ted/cupstream2distro-config/coverage-ual-urld/+merge/199195 | 21:40 |
| alesage | tedg, yes but let's as fginther for a pass too :) | 21:41 |
| dobey | are there any current things running qmltestrunner in CI? | 21:43 |
| dobey | i'm having some trouble with it crashing on missing glx when i try to run it under xvfb :( | 21:44 |
| fginther | ChrisTownsend, can you determine if this test used llvmpipe as expected? http://s-jenkins.ubuntu-ci:8080/job/unity-trusty-amd64-ci/63/consoleFull | 21:44 |
| tedg | dobey, I'm sure there's a way to make QML not use GLX, but you can also run X11 with the dummy driver, which should support GLX. | 21:46 |
| fginther | dobey, lp:ubuntu-ui-toolkit uses qmltestrunner | 21:46 |
| dobey | tedg: how do you do that with xvfb-run? | 21:47 |
| fginther | tedg, approved | 21:48 |
| tedg | dobey, Not sure how to do it there, I've only done it with xorg-gtest | 21:48 |
| tedg | dobey, But I believe it just generates an X11 config file and then executes X11 using that. | 21:48 |
| ChrisTownsend | fginther: Yeah, it looks like LIBGL_ALWAYS_SOFTWARE=1 worked and it used llvmpipe. | 21:48 |
| tedg | fginther, Thanks! | 21:49 |
| ChrisTownsend | fginther: For reference: | 21:49 |
| ChrisTownsend | + export LIBGL_ALWAYS_SOFTWARE=1 | 21:49 |
| ChrisTownsend | + LIBGL_ALWAYS_SOFTWARE=1 | 21:49 |
| ChrisTownsend | I: user script /var/cache/pbuilder/build//90670/tmp/hooks/D00force_llvmpipe finished | 21:49 |
| fginther | ChrisTownsend, that's the bit I added, was wondering if there was another way to tell if the envvar was honored. | 21:50 |
| dobey | fginther: and you're certain it's running them during the package build? i can't see any clear indication of that :( | 21:50 |
| fginther | dobey, the build logs (example: http://s-jenkins.ubuntu-ci:8080/job/ubuntu-ui-toolkit-trusty-amd64-ci/413/console) contain lots of "Start testing of qmltestrunner" | 21:51 |
| ChrisTownsend | fginther: Oh, that script you added. | 21:51 |
| dobey | how is it not crashing? | 21:52 |
| ChrisTownsend | fginther: Hmm, I don't see any way to tell for sure. | 21:54 |
| dobey | does anyone know how to make qmltestrunner work ok under xvfb-run? it seems ubuntu-ui-toolkit isn't doing that (and in fact, i have no idea why it isn't crashing in jenkins) | 22:09 |
| === slangase` is now known as slangasek | ||
| === fginther changed the topic of #ubuntu-ci-eng to: Ubuntu CI Engineering Team | Vanguard: cihelp | Landing instructions: http://paste.ubuntu.com/6292280/ | Known issues: - | ||
Generated by irclog2html.py 2.7 by Marius Gedminas - find it at mg.pov.lt!
