[05:06] 19 [05:06] mt [06:44] hey mmrazik, how are you? [06:44] didrocks: morning [06:44] mmrazik: yesterday, unity FTBFS on armhf in our buildds, I'm wondering, you are still building unity on armhf in jenkins, right? [06:45] didrocks: we never did due to the build times [06:45] we recently had a discussion about this with bregma [06:45] mmrazik: did you put in place the improvements and checked with jibel as I told you? [06:46] didrocks: like what improvements? We are using libeatmydata, updagint the base images, etc [06:46] didrocks: there is actually something we plan to do and it is on its way [06:47] mmrazik: having the pbuilder on RAM for instance, he had other tricks iirc as wwell [06:47] didrocks: pbuilder on RAM on pandas is not really an option [06:47] mmrazik: ok, do you have a rough idea on when those improvements will happen? [06:47] didrocks: and we do have pbuilder on ARM btw [06:50] didrocks: ehm... I mean pbuilder on RAM [06:50] it doesn't make a huge difference, though [06:50] ok [07:22] I always wondered why we couldn't just cross-compile for ARM, but given that the last time I tried to do that it was a total pain [07:22] I guess I can see why [07:22] (python being the main problem) [07:59] didrocks: can I re-run the ps-indicators-autopilot-release-testing job? [07:59] would like to try the logging [08:00] mmrazik: it will dep-wait on the unity one which is running [08:00] mmrazik: as there are still some install issues, do you mind if we do that a little bit later? [08:00] mmrazik: I think jibel will again have to log in on the machine to see why we had this code return 2 [08:00] didrocks: ok. Can I save the config or it would be better if I don't touch that either [08:01] save the config == save the ps-indicators-autopilot-release-testing job in jenkins [08:01] mmrazik: oh please do save the config :) [08:01] ok [08:01] mmrazik: then, I'll tell you once we will rerun the job [08:01] thanks a lot :) [08:01] didrocks: cool. thanks [08:22] good morning! [08:54] sil2100: hey, how are you? [08:55] didrocks: hey! All is well, how about you? [08:56] sil2100: well, better than UTAH tbh :p [08:56] or rather the UTAH state is impacting me more and more [08:56] no release for a week now :/ [08:56] Still some more UTAH problems? [08:56] WTF? Why suddenly those problems? We never had problems like those [08:57] we always had problems [08:57] and have to relaunch [08:57] and so on [08:57] but yeah, now, this went from 60 to 100% of failures on at least one of the 3 machines [08:58] Woha! Ok, I'm clean! I didn't break anything ;) [08:58] sil2100: heh, however, we still get some partial results [08:58] sil2100: and random failure [08:58] sil2100: mind having a look? [08:58] didrocks: hate those - on which job? [08:58] sil2100: ps-indicators-autopilot-release-testing, ati [08:58] * sil2100 needs to poke someone today to review his ibus fixes [08:58] the only which ran :/ [08:58] run 122 [09:00] sil2100: also, if you want relatively fresh unity results: [09:00] ps-unity-autopilot-release-testing/82 [09:00] ati and intel ran [09:00] This one again, I'm starting to think it's another one of those hidden, randomly-appearing regressions, focus from the HUD is obviously broken [09:00] we do have 25 and 26 tests failing [09:01] sil2100: at least, if it is a regression, it's already a start that we catch those :) [09:01] didrocks: I think now that it's been reproduced more than a few times in the past few builds, I think I can try patching it up [09:02] didrocks: I'm also still working on the introspection-problem with preview tests [09:02] That's 3 failing tests from-time-to-time [09:05] sil2100: excellent :-) [09:05] thanks sil2100 [09:07] np, hope we'll be able to resolve all problems - both AP and UTAH [09:08] sil2100: I cross fingers as well [09:28] hah, ok, good thing! I was able to reproduce it locally, the failing test-case [09:28] sil2100: \o/ [11:24] didrocks: sil2100: mmrazik: hey, do any of you guys know what the status of xorg-gtest is in the distro [11:24] I've been told that its been upgraded to 0.8, but it seems like the CI builders have 0.3 [11:25] smspillaz: we do have 0.7 in the distro https://launchpad.net/ubuntu/+source/xorg-gtest [11:25] smspillaz: 0.3 sounds like some builders are under quantal, not raring [11:25] didrocks: ah okay [11:26] mmrazik: I know you guys are busy today, so I won't bother you too much, just something to keep on the radar: http://jenkins.qa.ubuntu.com/job/compiz-clang-ci/build=pbuilder,distribution=quantal,flavor=amd64/63/console (we should probably upgrade the builders to raring) [11:27] in the meantime, I'll add a proper version check as I see we are missing that [11:27] smspillaz: ack. Will do [11:27] its easy. let me do it right away so I don't forget [11:27] ah okay. I'll add the version check now too, since we really should be doing that on our end :) [11:28] smspillaz: should be fixed [11:29] great, thanks [11:30] didrocks: mmrazik: we should now have all the compiz tests run under CI now \o/ [11:33] smspillaz: sweet! :) nice work! [11:40] smspillaz: btw. do we need to run the memcheck tests also for clang/gles? [11:40] they take aaaages [11:40] (just saying:-) [11:43] mmrazik: we might need them for gles, maybe not clang [11:44] mmrazik: they do take ages though - and they are not really useful for catching regressions in branch submissions because they are run in silent mode [11:44] maybe it would be better for compiz-mbs-autolanding ? [11:45] smspillaz: I don't have any strong opinion. If you tell me we should do it only for autolanding and disable for ci then we can do it [11:46] smspillaz: the reason why I'm asking that I've a lot of ci jobs in the queue in the last two days or so [11:46] if its taking up your resources, I would be behind it [11:46] mmrazik: heh, yeah, mea culpa :p [11:46] and they were waiting just for the others to finish (we should try to run them in parallel for different MPs but that is another story) [11:46] mmrazik: hehe, maybe we should borrow google's test infrastructure :p [11:47] they have this big system which determines which tests actually need to be re-run for every change [11:47] I think chromium has something like 30k tests, so it saves them a bit :p [11:47] of course, that is very complicated :) === _salem is now known as salem_ === mmrazik is now known as mmrazik|lunch === mmrazik|lunch is now known as mmrazik === dandrader is now known as dandrader|afk === dandrader|afk is now known as dandrader [12:56] smspillaz: Trevinho: we are getting regular compiz crashes on one intel machine where autopilot is run, the best stacktrace I could get is http://paste.ubuntu.com/1643707/ though :/ [13:02] ;/ [13:29] didrocks: on startup or at runtime ? [13:30] smspillaz: hard to know TBH, I would suspect startup/shutdown [13:30] (even more shutdown TBH) [13:30] it happens reliably once per multiple shutdown on that machine [13:30] not sure why [13:30] didrocks: do you have the stdout ? [13:31] so farno one has been able to pinpoint exactly where that stacksmasher crash happens, since there is lag between the crash and the reporting and no one can repro it outside of the autobuilder [13:31] maybe unity is crashing when it gets unloaded ? *shrug* [13:31] smspillaz: unfortunately not, but good point, will try to get it for next time [13:31] bregma: oh? [13:31] bregma: when all else fails, printf-style debugging can help a bit [13:31] as far as I know [13:31] also make sure you build with -g and no optimization [13:32] (though afaict you're doing that already) [13:32] -ggdb3 -O0 won;t help with a stack smasher [13:33] problem is either (1) ABI mismatch (2) invalid function pointer (3) local array walker [13:33] or, like with javascript, invalid JIT code [13:33] which does not apply here [13:33] didrocks: bregma: I'd suggest setting up a new job just for that machine which just loads compiz + unity and shuts down again [13:33] good plan [13:34] that will at least make the feedback loop a lot shorter [13:34] bregma: do you have another way of reproducing it than this machine? [13:34] bregma: actually, I don't trust gnome not to pull in a JIT [13:34] ;-) [13:34] because for this one, reinstalling is mandatory with the system and it's taking a lot of time [13:34] no one I have spoke to has been able to reproduce it [13:34] bregma: I heard that gnome is switching to another favorite language again, lol [13:35] of course, maybe someone has reproduced the problem and not told me [13:35] C! wait, python! wait, C#!, wait, vala! wait, javascript! [13:35] didrocks: is reinstalling /really/ mandatory ? [13:35] surely it isn;t [13:35] *isn't [13:36] smspillaz: well, it is with this process, unfortunately [13:36] smspillaz: as the machine is used that way and dont' really have an access to it [13:36] how can we not have access to one of our own machines? [13:37] smspillaz: I can't, some can, but those people don't know how to debug/run that [13:39] this really is the joy of "it fails on CI and I don't know why" [13:40] similarly, yeah [13:40] didrocks: how long does the reinstall process take ? [13:40] and for comparison, how long does the full AP take ? [13:41] a good strategy would be to grab the stdout, figure out which test its crashing on, and then create a new job for that machine which just runs the suite [13:41] smspillaz: well, more strangely, if we run *all* tests, including those ones, it's not crashing [13:41] smspillaz: but maybe autopilot doesn't act the same for session restart between "running just some tests or all tests" [13:42] thomi should know though ^ [13:42] wait, so do we know which test its crashing on? [13:42] or at least, a way to reproduce the problem which doesn't take forever ? [13:42] smspillaz: well, it's taking 40 minutes in total [13:42] smspillaz: with only those 89 tests running [13:43] but, the session is randomly restarted [13:43] the first thing I'd suggest doing at least, would be to run compiz --debug [13:43] by "session" are you talking about X, or compiz ? [13:43] smspillaz: this needs a lot of tweaking [13:43] X session [13:43] so X is crashing!?? [13:43] smspillaz: no, compiz [13:43] wait what [13:43] smspillaz: but lightdm is restarted regularly [13:43] between tests [13:43] from what I know [13:43] oh okay [13:45] I think at next crash, I'll look for the timestamp of the .crash file [13:45] at least, it will tell us if it's at the beginning of end of tests [13:47] huh? doesn't autopilot have some kind of method to handle the case where it doesn't receive a reply from dbus ? [13:47] not sure [13:47] I'm pretty sure theres a dbus error that you get when the relevant client sends a SIGHUP or whatever [13:49] didrocks: what makes running compiz with --debug so difficult btw ? [13:49] smspillaz: well, we need a different package and so on, because compiz is launched by the session [13:49] and we need a hook to save the .xsession-errors for each session [13:50] doesn't autopilot monitor the stdout? [13:50] ohmygawd.... [13:50] and again, only thomi knows how and when autopilot restarts the session and the difference between "run all tests" and "a set of tests provided" [13:50] smspillaz: no [13:50] ok, now going to do some exercice, already 3 hours late :) [13:51] * smspillaz thinks we should rewrite all of the AP tests in xorg-gtest [13:51] sil2100: still around ? [13:51] (maybe you might know some more about this) [13:53] smspillaz: hi! [13:53] smspillaz: what's up? Some tests need rewriting ;) ? [13:54] sil2100: how hard would it be to roll a ppa to the machine with the failing AP test to run compiz with --debug and capture the stdout [13:57] smspillaz: I don't think it should be that hard, but probably someone from the jenkins-maintainers would have to help out [13:57] sil2100: okay [13:57] sil2100: make it happen [13:58] smspillaz: ok :) Will poke people about it then! You only need compiz --debug and the output, or something else? Should the tests run? [13:58] * sil2100 needs to get the context correctly [13:58] yeah keep the tests running [13:58] we just need to get [13:58] a) Which test autopilot detected that the compiz process stopped responding to dbus calls [13:59] b) The compiz stdout/stderr (eg .xsession-errors) from --debug [13:59] from there we can start to pinpoint the source of this === dandrader is now known as dandrader|afk [15:47] mterry: cyphermox: hey guys, just a quick summary of the day for dailyrelease [15:47] mterry: cyphermox: so, we got the UTAH new failure of the day, I debugged it and workarounded it [15:47] mterry: cyphermox: unfortunately, we are still hitting another random $install issue if we run only the indicator tests [15:48] with mterry's manual run of unity, I hope we can have that passing [15:48] if so, I would propose: [15:48] - manually publishing unity [15:48] - manually publishing indicator (even if the tests didn't pass because of the installed, they are run as part of unity) [15:48] - manual publishing of oif which failed for a similar random behavior [15:49] so that tomorrow, we can try transitionning from the HUD [15:49] * didrocks spent a big part of his day debugging UTAH for upstream, not pleasant… [15:49] mterry: cyphermox: wdyt? [15:49] yeah [15:49] the tests for indicators look okay enough, AFAICS [15:49] (the new failure was "not enough loop device for mounting the iso") [15:50] oh boy [15:50] cyphermox: yeah, seems so, if you try to agregate the results from different runs on all configs :) [15:50] cyphermox: I think oif is safe already, mind a manual publishing? [15:50] didrocks, OK [15:50] at least, this one will be done :) [15:50] ok [15:51] meanwhile, misc continues to publish everyday nicely (as not UTAH/integration tests involved) :p [15:51] no* [15:51] hehe [15:51] andyrock: thanks for the amrhf fix btw :) [15:52] np [15:53] ok starting with oif [15:53] ./cu2d-run -P oif [15:53] didrocks, let's just put unity in misc then! [15:53] * mterry is a problem solver [15:53] * cyphermox likes mterry's idea [15:54] cyphermox, watch out! you own misc stack [15:54] didrocks: ^ confirm? ;) [15:54] ah crap you're right [15:54] cyphermox: confirmed! [15:54] muhaha [15:54] mterry: hum hum [15:54] mterry: don't even start that :p [15:54] let's NOT put unity in misc ;) [15:55] mterry: TBH, you just need to remove one parameter from the yaml file and republish the jobs [15:55] mterry: and then, it will forget about the -check step :p [15:55] not sure it's what we want though :p [15:55] didrocks: it would be fine [15:55] it's all very stable code, no? [15:56] oif publishing... [15:56] cyphermox: hem hem :p [15:56] we saw that everyday ;) [16:00] and oif released :) [16:04] \o/ [16:05] yay [16:06] so now, iindicators or unity first? === dandrader|afk is now known as dandrader [16:13] cyphermox: unity is rerunning with full tests [16:13] cyphermox: let's see the results [16:13] if ok, we can publish indicators and unity [16:13] manual publishing as well [16:14] (I'm finally happy to have enabled that manual publishing can bystep the status of build/check for all "prepared" source [16:14] seems to be handy when I thought it wouldn't :p [16:14] ok, at least, the 3 machines installed and rebooted successfully [16:14] they are now running tests [16:15] hmmm [16:16] sil2100: shaking? :) [16:16] I almost found something, but I need to jump out for practice now - I'll resume my work in the evening [16:16] sil2100: sweet, do you mind having a look at the results of the unity test job once you are back? [16:16] didrocks: shaking what? [16:16] didrocks: no problem - both indicators and unity, right? [16:17] sil2100: just thinking as I was telling "the tests are now running" that you were troubled as it means your new tests are running :p [16:17] sil2100: just unity [16:17] all would be run that way anyway [16:17] didrocks: ok, no problem ;) [16:17] thanks sil2100 :) [16:17] sil2100: enjoy your exercice! === xnox is now known as foxtrot === foxtrot is now known as xnox [17:18] mterry: cyphermox: the tests results sounds good to me on ati, and nvidia, I've a doubt on intel though [17:18] mterry: cyphermox: I don't see anything weird, using the ppa, do you? [17:18] (I've some intel hw) [17:19] let me double check [17:19] I'm on intel too anyway [17:21] thanks :) [17:22] didrocks, I haven't rebooted in days [17:22] I'll update and restart [17:24] didrocks, you aren't suggesting that we ignore the threshold here and manually publish are you? tsk tsk [17:24] Where's that ibus fix? That should drop the tests nicely [17:24] sil2100: ^ [17:24] mterry: well, it's as you wish, you are stack holders :-) [17:25] Hrm, not reviewed yet. sil2100 has it waiting [17:25] for review that is [17:25] mterry: just that knowing how all those things are fuzzy… [17:25] then you decide with cyphermox if you publish indicators/unity ;) [17:25] didrocks, a reason to drive them to perfection! :) You mentioned a reason you wanted a publish checkpoint? HUD stuff? [17:26] mterry: yeah, I wanted a before, and then after HUD [17:26] as we have the HUD transition [17:26] and the day after that libcolumbus [17:27] mterry: it's really as you feel it, maybe we shouldn't push the trigger, yeah [17:28] you're talking about unity specifically there right? [17:28] because we really should finish up publishing indicators so that we can move forward on hud [17:31] cyphermox, yeah unity. The new test failures all seemed to come from dash tests [17:31] cyphermox: if we are sure we don't have dep between indicators and unity publishin, I think we can (I don't see indicators tests failing in the unity suite run) [17:31] mterry: I wonder why the dash failed so much on intel [17:31] still a little bit on nvidia [17:35] didrocks, cyphermox: Is publishing indicators sufficient to not block hud? [17:36] cyphermox, ah, still getting crashes on the indicator tests... :-/ [17:36] mterry: yeah [17:37] it's sufficient [17:37] well what we really need is to ship libcolumbus and all so that we can ship hud; then do the hudectomy in indicator-appmenu, and then republish hud for libcolumbus, IIRC [17:37] OK, then I'd just as soon leave unity to its own devices as long as there's no pressing need for that stack specifically [17:38] sil2100's ibus branch should get us under threshold again (at which point, we should probably lower the threshold a bit) [17:56] cyphermox: so, manual publishing for the indicators? [17:57] yeah, let's [17:57] aside from that, was libcolumbus ready to be uploaded, added to the stacks and everything? [17:57] cyphermox: we need the hud first [17:57] then, libcolumbus :) [17:58] but IIRC, libcolumbus is ready [17:58] well, yeah, but I mean, libcolumbus can land now and hud not depend on it immediately [17:58] then it give us time for the MIR in parallel, even if it won't take so much time === dandrader is now known as dandrader|afk [18:01] cyphermox: well, you need to rerun a rebuild if you do that, before the publishing [18:02] libcolumbus will be in indicators stack right? [18:02] I'm doing the publishing now, but then I'll add it to the stack and update [18:02] cyphermox: sounds good to me :) [18:03] ./cu2d-run -P indicators [18:03] cyphermox: I'm just leaving, but I think we won't have it landing automatically tomorrow as it will be at least in manual publishing mode :) [18:03] cyphermox: yep ;) [18:03] due to the new package [18:03] yeah [18:03] so don't need to refresh the AA side right away [18:03] I'll do that tomorrow morning [18:03] no [18:03] I'll only update the stack later [18:03] cyphermox, didrocks: I can NEW review if you want [18:04] seb128: ok [18:04] seb128: not the issue, you need to pull on lillypilly [18:04] it won't land today anyway [18:04] seb128: for the AA-side filtering [18:04] didrocks, I was proposing to review it once it's in the raring NEW queue [18:04] didrocks, I didn't follow what you are doing out of "it's going to be uploaded in the next days" ;-) [18:04] seb128: yeah, it will be marked as a sync :) [18:05] oh, right [18:05] seb128: but once cyphermox updated the list, first, to be uploaded to the archive, we need to refresh the filtering [18:05] well in any case if NEW review is needed just ping me [18:05] seb128: yeah, but you can do the filter refreshing as well once cyphermox will add it to the list (after this manual publishing round) [18:05] ok! [18:05] seb128: on lillypilly, just log as archive admin [18:05] cd cu2d [18:06] cd cupstream2distro [18:06] bzr pull [18:06] you should see the updated yaml file once cyphermox did it :) [18:06] didrocks, ok, cool, good to know ;-) [18:06] * seb128 tomboys that [18:11] seb128: https://wiki.ubuntu.com/DailyRelease/FAQ#Adding.2BAC8-removing_components_to_a_stack [18:11] second part :) [18:12] didrocks, ahah, excellent ;-) [18:12] hot from the press! :) === dandrader|afk is now known as dandrader [20:01] sil2100, did someone mention that last night intel had 10 new dash failures? http://10.97.0.1:8080/job/ps-unity-autopilot-release-testing/label=autopilot-intel/lastCompletedBuild/testReport/ [20:01] sil2100, do you know why that might be? [20:19] mterry: checking [20:20] sil2100, thanks for rocking that ibus merge proposal too [20:20] I'm looking forward to that landing [20:20] mterry: np! Hope it fixes the problems for good [20:50] hm [21:05] mterry: those failing tests... they seem to be related to the previous failing tests [21:06] Not sure why now they started failing [21:06] But the reason seems to be a problem while fetching the state of the preview open, I'm already looking into it regarding the previous failures [21:06] mterry: if anything, I'm currently looking at two failure types (so that we don't step on eachothers toes ;) ) [21:07] The first thing: the dash previews having problems [21:07] The second: HUD focus issues [21:10] sil2100, awesome, thanks! [21:10] One day we'll have no failures! === salem_ is now known as _salem [22:26] thomi: hi! Once you're up and running, could you take a look on two of my autopilot merges for lp:unity? [22:27] thomi: one regarding ibus, the second regarding a new test suite (for search) [22:28] mterry: just to make sure - the testing environment for the release test jobs is single-monitor, right? [22:44] sil2100: sure [22:44] * thomi looks now [22:44] \o/ Thanks :) [22:45] https://code.launchpad.net/~sil2100/unity/autopilot_search_test_suite [22:45] https://code.launchpad.net/~sil2100/unity/autopilot_ibus_improve === _salem is now known as salem_