didrockshey mmrazik, how are you?06:44
mmrazikdidrocks: morning06:44
didrocksmmrazik: yesterday, unity FTBFS on armhf in our buildds, I'm wondering, you are still building unity on armhf in jenkins, right?06:44
mmrazikdidrocks: we never did due to the build times06:45
mmrazikwe recently had a discussion about this with bregma06:45
didrocksmmrazik: did you put in place the improvements and checked with jibel as I told you?06:45
mmrazikdidrocks: like what improvements? We are using libeatmydata, updagint the base images, etc06:46
mmrazikdidrocks: there is actually something we plan to do and it is on its way06:46
didrocksmmrazik: having the pbuilder on RAM for instance, he had other tricks iirc as wwell06:47
mmrazikdidrocks: pbuilder on RAM on pandas is not really an option06:47
didrocksmmrazik: ok, do you have a rough idea on when those improvements will happen?06:47
mmrazikdidrocks: and we do have pbuilder on ARM btw06:47
mmrazikdidrocks: ehm... I mean pbuilder on RAM06:50
mmrazikit doesn't make a huge difference, though06:50
smspillazI always wondered why we couldn't just cross-compile for ARM, but given that the last time I tried to do that it was a total pain07:22
smspillazI guess I can see why07:22
smspillaz(python being the main problem)07:22
mmrazikdidrocks: can I re-run the ps-indicators-autopilot-release-testing job?07:59
mmrazikwould like to try the logging07:59
didrocksmmrazik: it will dep-wait on the unity one which is running08:00
didrocksmmrazik: as there are still some install issues, do you mind if we do that a little bit later?08:00
didrocksmmrazik: I think jibel will again have to log in on the machine to see why we had this code return 208:00
mmrazikdidrocks: ok. Can I save the config  or it would be better if I don't touch that either08:00
mmraziksave the config == save the ps-indicators-autopilot-release-testing job in jenkins08:01
didrocksmmrazik: oh please do save the config :)08:01
didrocksmmrazik: then, I'll tell you once we will rerun the job08:01
didrocksthanks a lot :)08:01
mmrazikdidrocks: cool. thanks08:01
jibelgood morning!08:22
didrockssil2100: hey, how are you?08:54
sil2100didrocks: hey! All is well, how about you?08:55
didrockssil2100: well, better than UTAH tbh :p08:56
didrocksor rather the UTAH state is impacting me more and more08:56
didrocksno release for a week now :/08:56
sil2100Still some more UTAH problems?08:56
sil2100WTF? Why suddenly those problems? We never had problems like those08:56
didrockswe always had problems08:57
didrocksand have to relaunch08:57
didrocksand so on08:57
didrocksbut yeah, now, this went from 60 to 100% of failures on at least one of the 3 machines08:57
sil2100Woha! Ok, I'm clean! I didn't break anything ;)08:58
didrockssil2100: heh, however, we still get some partial results08:58
didrockssil2100: and random failure08:58
didrockssil2100: mind having a look?08:58
sil2100didrocks: hate those - on which job?08:58
didrockssil2100: ps-indicators-autopilot-release-testing, ati08:58
* sil2100 needs to poke someone today to review his ibus fixes08:58
didrocksthe only which ran :/08:58
didrocksrun 12208:58
didrockssil2100: also, if you want relatively fresh unity results:09:00
didrocksati and intel ran09:00
sil2100This one again, I'm starting to think it's another one of those hidden, randomly-appearing regressions, focus from the HUD is obviously broken09:00
didrockswe do have 25 and 26 tests failing09:00
didrockssil2100: at least, if it is a regression, it's already a start that we catch those :)09:01
sil2100didrocks: I think now that it's been reproduced more than a few times in the past few builds, I think I can try patching it up09:01
sil2100didrocks: I'm also still working on the introspection-problem with preview tests09:02
sil2100That's 3 failing tests from-time-to-time09:02
didrockssil2100: excellent :-)09:05
didrocksthanks sil210009:05
sil2100np, hope we'll be able to resolve all problems - both AP and UTAH09:07
didrockssil2100: I cross fingers as well09:08
sil2100hah, ok, good thing! I was able to reproduce it locally, the failing test-case09:28
didrockssil2100: \o/09:28
smspillazdidrocks: sil2100: mmrazik: hey, do any of you guys know what the status of xorg-gtest is in the distro11:24
smspillazI've been told that its been upgraded to 0.8, but it seems like the CI builders have 0.311:24
didrockssmspillaz: we do have 0.7 in the distro https://launchpad.net/ubuntu/+source/xorg-gtest11:25
didrockssmspillaz: 0.3 sounds like some builders are under quantal, not raring11:25
smspillazdidrocks: ah okay11:25
smspillazmmrazik: I know you guys are busy today, so I won't bother you too much, just something to keep on the radar: http://jenkins.qa.ubuntu.com/job/compiz-clang-ci/build=pbuilder,distribution=quantal,flavor=amd64/63/console (we should probably upgrade the builders to raring)11:26
smspillazin the meantime, I'll add a proper version check as I see we are missing that11:27
mmraziksmspillaz: ack. Will do11:27
mmrazikits easy. let me do it right away so I don't forget11:27
smspillazah okay. I'll add the version check now too, since we really should be doing that on our end :)11:27
mmraziksmspillaz: should be fixed11:28
smspillazgreat, thanks11:29
smspillazdidrocks: mmrazik: we should now have all the compiz tests run under CI now \o/11:30
didrockssmspillaz: sweet! :) nice work!11:33
mmraziksmspillaz: btw. do we need to run the memcheck tests also for clang/gles?11:40
mmrazikthey take aaaages11:40
mmrazik(just saying:-)11:40
smspillazmmrazik: we might need them for gles, maybe not clang11:43
smspillazmmrazik: they do take ages though - and they are not really useful for catching regressions in branch submissions because they are run in silent mode11:44
smspillazmaybe it would be better for compiz-mbs-autolanding ?11:44
mmraziksmspillaz: I don't have any strong opinion. If you tell me we should do it only for autolanding and disable for ci then we can do it11:45
mmraziksmspillaz: the reason why I'm asking that I've a lot of ci jobs in the queue in the last two days or so11:46
smspillazif its taking up your resources, I would be behind it11:46
smspillazmmrazik: heh, yeah, mea culpa :p11:46
mmrazikand they were waiting just for the others to finish (we should try to run them in parallel for different MPs but that is another story)11:46
smspillazmmrazik: hehe, maybe we should borrow google's test infrastructure :p11:46
smspillazthey have this big system which determines which tests actually need to be re-run for every change11:47
smspillazI think chromium has something like 30k tests, so it saves them a bit :p11:47
smspillazof course, that is very complicated :)11:47
=== _salem is now known as salem_
=== mmrazik is now known as mmrazik|lunch
=== mmrazik|lunch is now known as mmrazik
=== dandrader is now known as dandrader|afk
=== dandrader|afk is now known as dandrader
didrockssmspillaz: Trevinho: we are getting regular compiz crashes on one intel machine where autopilot is run, the best stacktrace I could get is http://paste.ubuntu.com/1643707/ though :/12:56
smspillazdidrocks: on startup or at runtime ?13:29
didrockssmspillaz: hard to know TBH, I would suspect startup/shutdown13:30
didrocks(even more shutdown TBH)13:30
didrocksit happens reliably once per multiple shutdown on that machine13:30
didrocksnot sure why13:30
smspillazdidrocks: do you have the stdout ?13:30
bregmaso farno one has been able to pinpoint exactly where that stacksmasher crash happens, since there is lag between the crash and the reporting and no one can repro it outside of the autobuilder13:31
smspillazmaybe unity is crashing when it gets unloaded ? *shrug*13:31
didrockssmspillaz: unfortunately not, but good point, will try to get it for next time13:31
didrocksbregma: oh?13:31
smspillazbregma: when all else fails, printf-style debugging can help a bit13:31
bregmaas far as I know13:31
smspillazalso make sure you build with -g and no optimization13:31
smspillaz(though afaict you're doing that already)13:32
bregma-ggdb3 -O0 won;t help with a stack smasher13:32
bregmaproblem is either (1) ABI mismatch (2) invalid function pointer  (3) local array walker13:33
bregmaor, like with javascript, invalid JIT code13:33
bregmawhich does not apply here13:33
smspillazdidrocks: bregma: I'd suggest setting up a new job just for that machine which just loads compiz + unity and shuts down again13:33
bregmagood plan13:33
smspillazthat will at least make the feedback loop a lot shorter13:34
didrocksbregma: do you have another way of reproducing it than this machine?13:34
smspillazbregma: actually, I don't trust gnome not to pull in a JIT13:34
didrocksbecause for this one, reinstalling is mandatory with the system and it's taking a lot of time13:34
bregmano one I have spoke to has been able to reproduce it13:34
smspillazbregma: I heard that gnome is switching to another favorite language again, lol13:34
bregmaof course, maybe someone has reproduced the problem and not told me13:35
smspillazC! wait, python! wait, C#!, wait, vala! wait, javascript!13:35
smspillazdidrocks: is reinstalling /really/ mandatory ?13:35
smspillazsurely it isn;t13:35
didrockssmspillaz: well, it is with this process, unfortunately13:36
didrockssmspillaz: as the machine is used that way and dont' really have an access to it13:36
smspillazhow can we not have access to one of our own machines?13:36
didrockssmspillaz: I can't, some can, but those people don't know how to debug/run that13:37
smspillazthis really is the joy of "it fails on CI and I don't know why"13:39
didrockssimilarly, yeah13:40
smspillazdidrocks: how long does the reinstall process take ?13:40
smspillazand for comparison, how long does the full AP take ?13:40
smspillaza good strategy would be to grab the stdout, figure out which test its crashing on, and then create a new job for that machine which just runs the suite13:41
didrockssmspillaz: well, more strangely, if we run *all* tests, including those ones, it's not crashing13:41
didrockssmspillaz: but maybe autopilot doesn't act the same for session restart between "running just some tests or all tests"13:41
didrocksthomi should know though ^13:42
smspillazwait, so do we know which test its crashing on?13:42
smspillazor at least, a way to reproduce the problem which doesn't take forever ?13:42
didrockssmspillaz: well, it's taking 40 minutes in total13:42
didrockssmspillaz: with only those 89 tests running13:42
didrocksbut, the session is randomly restarted13:43
smspillazthe first thing I'd suggest doing at least, would be to run compiz --debug13:43
smspillazby "session" are you talking about X, or compiz ?13:43
didrockssmspillaz: this needs a lot of tweaking13:43
didrocksX session13:43
smspillazso X is crashing!??13:43
didrockssmspillaz: no, compiz13:43
smspillazwait what13:43
didrockssmspillaz: but lightdm is restarted regularly13:43
didrocksbetween tests13:43
didrocksfrom what I know13:43
smspillazoh okay13:43
didrocksI think at next crash, I'll look for the timestamp of the .crash file13:45
didrocksat least, it will tell us if it's at the beginning of end of tests13:45
smspillazhuh? doesn't autopilot have some kind of method to handle the case where it doesn't receive a reply from dbus ?13:47
didrocksnot sure13:47
smspillazI'm pretty sure theres a dbus error that you get when the relevant client sends a SIGHUP or whatever13:47
smspillazdidrocks: what makes running compiz with --debug so difficult btw ?13:49
didrockssmspillaz: well, we need a different package and so on, because compiz is launched by the session13:49
didrocksand we need a hook to save the .xsession-errors for each session13:49
smspillazdoesn't autopilot monitor the stdout?13:50
didrocksand again, only thomi knows how and when autopilot restarts the session and the difference between "run all tests" and "a set of tests provided"13:50
didrockssmspillaz: no13:50
didrocksok, now going to do some exercice, already 3 hours late :)13:50
* smspillaz thinks we should rewrite all of the AP tests in xorg-gtest13:51
smspillazsil2100: still around ?13:51
smspillaz(maybe you might know some more about this)13:51
sil2100smspillaz: hi!13:53
sil2100smspillaz: what's up? Some tests need rewriting ;) ?13:53
smspillazsil2100: how hard would it be to roll a ppa to the machine with the failing AP test to run compiz with --debug and capture the stdout13:54
sil2100smspillaz: I don't think it should be that hard, but probably someone from the jenkins-maintainers would have to help out13:57
smspillazsil2100: okay13:57
smspillazsil2100: make it happen13:57
sil2100smspillaz: ok :) Will poke people about it then! You only need compiz --debug and the output, or something else? Should the tests run?13:58
* sil2100 needs to get the context correctly13:58
smspillazyeah keep the tests running13:58
smspillazwe just need to get13:58
smspillaza) Which test autopilot detected that the compiz process stopped responding to dbus calls13:58
smspillazb) The compiz stdout/stderr (eg .xsession-errors) from --debug13:59
smspillazfrom there we can start to pinpoint the source of this13:59
=== dandrader is now known as dandrader|afk
didrocksmterry: cyphermox: hey guys, just a quick summary of the day for dailyrelease15:47
didrocksmterry: cyphermox: so, we got the UTAH new failure of the day, I debugged it and workarounded it15:47
didrocksmterry: cyphermox: unfortunately, we are still hitting another random $install issue if we run only the indicator tests15:47
didrockswith mterry's manual run of unity, I hope we can have that passing15:48
didrocksif so, I would propose:15:48
didrocks- manually publishing unity15:48
didrocks- manually publishing indicator (even if the tests didn't pass because of the installed, they are run as part of unity)15:48
didrocks- manual publishing of oif which failed for a similar random behavior15:48
didrocksso that tomorrow, we can try transitionning from the HUD15:49
* didrocks spent a big part of his day debugging UTAH for upstream, not pleasant…15:49
didrocksmterry: cyphermox: wdyt?15:49
cyphermoxthe tests for indicators look okay enough, AFAICS15:49
didrocks(the new failure was "not enough loop device for mounting the iso")15:49
cyphermoxoh boy15:50
didrockscyphermox: yeah, seems so, if you try to agregate the results from different runs on all configs :)15:50
didrockscyphermox: I think oif is safe already, mind a manual publishing?15:50
mterrydidrocks, OK15:50
didrocksat least, this one will be done :)15:50
didrocksmeanwhile, misc continues to publish everyday nicely (as not UTAH/integration tests involved) :p15:51
didrocksandyrock: thanks for the amrhf fix btw :)15:51
cyphermoxok starting with oif15:53
cyphermox./cu2d-run -P oif15:53
mterrydidrocks, let's just put unity in misc then!15:53
* mterry is a problem solver15:53
* cyphermox likes mterry's idea15:53
mterrycyphermox, watch out!  you own misc stack15:54
cyphermoxdidrocks: ^ confirm? ;)15:54
cyphermoxah crap you're right15:54
didrockscyphermox: confirmed!15:54
didrocksmterry: hum hum15:54
didrocksmterry: don't even start that :p15:54
cyphermoxlet's NOT put unity in misc ;)15:54
didrocksmterry: TBH, you just need to remove one parameter from the yaml file and republish the jobs15:55
didrocksmterry: and then, it will forget about the -check step :p15:55
didrocksnot sure it's what we want though :p15:55
cyphermoxdidrocks: it would be fine15:55
cyphermoxit's all very stable code, no?15:55
cyphermoxoif publishing...15:56
didrockscyphermox: hem hem :p15:56
didrockswe saw that everyday ;)15:56
didrocksand oif released :)16:00
cyphermoxso now, iindicators or unity first?16:06
=== dandrader|afk is now known as dandrader
didrockscyphermox: unity is rerunning with full tests16:13
didrockscyphermox: let's see the results16:13
didrocksif ok, we can publish indicators and unity16:13
didrocksmanual publishing as well16:13
didrocks(I'm finally happy to have enabled that manual publishing can bystep the status of build/check for all "prepared" source16:14
didrocksseems to be handy when I thought it wouldn't :p16:14
didrocksok, at least, the 3 machines installed and rebooted successfully16:14
didrocksthey are now running tests16:14
didrockssil2100: shaking? :)16:16
sil2100I almost found something, but I need to jump out for practice now - I'll resume my work in the evening16:16
didrockssil2100: sweet, do you mind having a look at the results of the unity test job once you are back?16:16
sil2100didrocks: shaking what?16:16
sil2100didrocks: no problem - both indicators and unity, right?16:16
didrockssil2100: just thinking as I was telling "the tests are now running" that you were troubled as it means your new tests are running :p16:17
didrockssil2100: just unity16:17
didrocksall would be run that way anyway16:17
sil2100didrocks: ok, no problem ;)16:17
didrocksthanks sil2100 :)16:17
didrockssil2100: enjoy your exercice!16:17
=== xnox is now known as foxtrot
=== foxtrot is now known as xnox
didrocksmterry: cyphermox: the tests results sounds good to me on ati, and nvidia, I've a doubt on intel though17:18
didrocksmterry: cyphermox: I don't see anything weird, using the ppa, do you?17:18
didrocks(I've some intel hw)17:18
cyphermoxlet me double check17:19
cyphermoxI'm on intel too anyway17:19
didrocksthanks :)17:21
mterrydidrocks, I haven't rebooted in days17:22
mterryI'll update and restart17:22
mterrydidrocks, you aren't suggesting that we ignore the threshold here and manually publish are you?  tsk tsk17:24
mterryWhere's that ibus fix?  That should drop the tests nicely17:24
didrockssil2100: ^17:24
didrocksmterry: well, it's as you wish, you are stack holders :-)17:24
mterryHrm, not reviewed yet.  sil2100 has it waiting17:25
mterryfor review that is17:25
didrocksmterry: just that knowing how all those things are fuzzy…17:25
didrocksthen you decide with cyphermox if you publish indicators/unity ;)17:25
mterrydidrocks, a reason to drive them to perfection!  :)  You mentioned a reason you wanted a publish checkpoint?  HUD stuff?17:25
didrocksmterry: yeah, I wanted a before, and then after HUD17:26
didrocksas we have the HUD transition17:26
didrocksand the day after that libcolumbus17:26
didrocksmterry: it's really as you feel it, maybe we shouldn't push the trigger, yeah17:27
cyphermoxyou're talking about unity specifically there right?17:28
cyphermoxbecause we really should finish up publishing indicators so that we can move forward on hud17:28
mterrycyphermox, yeah unity.  The new test failures all seemed to come from dash tests17:31
didrockscyphermox: if we are sure we don't have dep between indicators and unity publishin, I think we can (I don't see indicators tests failing in the unity suite run)17:31
didrocksmterry: I wonder why the dash failed so much on intel17:31
didrocksstill a little bit on nvidia17:31
mterrydidrocks, cyphermox: Is publishing indicators sufficient to not block hud?17:35
mterrycyphermox, ah, still getting crashes on the indicator tests...  :-/17:36
didrocksmterry: yeah17:36
didrocksit's sufficient17:37
cyphermoxwell what we really need is to ship libcolumbus and all so that we can ship hud; then do the hudectomy in indicator-appmenu, and then republish hud for libcolumbus, IIRC17:37
mterryOK, then I'd just as soon leave unity to its own devices as long as there's no pressing need for that stack specifically17:37
mterrysil2100's ibus branch should get us under threshold again (at which point, we should probably lower the threshold a bit)17:38
didrockscyphermox: so, manual publishing for the indicators?17:56
cyphermoxyeah, let's17:57
cyphermoxaside from that, was libcolumbus ready to be uploaded, added to the stacks and everything?17:57
didrockscyphermox: we need the hud first17:57
didrocksthen, libcolumbus :)17:57
didrocksbut IIRC, libcolumbus is ready17:58
cyphermoxwell, yeah, but I mean, libcolumbus can land now and hud not depend on it immediately17:58
cyphermoxthen it give us time for the MIR in parallel, even if it won't take so much time17:58
=== dandrader is now known as dandrader|afk
didrockscyphermox: well, you need to rerun a rebuild if you do that, before the publishing18:01
cyphermoxlibcolumbus will be in indicators stack right?18:02
cyphermoxI'm doing the publishing now, but then I'll add it to the stack and update18:02
didrockscyphermox: sounds good to me :)18:02
cyphermox./cu2d-run -P indicators18:03
didrockscyphermox: I'm just leaving, but I think we won't have it landing automatically tomorrow as it will be at least in manual publishing mode :)18:03
didrockscyphermox: yep ;)18:03
didrocksdue to the new package18:03
didrocksso don't need to refresh the AA side right away18:03
didrocksI'll do that tomorrow morning18:03
cyphermoxI'll only update the stack later18:03
seb128cyphermox, didrocks: I can NEW review if you want18:03
cyphermoxseb128: ok18:04
didrocksseb128: not the issue, you need to pull on lillypilly18:04
cyphermoxit won't land today anyway18:04
didrocksseb128: for the AA-side filtering18:04
seb128didrocks, I was proposing to review it  once it's in the raring NEW queue18:04
seb128didrocks, I didn't follow what you are doing out of "it's going to be uploaded in the next days" ;-)18:04
didrocksseb128: yeah, it will be marked as a sync :)18:04
seb128oh, right18:05
didrocksseb128: but once cyphermox updated the list, first, to be uploaded to the archive, we need to refresh the filtering18:05
seb128well in any case if NEW review is needed just ping me18:05
didrocksseb128: yeah, but you can do the filter refreshing as well once cyphermox will add it to the list (after this manual publishing round)18:05
didrocksseb128: on lillypilly, just log as archive admin18:05
didrockscd cu2d18:05
didrockscd cupstream2distro18:06
didrocksbzr pull18:06
didrocksyou should see the updated yaml file once cyphermox did it :)18:06
seb128didrocks, ok, cool, good to know ;-)18:06
* seb128 tomboys that18:06
didrocksseb128: https://wiki.ubuntu.com/DailyRelease/FAQ#Adding.2BAC8-removing_components_to_a_stack18:11
didrockssecond part :)18:11
seb128didrocks, ahah, excellent ;-)18:12
didrockshot from the press! :)18:12
=== dandrader|afk is now known as dandrader
mterrysil2100, did someone mention that last night intel had 10 new dash failures?
mterrysil2100, do you know why that might be?20:01
sil2100mterry: checking20:19
mterrysil2100, thanks for rocking that ibus merge proposal too20:20
mterryI'm looking forward to that landing20:20
sil2100mterry: np! Hope it fixes the problems for good20:20
sil2100mterry: those failing tests... they seem to be related to the previous failing tests21:05
sil2100Not sure why now they started failing21:06
sil2100But the reason seems to be a problem while fetching the state of the preview open, I'm already looking into it regarding the previous failures21:06
sil2100mterry: if anything, I'm currently looking at two failure types (so that we don't step on eachothers toes ;) )21:06
sil2100The first thing: the dash previews having problems21:07
sil2100The second: HUD focus issues21:07
mterrysil2100, awesome, thanks!21:10
mterryOne day we'll have no failures!21:10
=== salem_ is now known as _salem
sil2100thomi: hi! Once you're up and running, could you take a look on two of my autopilot merges for lp:unity?22:26
sil2100thomi: one regarding ibus, the second regarding a new test suite (for search)22:27
sil2100mterry: just to make sure - the testing environment for the release test jobs is single-monitor, right?22:28
thomisil2100: sure22:44
* thomi looks now22:44
sil2100\o/ Thanks :)22:44
=== _salem is now known as salem_

Generated by irclog2html.py 2.7 by Marius Gedminas - find it at mg.pov.lt!