[10:47] <ogra> lamont, sigh ... arcorn is dead once again
[10:48] <ogra> i dont get why it suddenly fails so often
[11:27] <lamont> ogra: ditto for hawthorn, it appears
[11:27] <lamont> interestingly, it seems to be the same machines each time.
[11:28] <ogra> yes
[11:28] <lamont> one could almost suspect that it's not exactly workload related
[11:28] <ogra> and i have a feeling that its either related to dove builds or to kubuntu ... it seems to have started when NCommander enabled either
[11:28] <ogra> and they were enabled around the same time
[11:29] <ogra> sycamore is rather rarely affected and doesnt build dove or kubuntu images atm
[11:29] <persia> Do we know when it crashes, or what it was doing when it crashed?
[11:29]  * ogra doesnt, i dont get a long and usually only notice it once the machine is gone 
[11:30] <ogra> s/long/log/
[11:32]  * NCommander runs
[11:32] <NCommander> Its not my fault it broke! :-P
[11:34] <lamont> ogra: acorn is up...
[11:34] <ogra> thanks
[11:34] <lamont> heh.  up 33 minutes
[11:34] <ogra> oh
[11:34] <lamont> I suspect someone cycled it
[11:35] <ogra> i tried a build about 1h ago :)
[11:38] <lamont> persia: sadly, it is opaque in that regard
[11:39] <lamont> we know that it does not spew upon the serial console during its death throws
[11:39] <NCommander> lamont: you have console debug messages on?
[11:39] <lamont> NCommander: and I stuff what where to get that?
[11:40] <NCommander> lamont: huh?
[11:41] <persia> syslog config, I'd think.
[11:41]  * NCommander kicks stuff
 lamont: you have console debug messages on? <-- how turn on console debug messages?  IO
[11:44] <lamont> I'm betting either printk foolery in sysctl.conf
[11:45] <NCommander> lamont: remove quiet from the command line and add verbose ;-)
[11:45] <persia> Oh, heh.
[11:54] <lamont> heh. ok
[11:54] <lamont> NCommander: and then reboot, I bet. :-p
[11:55] <lamont> I'll do that later today
[11:55] <NCommander> lamont: acord is omap4, right?
[11:55] <lamont> they're all bbg3
[11:55] <lamont> we have nothing else online
[11:57] <NCommander> lamont: ah, acorn realized it was building the competition and committed suidice!
[11:57] <ogra> NCommander, acorn is omap3 and dove
[11:57] <ogra> sycamore is omap4
[11:57] <ogra> (target wise)
[11:58] <ogra> *and* we dont build kubuntu for omap4 yet
[11:58] <ogra> so acron is omap3/dove for ubuntu-netbook and kubuntu currently
[11:58] <ogra> dove and kubuntu were both enabled at a similar time
[12:00]  * ScottK would say that at work Kubuntu or Dove exposed an existing problem.  I don't they either actually "caused" a problem.
[12:00] <ogra> yeah, i didnt say they case one :)
[12:00] <ogra> *cause
[12:01] <ogra> we just dont see it on sycamore which onyl builds ubuntu-netbook-omap4 currently, it might as well be just a load issue or something
[12:02] <lamont> except that they all serialize, so you're only ever building one
[12:02] <lamont> I suppose you could trade roles and see if sycamore starts faceplanting
[12:03] <ogra> i could do that but we have an agreement to keep one buildd exclusively for omap4
[12:04] <ogra> though i dont really expect it to be caused by dove ubuntu builds they are 98% identical to omap3 or 4 (just different kernel packages being unpacked)
[12:05] <ogra> that shouldnt cause more load or anything, whereas kubuntu builds install a completely different set of packages
[12:07] <ogra> it would be good to know the exact time it dies
[12:07] <ogra> then we could easily verify against the crontab
[12:11] <lamont> 1 hr 9 min 20 seconds ago
[12:11] <lamont> or as much as 5 min before that..
[12:19] <ogra> very very weird
[12:20] <ogra> it should have been idlin at that time
[12:20] <ogra> *idling
[12:20] <ogra> 55 1 * * *      ARCHES='armel+omap armel+omap4' OPTIONS="-f ext3" buildlive ubuntu-netbook && ARCHES='armel+omap armel+omap4' for-project ubuntu-netbook cron.ports_daily-preinstalled
[12:20] <ogra> 13 3 * * *      ARCHES='armel+omap armel+omap4' OPTIONS="-f ext3" buildlive kubuntu && ARCHES='armel+omap armel+omap4' for-project kubuntu cron.ports_daily-preinstalled
[12:21] <ogra> oh, my copy was outdated
[12:21] <ogra> there is 45 6 * * *      ARCHES='armel+omap armel+omap4' OPTIONS="-f ext3" buildlive kubuntu-mobile && ARCHES='armel+omap armel+omap4' for-project kubuntu-mobile cron.ports_daily-preinstalled
[12:22] <ogra> NCommander, why the heck are we building all these omap4 targets ?
[12:22] <ogra> (i'm pretty sure i asked you not to)
[12:24] <ogra> bah
[12:24] <ogra> the crontab in the branch *totally* differs from the actual one
[12:24]  * ogra sighs deeply
[12:25] <ogra> so kubuntu-mobile is actually the last thing we build, starting at 6:45
[12:26] <ogra> and according to http://cdimage.ubuntu.com/kubuntu-mobile/ports/daily-preinstalled/20100908/ they finished today at ~8:00
[12:26] <ogra> so it was actually idle when it died
[12:31] <ogra> lamont, are both systems identical software wise (same cron-daily entries etc) ?
[12:31] <ogra> i wonder if its a system cronjob that makes it die
[12:36] <lamont> as far as I know, yes
[12:44] <lamont> sigh. brainfart.  that was when the machine was rebooted.
[12:44] <lamont> I'll be able to give you a time the next time it dies...
[12:44] <NCommander> ogra: ?, how does it diff, last time I updated the cron, it matched bzr 100%
[21:41] <ScottK> I'd appreciate it if someone else on the release team would have a look at Bug #631413.  They are reasonably ready to push an upload and IME it's demonstrably better than what we have now, but I'm hardly dispassionate on the question.
[21:41] <ubot4> Launchpad bug 631413 in mesa (Ubuntu) "[FFe] Mesa 7.9 (affects: 4) (heat: 26)" [Undecided,New] https://launchpad.net/bugs/631413