[08:52] <apw> smb this thinkpad brightness/volume thing is all over the map ... most processes seems to get a hand in the handling of a simple screen update ... 
[08:53] <smb> That was to be feared
[08:53] <apw> i think it has reached a level of complexity that a wiki page is required to contain the information
[08:53] <smb> So the entry is the key event, who comes next? hal probably
[08:53] <apw> well the key is generated by hal as its a fake key
[08:53] <smb> Hm, this sounds definitely like an asset
[08:54] <apw> then it goes to either power-manager or settings-daemon depending which it is
[08:54] <apw> then that talks to hal to change things and also send out a notification
[08:56] <smb> I assume power-manager for brightness and settings-daemon for sound
[08:56] <apw> yeah
[08:56] <apw> now brightness does have a passive mode already though it doesn't seem to work quite right
[08:56] <smb> Those are those things that would benefit from a picture...
[08:56] <apw> (passive being a 'just display the currentlt level on keypress')
[08:57] <smb> Ok, that is one step. So _whenever_ that works, something similar would be required for sound
[08:57] <apw> volume does nto as yet
[08:57] <apw> yeah ... something like that
[08:57] <apw> and that is basically what the maintainer is talking about ... sort of
[08:57] <apw> i guess i need to just dump it all out
[08:58] <smb> Yeah, that would be helpful, then print it and confront the right people at UDS
[08:59] <apw> heh yeah
[11:25] <apw> bah have spent too much time thinking about this on my own
[11:25] <apw> https://wiki.ubuntu.com/ThinkpadBrightnessVolume
[11:38] <cking> apw: wassup?
[11:39] <apw> just getting sucked into overanalysis of a problem
[11:39] <apw> was slapping self to get it documented and move on
[11:39] <cking> good wiki page by the way
[11:41] <ogra> just send a mail to evolution on each event ... then you get a mail notification at least :P
[11:42]  * ogra agress its a very good research 
[11:42] <ogra> *agrees
[11:45] <ogra> apw, are you sure we will even use the alsamixer in karmic ? i would suspect us to migrate to pavucontrol at some point anyway (so it would completely be in software and on the highest level)
[11:46] <apw> ogra, no idea really.  if so someone else can figure out how to make pulse work with a hardware mixer :)
[11:46] <ogra> you cant, but you could do the same you do for brightness more easily with it i think
[13:05] <Riddell> linux dudes: what have you done with /usr/include/asm/errno.h from linux-libc-dev ?
[13:05] <amitk> Riddell: karmic?
[13:05] <Riddell> yes
[13:06]  * amitk looks at rtg ^
[13:06] <Riddell> kde4libs not happy "/usr/include/linux/errno.h:4:23: error: asm/errno.h: No such file or directory"
[13:08] <rtg> Riddell: start a bug and assign me to it. I've a conf call coming up in a bit, so I won't get a chance to look at it until after that.
[13:19] <Riddell> bug 373214
[13:19] <ubot3> Malone bug 373214 in linux "errno.h missing" [Undecided,New] https://launchpad.net/bugs/373214
[13:22] <elmo> Riddell: which architecture(s)?
[13:23] <Riddell> elmo: i386 and amd64 build failures have arrived in my inbox so far
[13:30] <NCommander> rtg, /usr/include/asm/* is missing (we ran into this problem on powerpc; obviously this isn't a ports specific issue anymore :-/)
[13:31] <rtg> apw or amitk: do you have time to look at bug 373214? I'm booked for the next bit, and it appears to be causing wide spread panic.
[13:31] <ubot3> Malone bug 373214 in linux "errno.h missing" [Undecided,New] https://launchpad.net/bugs/373214
[13:33] <rtg> NCommander: if its the same bug, its affecting kde4libs as well.
[13:36] <amitk> rtg: looking it up now..
[13:36] <NCommander> rtg, well, the problem is /usr/include/asm is empty which breaks anything trying to use kernel headers
[13:37] <NCommander> rtg, it looked like something managed the package in the buildd (neither myself nor TheMuso have successfully reproduced this issue, nor could we in a jaunty PPA)
[13:37] <rtg> NCommander: which would certainly explain '/usr/include/asm/error.h is missing from linux-libc-dev in linux_2.6.30-3.4' as described in the bug report.
[13:37] <NCommander> rtg, I just checked all the architecture linux-libc-dev
[13:37] <NCommander> its empty on i386, amd64, lpia, armel, and powerpc
[13:38] <NCommander> ia64 and hppa are ok, SPARC is current FTBFS
[13:38] <NCommander> *currently
[13:39] <amitk> rtg: so you just uploaded meta yesterday and the karmic kernel started becoming available?
[13:40] <amitk> ...in the archive, that is
[13:40] <rtg> amitk: yep, -2 is the first ABI that is referenced by linux-meta, but I uploaded -3 yesterday
[13:55] <NCommander> rtg, the broken header packages make it impossible to build linux and linux-ports in a karmic chroot ...
[13:56] <amitk> NCommander: could you put all the info in bug 373214
[13:56] <ubot3> Malone bug 373214 in linux-ports "/usr/include/asm/* is not present in linux-libc-dev" [Critical,In progress] https://launchpad.net/bugs/373214
[13:56] <NCommander> amitk, already there
[13:56] <NCommander> amitk, I'm pulling the buildd chroot, and fudge it as necessary to get the kernel building again to see if I can see what went wrong
[14:03] <amitk> hmm.. after all the headers moves in .28 and .29, there is no include/asm-x86 
[14:03] <amitk> or rather all asm-<arch> has been moved
[14:05] <amitk> rtg: all include/asm-<arch> was moved to arch/<arch>/include/asm
[14:06] <amitk> NCommander: ^
[14:07] <amitk> essentially all include/asm-<arch> was 'moved' to arch/<arch>/include/asm
[14:08] <amitk> but arch/<arch>/include/asm is quite a bit larger than what include/asm used to be, so I am still studying it
[14:08] <NCommander> amitk, but isn't that how it was for 28 as well? (I have a include/asm here)
[14:09] <NCommander> amitk, I need to run to the DMV, I'll be back in ~45-60 minutes to continue debugging this w/ you
[14:09] <amitk> NCommander: it was partly done for .28. If you notice there is no include/asm-x86 in 2.6.28. I am checking if something was being fudged in the scripts
[14:10] <NCommander> amitk, not to my knowledge, but I may have overlooked something
[14:11] <cjwatson> shouldn't 'make headers_install' just work despite the rearrangements though?
[14:11] <cjwatson> the build log says 'INSTALL include/asm (52 files)'
[14:17] <ikepanhc> smb: ask a question, if I need the bug reporter to help me test the patch, shall I add some extra version string when make kernel package?
[14:19] <cjwatson> hmm, that's interesting, 'fakeroot debian/rules install-arch-headers' on jaunty seems to install a reasonable include/asm/
[14:19] <cjwatson> amitk: ^-
[14:20] <amitk> cjwatson: on the karmic tree?
[14:21]  * amitk tries it
[14:23] <amitk> cjwatson: curious indeed
[14:23] <cjwatson> amitk: yeah
[14:24] <cjwatson> hmm, and even in a karmic chroot
[14:24] <cjwatson> maybe that test is inadequate somehow
[14:26]  * cjwatson bootstraps his own chroot rather than using ronne's
[14:29] <amitk> hmm.. why is the i386 build passing ARCH=x86_64?
[15:01] <amitk> cjwatson: I don't get it. The buildds are not behaving like my local machine. fakeroot debian/rules binary-arch-headers works perfectly here...
[15:02] <cjwatson> just to save time, it's astonishingly rare for problems to actually be due to a genuine difference on the buildds
[15:02] <cjwatson> it does happen but not typically in the *middle* of a build like this ...
[15:03] <cjwatson> might be worth trying to match up package versions with those in the -3.4 build log?
[15:06] <cafetiere> apw switches to another machine for testing
[15:21] <cjwatson> amitk: no dice for me in a chroot with linux-libc-dev 2.6.30-2.3 either
[15:24] <amitk> cjwatson: I'm doing a full build here
[15:24]  * cjwatson tries downgrading gcc and friends to the versions in the build log
[15:26] <NCommander> amitk, any headway? (I see you confirmed that local builds do work correctly)
[15:26] <cjwatson> has anyone tried a PPA build?
[15:26] <NCommander> cjwatson, yes, on ports, works fine in the jaunty series
[15:26] <NCommander> no karmic PPAs yet
[15:26] <cjwatson> oh, of course, it breaks due to the new linux-libc-dev anyway
[15:27] <NCommander> cjwatson, I have a buildd amd64 chroot now, so I'm seeing if I can find what's causing the package to explode
[15:28] <cjwatson> this is weirding me the hell out
[15:28]  * NCommander notes this doesn't seem possible.
[15:44] <rtg> awe: sent email re: Broadcom update. I think I've got the 2 patches you mentioned. It at least builds correctly.
[15:46] <awe> rtg: thanks... i'll look at it in a little while.  i'm fighting to get voip configured so i can have a conf call w/asac.
[15:46] <awe> rtg: no more hp for me for now.  ;)
[15:46] <awe> rtg: i'm working as part of desktop for karmic.
[15:47] <awe> rtg: although i'll still work with y'all wrt to the oem kernel
[15:47] <rtg> awe: do you want me to ripple the BCM update up through Intrepid/Jaunty ?
[15:47] <awe> rtg: nah, i'll do it.  it's good practice for moi
[15:47] <rtg> awe: k
[15:49] <awe> course i may ask you to endorse my ubuntu-contributors MOTO app page in return.  ;)
[15:50] <rtg> awe: np
[15:50] <awe> cool
[15:50] <awe> i've worked more with y'all then anyone else to date...
[16:05] <cjwatson> amitk: how did the full build go?
[16:06] <amitk> cjwatson: still building. :-/ I don't have the fastest machine...
[16:06]  * NCommander waits for his build to complete
[16:07] <rtg> amitk: what package are you building?
[16:09] <calc> so the linux-libc-dev headers issue will be fixed tomorrow?
[16:10] <amitk> rtg: the kernel
[16:10]  * calc thinks it may be the cause of some of his OOo build failures atm
[16:10] <amitk> calc: we are looking at it
[16:10] <calc> ok
[16:11] <amitk> rtg: local builds seem to do the right thing (fakeroot debian/rules binary-arch-headers), I am trying a full build.
[16:12] <rtg> amitk: do the right thing with respect to what? I unpacked linux-libc-dev_2.6.30-3.4_amd64.deb and indeed there is nothing in the asm directory. Looks like errno.h got moved to asm-generic
[16:14] <cjwatson> rtg: userspace expects <asm/errno.h> to exist
[16:14] <rtg> cjwatson: so, this worked until I published linux-meta?
[16:14] <cjwatson> no, linux-libc-dev 2.6.30-3.4 inexplicably broke it
[16:15] <cjwatson> -2.3 had <asm/*.h> as one might expect
[16:15] <cjwatson> BTW errno.h was always in asm-generic as well (or at least for ages)
[16:15] <cjwatson> so not a move, it's just that asm/ vanished inexplicably
[16:15] <rtg> cjwatson: how is that so? There were no build rule changes between 2.3 and 3.4, nor were there any core linux updates.
[16:16] <cjwatson> that's what we'd all like to know too
[16:16] <cjwatson> nevertheless, it is true :-(
[16:16] <amitk> rtg: right thing meaning it has the .h files under include/asm
[16:16] <rtg> hmm, I'll rebuild -2.3 from scratch. should only take a few minutes
[16:17] <cjwatson> there's quite a bit of real stuff under asm/ BTW - errno.h is a bad example because it's just a shim
[16:18] <cjwatson> rtg: the build log is very strange - it says "INSTALL include/asm (52 files)" but then the verbose output from cpio -pvd says that there's nothing in include/asm/
[16:19] <amitk> right, and if you do it locally, it all works.
[16:19]  * cjwatson tries diffing the build logs
[16:20] <rtg> amitk: I don't get that. my asm directory is empty
[16:20] <cjwatson> ooh, you can reproduce it
[16:20] <cjwatson> don't suppose you could try with V=1 added to hmake?
[16:20] <amitk> rtg: with 'fdr binary-arch-headers'?
[16:21] <rtg> cjwatson: I think something changed from Linus tree that I missed. I'm pretty sure there was a rebase somewhere between 2.3 and 3.4.
[16:21] <cjwatson> rtg: I'm not entirely sure I believe that this is a kernel change
[16:22] <cjwatson> I'm wondering if it's collateral damage from something else
[16:22] <cjwatson> rtg: did you reproduce it with -2.3 or with -3.4?
[16:23] <amitk> rtg: the move of all arch headers from include/asm-<arch> to arch/<arch>/include/asm has been going on since 2.6.28. I believe it is now complete. But 'make headers_install' basically shields us from the underlying changes. (Or does it?)
[16:24] <cjwatson> the build log diff shows newer versions of coreutils, findutils, login and passwd, gcc-4.4 and friends, libselinux1, libsepol1, libvolume-id1, and cpio, as well as a bunch of other build-deps further up the tree
[16:24] <rtg> amitk: using 3.4, the asm directory _does_ have all of the headers when built locally using 'debian/rules binary-arch-headers'. But I just unpacked the result of 'debuild -b' and saw that it did _not_ have stuff in the asm directory. lemme rebuild from scratch just to confirm.
[16:25] <cjwatson> you'll need linux-libc-dev <= 2.6.30-2.3 installed in order to rebuild successfully obviously :-)
[16:26] <rtg> cjwatson: not if I'm building the kernel 'cause it produces linux-libc-dev
[16:26] <cjwatson> the kernel build builds some host binaries as part of its build process, no?
[16:27] <rtg> cjwatson: uh, yeah. seems kind of circular.
[16:27] <NCommander> cjwatson, indeed :-/
[16:27] <cjwatson> NCommander said it failed at scripts/basic/fixdep
[16:27] <cjwatson> so downgrade, then test
[16:27] <rtg> and in fact my build is borked after updating my chroot.
[16:27] <NCommander> cjwatson, any idea how we're going to unbreak the archive?
[16:28] <cjwatson> NCommander: infinity, or in his absence another sysadmin, can do manual builds)
[16:28] <cjwatson> we've done this in the past
[16:28] <cjwatson> this is not the first time that the process of bootstrapping a new release has left the entire world unbuildable :-)
[16:30] <cjwatson> no obvious sign of trouble in the build log diff
[16:30] <NCommander> cjwatson, indeed, although this is the first time I've ever seen a kernel upload break so oddly 
[16:32] <cjwatson> this is doing well at being truly weird, but nothing much really surprises me any more :-)
[16:32] <NCommander> cjwatson, well, I dumped a buildd chroot into sbuild, so if its something funny with pkgbinmangler or something else there, this will shake it out
[16:32] <NCommander> I just wish I had a faster machine
[16:33] <rtg> cjwatson: I downgraded to linux-libc-dev_2.6.28-12.43 and things are at least building now. it'll be done in 10 minutes or so.
[16:34] <cjwatson> NCommander: nothing's impossible, but I'd be truly astonished if any of the pkg* things were capable of affecting this. They just don't hook in at the relevant levels
[16:34] <NCommander> rtg, you can build a kernel in 10 minutes? 
[16:34] <rtg> NCommander: one arch
[16:34] <NCommander> rtg, I envy your build rig
[16:34] <NCommander> cjwatson, well, I'm not sure what else it could be. We built the same package with ports overlay, and only broke powerpc
[16:35] <cjwatson> NCommander: it's obviously something weird, but I bet you 10 euros (collectable at UDS) that it isn't pkg-create-dbgsym or pkgbinarymangler
[16:35] <NCommander> I'll take that bet :-)
[16:36] <NCommander> cjwatson, thanks, you just reminded me I need to turn some USD into Euros 
[16:36] <NCommander> (and to set aside 10 euros for when I loose this bet)
[16:36] <rtg> I thought it was clear this issue is cause by linux-libc-dev_2.6.30-2.3 ? am I confused ?
[16:37] <cjwatson> rtg: how? :-)
[16:37] <rtg> cjwatson: empty asm directory?
[16:37] <cjwatson> linux-libc-dev_2.6.30-3.4 is the one with the empty asm directory
[16:37] <cjwatson> but the question is *why* it has an empty asm directory
[16:38] <cjwatson> the evidence from the build log suggests that it is trying to install those files, but getting lost somewhere along the way
[16:38] <rtg> ah, then I was a little confused
[16:39] <cjwatson> since when people build the same source package locally, it seems to come out with /usr/include/asm/ populated as normal
[16:39] <NCommander> cjwatson, as a counter example, I just built it in pbuilder in a fresh chroot, I got the /usr/include/asm folder normally
[16:39] <NCommander> Oh
[16:39] <NCommander> :-)
[16:40] <cjwatson>   perl /home/cjwatson/src/ubuntu/linux/ubuntu/scripts/headers_install.pl /home/cjwatson/src/ubuntu/linux/ubuntu/arch/x86/include/asm /home/cjwatson/src/ubuntu/linux/ubuntu/debian/tmp-headers/install/include/asm x86 a.out.h auxvec.h boot.h bootparam.h byteorder.h debugreg.h e820.h errno.h fcntl.h ioctl.h ioctls.h ipcbuf.h ist.h kvm.h ldt.h mce.h mman.h msgbuf.h msr-index.h msr.h mtrr.h param.h poll.h posix_types.h ...
[16:41] <cjwatson> ... posix_types_32.h posix_types_64.h prctl.h processor-flags.h ptrace-abi.h ptrace.h resource.h sembuf.h setup.h shmbuf.h sigcontext.h sigcontext32.h siginfo.h signal.h socket.h sockios.h stat.h statfs.h swab.h termbits.h termios.h types.h ucontext.h unistd.h unistd_32.h unistd_64.h vm86.h vsyscall.h; perl /home/cjwatson/src/ubuntu/linux/ubuntu/scripts/headers_install.pl ...
[16:41] <cjwatson> ... /home/cjwatson/src/ubuntu/linux/ubuntu/debian/tmp-headers/arch/x86/include/asm /home/cjwatson/src/ubuntu/linux/ubuntu/debian/tmp-headers/install/include/asm x86 ; touch /home/cjwatson/src/ubuntu/linux/ubuntu/debian/tmp-headers/install/include/asm/.install
[16:41] <cjwatson> is the relevant install command - which is clearly getting run at least with the correct number of arguments
[16:41] <cjwatson> ("clearly" due to "52 files" in the build log)
[16:45] <amitk> local build finished successfully with include/asm populated
[16:45] <cjwatson> NCommander: where did the broken powerpc build happen?
[16:45] <NCommander> cjwatson, on the buildds. Same issue. The same package built in jaunty had no issue
[16:45] <cjwatson> this is almost seeming like a transient bug in find or cpio
[16:45] <cjwatson> anyway, dinner
[16:45] <rtg> cjwatson: well, we just found one in 'sort' a couple of days ago
[16:48] <rtg> amitk: I just synced my chroot 4 hours ago and my local build exhibits the problem.
[16:49] <rtg> damn, I wish I'd have made a full log of the build. guess I'll have to do it again.
[17:00] <amitk> i haven't synced by chroot yet, I will do so after dinner
[17:19] <rtg> amitk: I'm thinking this a perl problem. 'perl /home/rtg/kern/karmic/kern-64/ubuntu-karmic/scripts/headers_install.pl' appears to be doing the right thing, and that script hasn't been touched since Dec 2008.
[17:26] <NCommander> rtg, that script takes an input directory and copies it somewhere else removing compiler.h definitions
[17:26] <NCommander> rtg, I don't think it was affected by headers moving around :-/
[17:27] <NCommander> (nor does it explain why we only get funny failures on the buildds)
[17:27] <rtg> NCommander: I'm getting failures locally.
[17:27] <NCommander> rtg, you are?
[17:27] <NCommander> How?
[17:27] <rtg> sync your mirror
[17:28] <NCommander> rtg, you mean update my chroot?
[17:28] <rtg> yes
[17:28] <NCommander> rtg, I get the normal build failure because of the bad linux-libc-dev, but so far, I can build a good package
[17:29] <rtg> downgrade linux-libc-dev to -2.3, then try to build the install-arch-headers target for the kernel
[17:33] <rtg> NCommander: hmm, actually the issue is with 'find'
[17:34] <NCommander> rtg, ?
[17:37] <NCommander> find was merged May 4th
[17:38] <NCommander> 2.6.30-3.4 was uploaded on the 1st
[17:38] <rtg> NCommander: I updated the bug report
[17:38] <NCommander> linux-ports 2.6.30-1.1 was uploaded yesterday; same issue
[17:40] <rtg> NCommander: so the timing makes sense
[17:40] <NCommander> rtg, er, the kernel upload was before find was updated
[17:40] <NCommander> Very, very odd
[17:40] <NCommander> and TheMuso reported he built in a karmic chroot and reproduced the issue
[17:40] <NCommander> But this is the best candiate I've seen thus far.
[17:41] <NCommander> cjwatson, I owe you 10 dollars.
[17:41] <NCommander> s/dollars/euros
[18:02] <rtg> NCommander: the upload date for linux-libc-dev_2.6.30-2.3 in LP doesn't make sense. I created the Ubuntu-2.6.30-2.3 log entry on April 30, and I'm positive I uploaded the same day.
[18:02] <calc> don't forget the archive was frozen up until some point
[18:03] <rtg> which would explain by 2.3 built correctly and 3.4 did not (because it was after findutils was synced)
[18:03] <calc> ah that was apr 28 when it opened
[18:03] <NCommander> rtg, but we have kernel builds on ia64 and hppa which happened after the sync
[18:03] <NCommander> (I suspect the find bug is our issue, but its damn odd)
[18:04] <rtg> NCommander: dunno, perhaps there was race with findutils
[18:04] <NCommander> rtg, checked the upload dates, didn't seem so
[18:04] <rtg> was a*
[18:04] <rtg> NCommander: well, at any rate, I can absolutely reproduce the problem by hand.
[18:04] <calc> make sure you check build times of eg findutils vs the kernels since there were buildd backlogs too
[18:04] <NCommander> rtg, I'm still waiting for my kernel build to finish
[18:05] <NCommander> calc, I check the time of upload fromt he buildds
[18:05] <calc> iirc the buildd queues were fairly large at one point
[18:05] <calc> ok
[18:05] <NCommander> calc, find is Priority: required, so it will go fairly fast
[18:05] <NCommander> er
[18:05] <NCommander> Or is it important
[18:06] <calc> required
[18:08] <calc> do packages on buildds automatically get updated that are eg required?
[18:09] <calc> or only if something build-dep against a specific version (iow could some of the buildds still had the older working findutils)
[18:10] <calc> ^ that wrt the comment NCommander made about hppa/ia64
[18:10] <NCommander> calc, it just has to be in main
[18:10] <rtg> NCommander: if I downgrade to findutils_4.4.0-2ubuntu4 then all works as expected.
[18:10] <rtg> seems fairly conclusive.
[18:11] <calc> NCommander: so ubuntu buildds automatically do an upgrade daily (or something like that?)
[18:11] <NCommander> calc, I think dist-upgrade is run as part of sbuild, but thats a voodo that I know not
[18:11] <NCommander> rtg, indeed, now we need to rundown what broke in find
[18:11] <rtg> calc: IIRC the xen instances are created from scratch for every package
[18:11]  * NCommander wonders how you can break find :-/
[18:11] <NCommander> rtg, xen only done for PPAs
[18:11] <calc> rtg: oh
[18:11] <rtg> NCommander: and buildds
[18:12] <NCommander> rtg, Xen doesn't exist for powerpc, armel, ia64, hppa :-/
[18:12] <NCommander> That's why devirtualized PPAs are reserved for Canonical employees only
[18:12]  * NCommander would love xen for any of those archs so then we could get PPAs for them
[18:12] <rtg> but xen is used for x86, which is all _I_ care about most days.
[18:16] <calc> armel is the only really useful arch without xen support ;-)
[18:16] <calc> the rest are dead or dying
[18:17] <cjwatson> NCommander: result
[18:17] <calc> so anyone know what is broken about find now?
[18:17] <cjwatson> though still need a patch; let me try diving into findutils if nobody else already is
[18:17] <calc> oh ok
[18:18] <NCommander> cjwatson, still building, but I think I owe you money now
[18:18] <cjwatson> rtg: what was the sort bug you found?
[18:19] <cjwatson> this is interesting, though; the findutils bug does not manifest for me
[18:19] <rtg> cjwatson: yeah, I've been poking at it but can't figure out why it hates the asm directory
[18:19] <cjwatson> I wonder if it has something to do with path length or similar
[18:20] <rtg> Steve L. fixed the sort bug in coreutils.
[18:20] <cjwatson> the changelog is quite short
[18:20] <NCommander> cjwatson, a path length bug is unlikely; linux is longer than asm
[18:20]  * NCommander wonders/hopes upstream is git
[18:21] <cjwatson> I didn't say it was necessarily longer paths that caused the bug :-)
[18:21] <NCommander> Its a git repo
[18:21] <NCommander> We could just bisect 
[18:21]  * cjwatson remembers a bug in HP-UX somethingorother that manifested when the argument was a multiple of 16 characters long
[18:21] <NCommander> cjwatson, ........
[18:21] <NCommander> cjwatson, thanks, you just caused my brain to ABEND with SIGILL
[18:25] <NCommander> /usr/bin/pkgsanitychecks: inconsistent /CurrentlyBuilding file, Package: value is  (should be linux)
[18:25] <NCommander> :-/
[18:25] <cjwatson> rtg: what architecture is this?
[18:26] <cjwatson> I tried recreating your directory structure precisely, and the asm files show up as one would expect
[18:26] <rtg> cjwatson: amd64
[18:27] <cjwatson> (we know it happens on i386 too, of course)
[18:28] <cjwatson> rtg: could you get an strace -tt -s 1024 of that find command?
[18:28] <rtg> cjwatson: one sec.
[18:31] <cjwatson> there really aren't many relevant changes to find itself, but there's a gnulib update in there too
[18:33] <rtg> cjwatson: chinstrap.canonical.com:~rtg/log.txt
[18:33] <cjwatson> NCommander: if it's a bug in imported gnulib files, git bisect won't help, since the findutils maintainers don't commit gnulib files to git
[18:34] <NCommander> cjwatson, *grumble :-/* This will be fun.
[18:34] <cjwatson> rtg: thanks, working on that
[18:37] <cjwatson> hmm
[18:38] <cjwatson> I have a hypothesis, which MAY BE WRONG
[18:38] <rtg> isn't that why its a hypothesis?
[18:39] <cjwatson> I'm betting that the newer code in gnulib is attempting to rely on features (specifically some of the *at functions) not present, or perhaps not reliable, in the kernel running on the buildds
[18:39] <cjwatson> rtg: what kernel version is running on your test system?
[18:39] <rtg> 2.6.30-3-server
[18:39] <cjwatson> drat.
[18:39] <NCommander> so much for that hypothesis
[18:40] <cjwatson> not clearly disproven yet, it could just be that Tim is running into yet a different problem from the buildds ;-)
[18:40] <rtg> the buildds are typically 2.6.24, right?
[18:40] <cjwatson> I forget, something like that
[18:40] <cjwatson> rtg: also, what filesystem?
[18:41] <rtg> ext4
[18:41] <NCommander> cjwatson, powerpc is a dapper (.18 kernel), the rest are hardy
[18:42] <cjwatson> dapper was .15, but whatever ...
[18:42] <NCommander> oh, man, my memory going
[18:43] <cjwatson> .8 .10 .12 .15 .17 .20 .22 .24 .27 .28 .30. IIRC.
[18:44]  * NCommander envies cjwatson's memory
[18:44] <alex_joni> NCommander: he probably remembers the hash
[18:44] <rtg> cjwatson: don't get too comfortable with .30, that may yet change.
[18:45] <NCommander> alex_joni, that would be scary
[18:45] <cjwatson> rtg: yeah :)
[18:45] <cjwatson> NCommander: nah, just if you work with something for six months at a time it tends to settle in long-term memory
[18:45] <NCommander> cjwatson, my kernel build is finishing, I'll have another datapoint for you on this
[18:46]  * NCommander is trying to determine what powerpc, armel, amd64, i386 and lpia have in common that ia64 and hppa don't.
[18:54] <NCommander> bbiab, lunch
[18:55] <cjwatson> sorry, I have a fractious baby here and I think I'll need to feed her before attempting to dig into findutils any more
[19:19] <cjwatson> rtg: could you try this to get a bit more information out of find?  strace -tt -s 1024 find -D tree,search,stat,rates,opt,exec ...
[19:19] <cjwatson> (I'm just hitting it with a bigger hammer in the hope of extracting something useful about its logic)
[19:23] <rtg> cjwatson: one sec
[19:27] <rtg> cjwatson: chinstrap.canonical.com:~rtg/log2.txt
[19:33] <cjwatson> rtg: thanks. argh. no visible difference between asm (breaks) and drm (works).
[19:33] <cjwatson> rtg: I don't suppose I could get an account on this system?
[19:33] <cjwatson> I'm going to need to gdb this
[19:34] <rtg> cjwatson: gimme a bit, its pretty well buried behind a firewall.
[19:43] <cjwatson> phew, baby is asleep, touch wood
[19:54] <cjwatson> rtg: ok, now I'm *really* confused
[19:55] <rtg> cjwatson: dchroot -c karmic-amd64 ?
[19:55] <cjwatson> cjwatson@tyler-b:/home/rtg/kern/karmic/kern-64/ubuntu-karmic/debian/tmp-headers/install/include$ find . -name . -o -name .\* -prune -o -print | grep asm/
[19:55] <cjwatson> ./asm/bootparam.h
[19:55] <cjwatson> oh, whoops :-)
[19:55] <rtg> its a Jaunty box by default
[19:56] <cjwatson> I have no user in the chroot, but maybe that doesn't matter
[19:56] <rtg> cjwatson: do it as root, though I think chroots are bind mounted to /home
[19:57] <cjwatson> ok
[19:57] <cjwatson> reproduced, anyway - going to build myself a test findutils
[19:59] <cjwatson> installing gdb in the chroot
[20:09] <cjwatson> aha, the bug is governed by the presence of the file ..install.cmd
[20:09] <cjwatson> now to determine *why*, although the form of that name permits a good guess
[20:12] <rtg> cjwatson: kind of a strange file name. I din't think it was even legal
[20:13] <cjwatson> file names can contain anything except \0
[20:14] <cjwatson> (yes, even \n, though expect a lot of breakage if you do that!)
[20:14] <rtg> cjwatson: are you thinking the '..' is suspicious?
[20:14] <cjwatson> yes, I'm thinking that find is misinterpreting it as a change of directory level
[20:14] <cjwatson> find is pretty twisty though so it's going to take me a little while to find out exactly where
[20:14] <rtg> cjwatson: why just that directory? there are a bunch of those file names.
[20:15] <rtg> though we might not even notice if those other directories didn't get copied
[20:15] <cjwatson> maybe it needs to be at the top level or something? I'm not sure yet
[20:16] <cjwatson> again, just a hypothesis so far
[20:16] <cjwatson> but I can reproduce the bug on my own system if I touch include/..install.cmd
[20:39] <cjwatson> So. What's happening is that ..install.cmd matches .* and therefore triggers the -prune action. Now, normally -prune is a no-op for files. Unfortunately, in this case the stat buffer find is using is undefined (because it uses FTS_NOSTAT as an optimisation) and so happens to contain stat information from the previous thing it touched, which happened to be a directory; it therefore gets confused and thinks ..install.cmd ...
[20:39] <cjwatson> ... is a directory so sets the "stop_at_current_level" flag. This doesn't get cleared until the next time it exits a directory on its walk, which happens to be when it exits asm.
[20:41] <cjwatson> I'll take this up with upstream, but should be able to construct a band-aid
[20:41] <rtg> cjwatson: I have to wonder why anyone is messing with find to begin with. its so dang old that you'd think all of the bugs had been wrung out of  it. why fix what ain't broken?
[20:43] <cjwatson> 'cos all software still has bugs :-)
[20:43] <rtg> cjwatson: are you done with my server? I normally power it off when not in use as it is kinda noisy.
[20:43] <cjwatson> yeah, thanks
[20:43] <rtg> np
[20:44] <cjwatson> I'll have a go at constructing a minimal test case for findutils upstream
[20:44] <cjwatson> one of the bits of software I work on in my spare time (ho ho) is man-db. man is also ancient. When I took it on it was a buggy pile of shite ...
[21:37] <cjwatson> I think this will fix it but will obviously test:
[21:38] <cjwatson> http://paste.ubuntu.com/166326/
[21:55] <cjwatson> rtg: can I suggest that we should reopen 373214 against the kernel? even though it isn't actually a bug in the kernel, the bug won't in practice be fixed until the kernel is reuploaded
[22:05] <NCommander> cjwatson, I managed to reproduce it, I can give you a shell on my laptop
[22:18] <cjwatson> NCommander: no need, I've got it all now :)
[22:20] <NCommander> cjwatson, +1, and +10 euros to you at UDS
[22:23] <cjwatson> rtg,NCommander: can you guys upload new linux/linux-ports, please, so that it can build against the new findutils when lamont is ready? it might fail to build first time round but don't worry about that
[22:23] <cjwatson> s/might/will/ I guess
[22:23] <NCommander> cjwatson, I need a sponsor
[22:24] <cjwatson> rtg: can you sponsor NCommander? I need to crash ...
[22:24]  * TheMuso is around
[22:24] <NCommander> TheMuso, oh, cool. I need a SPARC patch to go in to fix that FTBFS, but I need to go poof until about 00-01 UTC
[22:25] <TheMuso> NCommander: right
[22:29] <lamont> NCommander: (et al) if you do the upload, pretty please make it build-depend the new findutils
[22:30] <lamont> I don't care if it fails to build, but I'll make you upload again if it succeeds and doesn't have include/asm :-)
[22:30] <cjwatson> err, I think it's just a Build-Conflicts
[22:30] <cjwatson> Build-Conflicts: findutils (= 4.4.1-1ubuntu1)
[22:39] <cjwatson> for anyone who's interested, http://launchpadlibrarian.net/26447089/findutils_4.4.1-1ubuntu1_4.4.1-1ubuntu2.diff.gz is the basic fix; a more complete patch with a test case is on its way upstream