[07:52] <dholbach> good morning
[08:09] <ion> hi
[09:12] <ev> pitti, bdmurray: https://wiki.ubuntu.com/ErrorTracker/ReducingRetracerDiskUsage
[09:16] <pitti> Good morning
[09:37] <ev> pitti, mpt: https://docs.google.com/a/canonical.com/document/d/1BWOIRdJvUueJa4cWV68UEY3nlmHgAVJY6aNGo6YSCQg/edit
[10:01] <pitti> ev, bdmurray: bug 1084996
[10:03] <bdmurray> pitti: so with that SRU also bug 1084296 should be fixed
[10:11] <mpt> ev, bug 1056061
[10:15] <mpt> ev, bug 1020988 is fixed now, right? :-)
[10:18] <mpt> ev, bug 1068060 looks like another example of a typo-test-level bug with a decent-sized benefit
[10:20] <bdmurray> ev: is bug 1021326 fixed too?
[10:24] <mpt> ev, bdmurray: reported bug 1085853
[10:33] <mlankhorst> slangasek: is a rebuild of the non-ddx reverse-depends of x11proto enough?
[11:09] <mlankhorst> slangasek: https://launchpad.net/~mlankhorst/+archive/ppa/+packages pushed there, libxcb on armel failed on uploading, building went fine and it wasn't a rdepends anyway, just thought I'd test it
[11:10] <pitti> mvo: have you ever used python-apt to construct an apt cache for a different architecture?
[11:10] <pitti> mvo: I tried to call apt.apt_pkg.config.set('Architecture', 'armhf') before apt.Cache(rootdir=...), but it isn't taken into account
[11:13] <cjwatson> mlankhorst: I think that's possibly transient, as it seems to just have fallen over trying to upload to the librarian - retry that build?
[11:15] <mlankhorst> yeah it's fine, armhf worked so no need to try again..
[11:51] <didrocks> cjwatson: FYI, as you didn't attach any bugs to https://code.launchpad.net/~cjwatson/frame/cross/+merge/137457, I think that tomorrow's frame daily build will only be "Automated snapshot from rev <…>" (nothing wrong with that personnaly), just a warning :)
[11:51] <cjwatson> didrocks: Ah, fair enough, I wasn't sure how to interact with it from a patch submitter's point of view
[11:51] <cjwatson> didrocks: Not a big deal, it's one of hundreds ...
[11:51] <didrocks> cjwatson: exactly :)
[11:52] <cjwatson> So, what, it uses the bug title by default?
[11:52] <didrocks> yeah, the bug title
[11:53] <cjwatson> Would it be helpful if I filed a bug now and attached it?
[11:53] <didrocks> it's trying to search for an attached bug for all merges branches (whatever the deepness of the merge is), and as well, trying to fetch bug #<...> and a tons of variations of it in the commit message if you didn't attach it
[11:53] <didrocks> cjwatson: hum, now that the upstream jenkins merger is doing its job, I think it's too late TBH
[11:53] <cjwatson> OK
[11:54] <cjwatson> I'll live
[11:54] <didrocks> same for me ;)
[11:54] <didrocks> I just prefer to warn beforehand :)
[12:06] <mvo> pitti: I used this before, yes, let me check whats going on, maybe multiarch oddness
[12:06] <pitti> mvo: or does root= reset this?
[12:06] <mvo> pitti: and its generally a important and useful use-case
[12:06] <mvo> pitti: could be, give me a minute to create a test
[12:07] <pitti> mvo: danke!
[12:08] <cjwatson> doko__: Have you had any opportunity to consider Debian #693220 yet?
[12:10] <cjwatson> doko__: My cross-build environment has some corner-case difficulties with including wookey_'s bootstrap archive, and I'd like to get to the point of being able to work in plain raring ASAP
[12:10] <doko__> cjwatson, yeah, will look at it this week
[12:10] <cjwatson> Thanks
[12:14] <mvo> pitti: http://paste.ubuntu.com/1407600/ <- that seems to be doing what needs to be done (modulo that archive does not have armhf but if I use ports there it seems to be fine
[12:15] <pitti> mvo: odd, that's pretty much what I did
[12:15] <mvo> pitti: do you have anything in your rootdir that migh reset something in the config? a apt.conf or apt.conf.d snippets?
[12:15] <pitti> mvo: ok, if that works for you, then I probably just did something wrong
[12:15] <pitti> mvo: a sources.list with http://ports.u.c/, just like you
[12:16] <pitti> mvo: (currently discussing stuff with ev, getting back to this soon)
[12:16] <pitti> mvo: thanks!
[12:16] <mvo> pitti: odd, happy to dig more with you once you have a spare minute :)
[12:19] <pitti> Get:3 http://ports.ubuntu.com precise/main armhf Packages [1258 kB]
[12:19] <pitti> mvo: ok, your test case works here
[12:19] <pitti> mvo: ooh
[12:19] <mvo> pitti: \o/
[12:20] <pitti>         apt.apt_pkg.config.set('Architecture', 'armhf')
[12:20] <pitti> mvo: spot the error :)
[12:20] <pitti> yay, works in apport now
[12:20]  * pitti hugs mvo, sorry for disturbing then; just blindness
[12:20] <mvo> pitti: :)
[12:20] <ogra-cb> oh, when did we drop the requirement for the /ubuntu-ports dir ?
[12:20] <mvo> pitti: no worries, the config synatax is archane
[12:21] <ogra-cb> it always needed to be http://ports.ubuntu.com/ubuntu-ports in the past
[12:23] <pitti> hm, I guess not since I got my first arm device :)
[12:32] <ev> pitti: https://wiki.ubuntu.com/ErrorTracker/MapReduce - what I just explained in written form
[14:26] <ev> pitti: https://dev.launchpad.net/LEP/BugsToFixedBinaries
[14:26] <ev> mpt: http://bazaar.launchpad.net/~daisy-pluckers/whoopsie/trunk/view/head:/src/whoopsie.c#L87
[14:27] <mpt> ta
[14:32] <bdrung> cjwatson: can you please look at https://code.launchpad.net/~logan/ubuntu/raring/partman-basicfilesystems/debian-merge/+merge/135032 ?
[14:33] <cjwatson> I'm going to reject it because it's against the wrong branch, but OK ...
[14:34] <ev> pitti, mpt, bdmurray: https://bugs.launchpad.net/whoopsie/+bug/1085987 - created this from our batches discussion
[14:35] <bdrung> micahg: can you look at the SRU at bug #1027977 (you sponsored the quantal fix)?
[14:38] <bdrung> xnox: can you look at the SRU in bug #1027977 (you sponsored the fix to raring)?
[14:39] <mpt> bdmurray, bug 1084979
[14:43] <mpt> bdmurray, https://bugs.launchpad.net/ubuntu-error-tracker
[14:43] <xnox> bdrung: Can you explain how you calculate micahg for quantal & me for raring?
[14:45] <xnox> bdrung: micahg sponsored it into quantal, when quantal was the dev release. I was looking at the bug, but didn't review it for sru.
[14:45] <bdrung> xnox: hm, there is a copy-paste mistake
[14:46] <bdrung> xnox: bug #973014 is for you
[14:47] <bdrung> xnox: my calculation was to look at the sponsor who uploaded the fix to the development version
[14:48] <xnox> bdrung: I see now what you did there =)
[14:49] <bdrung> xnox: :D
[14:49]  * bdrung tries to be a load balancer
[14:55] <cjwatson> xnox: Any thoughts on http://paste.ubuntu.com/1407904/ ?  (still testing)
[14:57] <pitti> ogra-cb: IIRC you used qemu-user in the past to run arm binaries on x86, didn't you?
[14:58] <ogra-cb> pitti, sudo apt-get install qemu-user-static && sudo qemu-debootstrap ....
[14:58] <micahg> bdrung: sorry, not this week
[14:58] <pitti> ogra-cb: oh, I just found the -L switch
[14:59] <pitti> ogra-cb: I was trying LD_LIBRARY_PATH=`pwd`/lib qemu-arm /tmp/sandbox/usr/bin/gdb
[14:59] <xnox> cjwatson: looks ok. As long as it boots & still shows ubuntu logo with christmas lights going on and off again, this should be fine.
[14:59] <pitti> ogra-cb: but that tried to use /lib/ld-linux-armhf.so.3 still; but with -L it seems to work
[14:59] <ogra-cb> pitti, better use qemu-debootstrap and create an arm chroot ;)
[14:59] <pitti> ogra-cb: nah, I want this for the apport retracers
[14:59] <cjwatson> xnox: seems to in kvm -snapshot at least
[15:00] <ogra-cb> pitti, ah, k
[15:00] <pitti> ogra-cb: they already build a sandbox with all hte necessary debs and ddebs unpacked; I just want to rum armhf's gdb in there
[15:00] <pitti> ogra-cb: and that actually seems to work
[15:00] <ogra-cb> yeah, as long as it finds the libs
[15:01] <ogra-cb> you shouldnt need to call qemu-arm explicitly though
[15:01] <ogra-cb> binfmt shoould take care
[15:02] <ogra-cb> read: LD_LIBRARY_PATH=`pwd`/lib /tmp/sandbox/usr/bin/gdb ... should just work as well
[15:02] <pitti> ogra-cb: ah, maybe I didn't install that properly
[15:03] <ogra-cb> qemu-arm-static should handle it in its postinst
[15:03] <ogra-cb> or qemu-user-static as it is called nowadays
[15:03] <bdrung> pitti: can you look at bug #1070581 (you are the last uploader of that package)?
[15:05] <stokachu> tjaalton: would it be possible to get a debdiff for bug 1012900 so we can get it through SRU?
[15:06] <tjaalton> stokachu: I'll push 1.8.5 for precise
[15:07] <tjaalton> it's maintained in git
[15:07] <stokachu> tjaalton: perfect thank you
[15:07] <tjaalton> i'll pull some packaging fixes too
[15:08] <tjaalton> that landed raring on saturday
[15:08] <stokachu> tjaalton: even better! :D
[15:08] <tjaalton> +in
[15:08] <tjaalton> I'll finalize the package tomorrow
[15:09] <stokachu> tjaalton: thats perfect, ill look for an update tomorrow, thanks again
[15:14] <cjwatson> xnox: Righto, pretty sure this is right, so uploading; wanted you to have a heads-up though :-)
[15:14] <xnox> cjwatson: ok. i will reboot today =)
[15:19] <pitti> ogra-cb: eek -- we have no armhf ddebs for quantal and raring
[15:20] <pitti> ogra-cb: I fixed the ddeb-retriever configuration, but urgh..
[15:20] <pitti> seems nobody was missing them so far
[15:43] <pitti> ogra-cb: how would qemu-user integrate itself with binfmt-misc?
[15:43] <pitti> ogra-cb: shouldn't it ship a file in /var/lib/binfmts/ the or so?
[15:44] <cjwatson> qemu-user-static does
[15:44] <pitti> oh, -static
[15:44] <cjwatson> Which is almost more useful since you can copy it into chroots and such
[15:47] <pitti> hm, but I guess I still need to explicitly call qemu-arm, for the -L option (LD_LIBRARY_PATH doesn't seem to work)
[15:50] <ev> pitti: https://bugs.launchpad.net/daisy/+bug/1086014
[15:53] <ev> pitti: also, can you track your progress in the ARM retracer work in https://bugs.launchpad.net/daisy/+bug/1044437
[16:08] <slangasek> mlankhorst: yes, that's fine for a test
[16:16] <mlankhorst> slangasek: good enough then?
[16:16] <slangasek> mlankhorst: yep, looks like it - I'll get those processed today
[16:25] <ScottK> pitti: Is there a reason to leave jockey in the archive?
[16:25] <pitti> ScottK: did kubuntu stop using it?
[16:25] <pitti> Package: jockey-kde
[16:25] <pitti> Task: kubuntu-desktop, kubuntu-full, kubuntu-active-desktop, kubuntu-active-full, kubuntu-active
[16:25] <ScottK> Hmmm.
[16:26] <pitti> ScottK: if you don't need it any more, I'll happily kick it out
[16:26] <ScottK> Let me make sure.
[16:27] <Riddell> ScottK: muon doesn't yet replace it
[16:27] <Riddell> JontheEchidna needs more poking
[16:27] <ScottK> Ah.  OK.
[16:33] <shnatsel> cjwatson: Hello! Could you point me to the scripts that generate the "/efi" folder on Ubuntu ISOs? I can't find anything relevant in lp:ubuntu-cdimage
[16:35] <cjwatson> shnatsel: tools/boot/<blah>/boot-amd64
[16:35] <cjwatson> shnatsel: in lp:~ubuntu-cdimage/debian-cd/ubuntu
[16:35] <mlankhorst> normally x11proto won't break anyway, ddx depend on xserver abi and xserver abi just doesn't use the extra fields unless available.
[16:36] <ricotz> infinity, hi :), i hope you can cherry pick the mentioned fix for eglibc here: https://bugzilla.gnome.org/show_bug.cgi?id=687265#c9
[16:37] <shnatsel> cjwatson: thanks a lot!
[16:37] <shnatsel> found it :)
[16:38] <pitti> slangasek: ah, so I did try gdb-multiarch, but didn't set a target so it coulnd't decipher the core properly; do you happen to know the option for this?
[16:41] <pitti> ah, 'set architecture'
[16:42] <smoser> slangasek, around?
[16:44] <pitti> slangasek: hm, with value "arm" I still get "/tmp/tmpQ9sZpI" is not a core dump: File format is ambiguous", and arm-linux-gnueabihf or armhf are not valid
[16:45] <pitti> and "show architecture" from arm's gdb is just "arm"
[16:46] <shnatsel> oh, one more question: I see Precise seeds have been converted to using 3.5 kernel backported from Quantal by default for x86 architectures. I wonder if I should use 3.2 or 3.5 for a derivative distro? I'm primarily concerned by two things: for how long will it be supported and will hardware support patches be backported to 3.5 like they are being backported to 3.2?
[16:46] <cjwatson> The point of the 3.5 kernel is to provide improved hardware enablement
[16:46] <cjwatson> Though I think the plan is to supersede it with whatever's in raring after raring's out
[16:47] <shnatsel> cjwatson: ah, so Ivy Bridge works OOTB, etc? I hear it doesn't work in 3.2
[16:47] <cjwatson> No idea about that
[16:47] <cjwatson> Not a kernel guy, I just do what I'm told with the seeds :)
[16:47] <shnatsel> cjwatson: thanks, I'll ask in the kernel channel then :)
[16:48] <smoser> it would seem that mountall is likely the difference between pretty green dots at https://jenkins.qa.ubuntu.com/view/ec2%20AMI%20Testing/view/Overview/job/quantal-server-ec2-daily/153/ and red ones at https://jenkins.qa.ubuntu.com/view/ec2%20AMI%20Testing/view/Overview/job/quantal-server-ec2-daily/154/ . the difference in manifests of those 2 builds are http://paste.ubuntu.com/1408166/ . seems bug 1059471 to have regressed.
[16:53] <xnox> ev: http://tldp.org/HOWTO/LVM-HOWTO/
[16:53] <xnox> ev: http://tldp.org/HOWTO/LVM-HOWTO/commontask.html
[17:01] <infinity> ricotz: I'll have a look at it.
[17:03] <mlankhorst> slangasek: just leaves the rest of the stack then :)
[17:04] <ricotz> infinity, thank you
[17:09] <slangasek> pitti: can you toss me a sample core file so I can poke at it?  the documentation on 'set architecture' is lousy IME, I generally find I have to do this by feel
[17:10] <pitti> slangasek: I was using the one from bug 1070952
[17:10] <slangasek> smoser: hey there - otp, but go ahead :)
[17:10] <pitti> slangasek: we don't have debugging symbols for quantal/raring, but at least a stack of ?? without error messages would do fine
[17:11] <smoser> slangasek, i did. see above.
[17:11] <smoser> and comment added to bug 1059471
[17:11] <slangasek> smoser: right, reading now
[17:11] <smoser> and i apologize
[17:11] <smoser> for not testing as solid as you woudl have liked. but sometimes it doesnt fail, and I *did* test.
[17:23] <slangasek> smoser: unfortunately, the tests being run here aren't very diagnosable, "assertTrue" doesn't give us any other debugging information; can you reproduce these failures?
[17:23] <smoser> slangasek, yes.
[17:23] <smoser> they fail > 50% of the time on canonistack and on ec2
[17:25] <slangasek> smoser: oh, the console output looks more interesting though... seems you're getting 'Permission denied'?  Do the cloud-init jobs need any tweaking to work correctly with the changed mountall?
[17:25] <smoser> where do you see permission denied?
[17:26] <slangasek> smoser: https://jenkins.qa.ubuntu.com/view/ec2%20AMI%20Testing/view/Overview/job/quantal-server-ec2-daily/154/ARCH=i386,REGION=eu-west-1,STORAGE=ebs,TEST=simple-user-data,label=ubuntu-server-ec2-testing/console
[17:26] <slangasek> smoser: in response to the ssh attempts
[17:26] <smoser> well, if fails to ssh in
[17:26] <smoser> because the instance failed to import hte keypair
[17:26] <smoser> becaus the instance failed to find the datasource
[17:26] <smoser> ie, there was no way into that instance.
[17:27] <smoser> the data you want is actually
[17:27] <slangasek> ah; where's the datasource failure?
[17:27] <smoser> https://jenkins.qa.ubuntu.com/view/ec2%20AMI%20Testing/view/Overview/job/quantal-server-ec2-daily/154/ARCH=i386,REGION=eu-west-1,STORAGE=ebs,TEST=simple-user-data,label=ubuntu-server-ec2-testing/artifact/None/i386/m1.small/ebs/i-aa125fe1/uec2-20121128-1030-297e755409fb4e-terminated.console.txt
[17:27] <smoser> wait
[17:27] <smoser> thats a bad one.
[17:27] <smoser> well, sor of a bad one
[17:28] <smoser> it demonstrates the "plymouth ate output" bug
[17:29] <smoser> slangasek, the "good news" is that when i add --verbose to kernel cmdline it gets much harder to reproduce
[17:29] <smoser> :)
[17:30] <slangasek> heh
[17:30] <smoser> (ie, i cant reproduce with that on just yet)
[17:30] <smoser> i'm 0/5
[17:30] <smoser> slangasek, can you confirm that you want me to try to get failure with --verbose ?
[17:30] <smoser> because if so, i'll work on that.
[17:31] <smoser> but i can give you access to an instance to poke around on and just look if you'd like
[17:31] <smoser> your choice of raring or quantal
[17:32] <smoser> ok. i have it failing. console output is very limited, but i'll get it and let you into instance.
[17:33] <slangasek> smoser: thanks
[17:36] <smoser> slangasek, console log at: http://paste.ubuntu.com/1408297/ .
[17:39] <slangasek> smoser: thanks.  which part shows me the failed access to the datasource?
[17:40] <smoser> :) plymouth ate that output
[17:40] <slangasek> hmm
[17:40] <smoser> but its in /var/log/boot.log and /var/log/cloud-init.log
[17:40] <slangasek> ok
[17:40] <smoser> and you can correlate timestamps
[17:41] <slangasek> which is one is the expected data source?
[17:41] <smoser> EC2.
[17:41] <slangasek> ok
[17:42] <smoser> slangasek, there is a big hint in that the EC2 datasource didn't run until uptime of 136 seconds.
[17:42] <zul> barry: ping
[17:42] <smoser> cloud-init-nonet waiting 120 seconds for a network device.
[17:42] <smoser> cloud-init-nonet gave up waiting for a network device.
[17:43] <smoser> it would appear to me that the race condition that we're seeing stops the network device added hooks from firing and bringing up network.
[17:46] <slangasek> smoser: eth0 was seemingly not found until 5 minutes after boot... any idea there?
[17:46] <slangasek> smoser: /var/log/udev confirms that the kernel didn't turn up eth0 until 265.527315 after boot
[17:47] <slangasek> so this is triggered by mountall, yes, but the real problem is that something's preventing the network interface from being found in a timely manner by the kernel
[17:47] <smoser> well, the reason for so long after is proably that cloud-init-nonet blocked for 120 seconds, and then cloud-init (which was blocking) looked for the datasource for probably the remaineder of that time.
[17:48] <slangasek> cloud-init-nonet shouldn't be blocking udev
[17:48] <smoser> i agree.
[17:48] <smoser> but it is
[17:48] <smoser> :)
[17:48] <slangasek> how?
[17:48] <smoser> well some race there i think.
[17:48] <slangasek> udev is 'start on virtual-filesystems', udevtrigger is 'start on startup and started udev and not-container'
[17:49] <slangasek> so as soon as virtual-filesystems is emitted, udev runs; and the virtual-filesystems event does not block on the rootfs, which is where cloud-init-nonet hooks in
[17:49] <smoser> 265 = 10 seconds of boot time + 120 timeout + 126 seconds of "look for datasource"
[17:49] <slangasek> ok
[17:49] <slangasek> so process list confirms udev didn't start until 17:35
[17:49] <slangasek> I think I need mountall --verbose output to diagnose further :/
[17:50] <smoser> slangasek, ok. i'll work on that.
[17:50] <smoser> you have any ideas on the "plymouth ate my output" issue ?
[17:50] <smoser> (which i'm opening a separate bug for now)
[17:50] <slangasek> not really
[17:51] <slangasek> other than that I'm not sure 'console output' is actually a good idea
[17:51] <slangasek> (*because* it's probably incompatible with plymouth)
[17:52] <smoser> well thats a bug in plymouth/upstart
[17:52] <smoser> if console output doesnt work
[17:52] <smoser> i need some way to get stuff to the console
[17:52] <smoser> (which i can see remotely)
[17:59] <slangasek> smoser: it may be a bug in plymouth, but not the bug you're thinking
[18:00] <slangasek> plymouth itself is supposed to replay output to the console
[18:00] <smoser> right
[18:00] <smoser> it does the ioctl to /dev/console that says "give me all that stuff"
[18:00] <smoser> and then nis supposed to replay it
[18:05] <smoser> slangasek, you want --verbose or --debug for mountall
[18:05] <slangasek> smoser: --verbose is sufficient
[18:06] <slangasek> smoser: you're familiar with the dance for /etc/init/mountall.conf to capture the output off somewhere?
[18:07] <slangasek> pitti: 'set gnutarget elf32-littlearm'
[18:08] <smoser> slangasek, i don thtink so. but it *should* go to console i think :)
[18:08] <slangasek> smoser: yeah, if plymouth doesn't eat it ;P
[18:08] <pitti> slangasek: oh, how obvious! merci beaucoup
[18:08] <pitti> slangasek: this goes on top of "set architecture arm", I figure?
[18:08] <smoser> but it stlil gets into /var/log/upstart/mountall.conf i think
[18:09] <slangasek> smoser: I generally disable 'console output' and do '> /run/mountall.log 2>&1' in the script when I'm debugging mountall; you can also just omit 'console output' and it'll capture to /var/log/upstart automatically, yeah
[18:09] <slangasek> pitti: well, seems that it gives me the same output with or without the 'set architecture arm'
[18:10] <slangasek> pitti: that output is not particularly pretty though :)
[18:10] <pitti> slangasek: I still get "warning: wrong size gregset struct in core file"
[18:11] <slangasek> yeah, I get that too; not sure what it means
[18:11] <pitti> slangasek: with qemu-arm gdb it is slightly bigger than just one line
[18:11] <slangasek> digging further
[18:11] <pitti> slangasek: and it at least gets addresses
[18:11]  * slangasek nods
[18:11] <pitti> slangasek: thanks
[18:13] <slangasek> pitti: maybe the original binary is also needed?
[18:13] <pitti> slangasek: I do have that
[18:13] <slangasek> hmm
[18:13] <pitti> slangasek: I have a complete apport-like sandbox from apport-retrace
[18:13] <slangasek> can you give me a pointer to it?
[18:13] <slangasek> ah ok
[18:13] <pitti> $ du -hs /tmp/sandbox/
[18:13] <pitti> that's not too bad, I could tar it up
[18:14] <pitti> OMGupload speed from the office
[18:14] <slangasek> :)
[18:14] <pitti> http://people.canonical.com/~pitti/tmp/sandbox.tar.bz2
[18:16] <pitti> slangasek: http://paste.ubuntu.com/1408422/
[18:16] <pitti> slangasek: these are my two commands (slightly hacked in-place, as I was running them out of apport-retrace)
[18:17] <pitti> slangasek: (the sandbox has armhf's gdb package and dependencies unpacked)
[18:18]  * slangasek nods
[18:21] <smoser> slangasek, http://paste.ubuntu.com/1408434/
[18:23] <smoser> slangasek, appears mount of /run was blocked
[18:23] <slangasek> smoser: where do you see this?  I'm not seeing any of the mountall --verbose output in the console log
[18:23] <smoser> its not there.
[18:23] <smoser> in instance, /var/log/boot.log
[18:24] <slangasek> ah, ok
[18:24] <slangasek> I do however see 'mounted-run' running at 15s, so that doesn't seem to have been blocked
[18:24] <slangasek> right, so /run is *mounted*, but something prevents 'mounted MOUNTPOINT=/run' from returning
[18:29] <slangasek> smoser: there are only three jobs on this system which start on that event: resolvconf, container-detect, and mounted-run.  container-detect doesn't do anything that should block, and in any case the timestamp on its log shows that it completes quickly; that means we need to look at the other two to see what's blocking
[18:29] <slangasek> smoser: in theory the only long-running code in mounted-dev shouldn't block because it's backgrounded, but you could try commenting out the 'run-parts' line
[18:30] <smoser> slangasek, sure.
[18:33] <smoser> slangasek, it hink you meant mounted-run
[18:35] <slangasek> smoser: yes :)
[18:45] <barry> xnox: thanks for porting gnome-menus to py3
[18:45] <arosen> I'm trying to modify an existing package; if I do apt-get source <package>; how can I recreate the package using those files?
[18:46] <penguin42> arosen: dpkg-buildpackage
[18:49] <smoser> slangasek, i think i've reproduced with "#    [ -d "/etc/update-motd.d" ] && run-parts --lsbsysinit /etc/update-motd.d > /run/motd &"
[18:49] <smoser> (ie commented out)
[18:49] <smoser> i will wait a bit to be sure.
[18:49] <smoser> yeah, definitely reproduced there.
[18:53] <ev> mpt, bdmurray, pitti: http://www.toptable.co.uk/opentables.aspx?m=72&p=6&d=03/12/2012%2019:00:00&lat=51.5063116&lng=-0.0998713999999836&di=0.5&ula=Bankside,%20London%20Borough%20of%20Southwark,%20London%20SE1%200SU,%20UK&t=loc&fsr=1&rp=opentables.aspx
[18:55] <slangasek> smoser: ok, that seems to leave only resolvconf as a possible culprit, hmm
[18:57] <ev> http://www.blackandbluerestaurants.com/page/home/11/11
[18:57] <ev> totes workin hard ;)
[18:58] <mpt> #ubuntu-devel, The Food Channel
[18:58] <smoser> slangasek, 10.55.60.136 if you want to check my work
[18:59] <micahg> Are we serving Raspberry Pi? /me ducks
[19:02] <slangasek> smoser: looks sane to me; resolvconf seems to be the only possible explanation
[19:06] <slangasek> smoser: how about a set -x in /bin/resolvconf for debugging?
[19:06] <slangasek> smoser: sorry, /sbin/resolvconf
[19:07] <smoser> slangasek, [   14.223470] init: resolvconf state changed from post-start to running
[19:08] <smoser> that is the last occurance of 'resolvconf' in the console log
[19:08] <slangasek> yep
[19:08] <smoser> shouldn't we see a stopped or killed ?
[19:08] <slangasek> no
[19:08] <slangasek> though, the fact that it goes to 'running' that early means that the pre-start script has finished
[19:08] <slangasek> hmm
[19:09] <slangasek> ok, this may be a straight up bug in mountall then
[19:10] <smoser> oh yeah. you're right. about the running
[19:10] <slangasek> unfortunately we have asymmetric debug output here, so we don't actually see when the mounted event is emitted
[19:10] <smoser> http://paste.ubuntu.com/1408556/ is console output
[19:10] <slangasek> but given that none of the three dependent jobs seem to be buggy, I suspect that 'mounted /run' isn't being emitted until after the rootfs is mounted rw
[19:10] <slangasek> and that's a bug
[19:11] <smoser> well, that is one of the resonss i put utc and kernel uptime in cloud-init output
[19:11] <smoser> so you can correlate if you have symettric logs
[19:11] <smoser> err... if you only have separate logs
[19:11] <smoser> anyway.
[19:12] <slangasek> smoser: could I have you test a patch to mountall to add more debugging, while I start looking for this bug?
[19:12] <smoser> remember when we had /dev/console, and writes there were not masked by some complex daemon ?
[19:12] <smoser> that was cool
[19:12] <slangasek> :P
[19:12] <smoser> slangasek, sure.
[19:12] <slangasek> yes, I believe that was before we had an event-based init system and you could actually rely on writes to the console happening in some meaningful order
[19:14] <slangasek> smoser: http://people.canonical.com/~vorlon/mountall-better-debugging.patch
[19:29] <slangasek> smoser: oh, at least I have a clue now why this change triggered the problem: the initramfs is mounting /run with the device name "tmpfs" whereas /lib/init/fstab lists "none", so I guess before this fix we were maybe not emitting mounted MOUNTPOINT=/run at all?
[19:30] <slangasek> that should have manifested as broken resolvconf handling in the cloud images though, AFAICS
[19:30] <smoser> well, i've not seen broken resolvconf handling in images.
[19:30] <smoser> certainly not at the 50% failure rate that we're seeing this.
[19:31] <slangasek> yeah, it seems that would've been a 100% failure rate, not 50%
[19:31] <slangasek> hmm
[19:31] <slangasek> smoser: you could try changing "none" to "tmpfs" in /lib/init/fstab, see if it's reproducible then
[19:34] <smoser> slangasek, sure.
[19:34] <slangasek> smoser: (would still like to see debug output with that mountall patch first, however, to make sure this isn't a red herring)
[19:36] <smoser> slangasek, ok. i'm building that, and will build an image with it.
[19:36] <slangasek> cheers
[19:39] <smoser>  sbuild-build-depends-mountall-dummy : Depends: plymouth-dev (>= 0.8.5.1) but it is not installable
[19:39] <smoser> that happened when trying to build patched mountall. :-(
[19:39] <smoser> in schroot
[19:40] <infinity> rly?
[19:41] <infinity> E: Unable to locate package plymouth-dev
[19:42] <infinity> That's a more helpful error message.
[19:42] <slangasek> should be  plymouth-dev (>= 0.8.5.1) | libplymouth-dev
[19:42] <slangasek> and we have the latter
[19:42] <smoser> probably some user error. http://paste.ubuntu.com/1408628/
[19:42] <penguin42> I'd appreciate a review on the following trivial crash fix at some point; https://code.launchpad.net/~ubuntu-treblig/ubuntu/raring/xpilot-ng/bug-1033250
[19:42] <infinity> slangasek: plymouth-dev is a virtual...
[19:43] <slangasek> infinity: shouldn't be?
[19:43] <slangasek> infinity: it's the divergent Debian package name
[19:43] <infinity> Oh, it's a virtual in Ubuntu only.  Ick.
[19:43] <penguin42> you can't do version comparisons like that on virtuals can you?
[19:43] <slangasek> smoser: I think there's a toggle you have to set somewhere to get sbuild to behave correctly and not ignore alternate build-deps
[19:44] <slangasek> i.e., to get the Ubuntu buildd behavior
[19:44] <infinity> Pretty sure that versioned build-deps on virtuals explodes the apt resolver, hence the buildd sbuild is fine, the one on my system, not so much.
[19:44] <infinity> Oh, or maybe it's just the alt toggle.
[19:44] <slangasek> infinity: it's not "virtual", it's *non-existent*
[19:44] <mlankhorst> yeah if only i could do versioned virtuals :/
[19:44] <slangasek> the only reason apt reports it as "virtual" is mountall's own reference to it
[19:44] <SpamapS> slangasek: I know for php I have to use the aptitude build dep reosolver
[19:45] <infinity> slangasek: Oh, so it is.  That helps.
[19:45] <slangasek> really seems like we should set this toggle by default in the Ubuntu sbuild package
[19:46] <SpamapS> slangasek: the man page claims that the aptitude resolver is not the default because of performance concerns
[19:46] <SpamapS> I can't imagine it adds more than a few seconds to any given build
[19:47] <smoser> what is the toggle ?
[19:47] <infinity> No need to use aptitude.
[19:47] <infinity> smoser: Add "$resolve_alternatives = 1;" to your ~/.sbuildrc
[19:47] <smoser> gracias
[19:47] <slangasek> right, there it is
[19:47] <infinity> slangasek: If we toggle it by default, it breaks my sid builds, mind you.  I'm not sure there's a right answer here.
[19:48] <slangasek> infinity: I guess you could be fancy and make it vary with build target? :)
[19:48] <infinity> Yeah, which is the glory of config files that happen to be perl code.
[19:49] <infinity> But that's harder to enforce at the packaging level, since the package doesn't know what you've named all your chroots/targets.
[19:49] <infinity> Easier for a user to mangle.
[19:49] <infinity> Maybe our default sbuildrc could include a commented-out snippet, though.
[19:50] <ScottK> Probably something the sbuild wrapper in u-d-t could handle though.
[19:50] <infinity> There's a wrapper?
[19:50]  * ScottK thought so.
[19:51] <slangasek> I thought there was just mk-sbuild
[19:51] <ScottK> Yeah, that's what I was thinking of.
[19:51] <infinity> Yeah, that's all I see.
[19:51] <slangasek> oh, but does that generate sbuildrc snippets?  I don't recall
[19:51] <ScottK> Sorry for the distractin.
[19:51] <ScottK> distraction even.
[19:52] <infinity> Although, Andy just did some mangling to do proper pocket/component manipulation, and I should massage that and figure out if it belongs in sbuild or u-d-t.
[19:52] <infinity> Maybe an ubuntu-sbuild wrapper that employs his hacks, plus sets the right variables (resolve_alternatives can be set in the environmnet too) might do the trick.
[19:53] <ScottK> This is how pbuilder-dist got started ...
[19:54] <infinity> On balance, though, I'd prefer sbuild DTRT out of the box, if that's plausible.  The alternative thing is a minor mess, that's all.
[19:54]  * slangasek runs away
[19:54] <ScottK> (see pbuilder-dist-simple for the simple idea from which the monster grew)
[19:54] <infinity> Actually, I should turn that off in my .sbuildrc again before I forget and do a Debian upload with it on.
[19:54] <infinity> And remind myself to make it conditional later.
[19:56] <slangasek> pitti: do you also get this error? seems some deps are missing from the tarball: warning: Could not load shared library symbols for 17 libraries, e.g. /usr/lib/arm-linux-gnueabihf/libX11.so.6.
[20:22] <arosen> When I do apt-get install package vs apt-get source package for some reason a different version of package is received
[20:22] <arosen> any ideas why?
[20:27] <micahg> source files from other releases?
[20:30] <arosen> micahg: yes
[20:30] <micahg> arosen: there's your answer
[20:33] <arosen> micahg: also I was wondering if there was anyway to use dpkg-buildpackage without it having to look for an upstream tarball and instead use the untar'ed code from the current working directory ?
[20:38] <smoser> slangasek, 10.55.60.188 has reproduced raring+debug patch
[20:38] <smoser> there is no swap on those instances though
[20:40] <slangasek> smoser: random aside, do you have an .ssh config snippet of some kind to not save fingerprints for cloud hosts? :)
[20:41] <slangasek> smoser: ok, very interesting output, it shows the mounted event is sent *very* early but doesn't complete until we hit those timeouts
[20:41] <slangasek> so the device mismatch was a red herring after all
[20:42] <smoser> slangasek, http://paste.ubuntu.com/1408767/
[20:42] <smoser> oh. do you want the console output for that ?
[20:43] <smoser> http://paste.ubuntu.com/1408769/
[20:44] <smoser> duh.
[20:44] <smoser> i completely wasn't thinking. and didn't realize your patch added debug in both swap and "other"
[20:45] <slangasek> smoser: there's no chance I'm missing something here, is there?  like an upstart job that starts on /run and deletes itself on first boot?
[20:46]  * slangasek keeps living in denial that this is his bug :)
[20:47] <smoser> slangasek, no. nothing evil like that.
[20:48] <slangasek> mmk
[20:48] <smoser> slangasek, i was about to comment that cloud-init in "config" does execute a 'mount -a' as it might update /etc/fstab.
[20:48] <smoser> but it doesn't get to the point where it would/could do that.
[20:48] <slangasek> yep
[21:01] <slangasek> smoser: ok, am going to have to meditate over this code for a bit :/
[21:02]  * infinity hands slangasek an Amiga.
[21:06]  * ScottK thinks whisky might work better.
[21:12] <slangasek> smoser: aha
[21:13] <slangasek> smoser: try_mounts(), once it's seen that all mounts are attempted, loops waiting for each of the outstanding events to finish; and it happens that the outstanding event for / is in the list ahead of /run :P
[21:14]  * slangasek claims his whiskey reward
[21:16] <infinity> slangasek: Hrm, could that be the source of my on-again-off-again "waiting for /tmp" bug that I keep forgetting to file?
[21:16] <slangasek> infinity: shouldn't be, no
[21:16] <infinity> Hrmph.
[21:17] <infinity> Alright, I should do a few dozen reboots and see if I still have that bug, and then file it.
[21:17] <infinity> And we can go spelunking.
[21:17] <smoser> infinity, i see a waiting for swap all the time
[21:17] <smoser> (encrypted swap)
[21:17] <smoser> it ends up being mounted, just annoying.
[21:17] <smoser> i dont have /tmp on separate mp
[21:17] <infinity> smoser: The special thing about my case is that /tmp isn't actually a mountpoint here.
[21:17] <slangasek> smoser: does it time out?
[21:18] <slangasek> smoser: or do you only see the message briefly because it really is taking a while to find itself?
[21:19] <smoser> slangasek, briefly during boot... i dont knwo tha tit goes away, but it goes to login screen. (i didn't know if it was the login screen that wiped it off the display). it does get mounted.
[21:19] <smoser> but in most cases where i have swap i see the message on the boot screen
[21:19] <slangasek> hmm
[21:19] <smoser> granted i probably have encrypted swap everywhere
[21:20] <smoser> as i have a encrypted home user everywhere.
[21:20] <slangasek> yep
[21:28] <slangasek> smoser: build-tested only: http:?/people.canonical.com/~vorlon/mountall-more-asyncer.patch
[21:28] <slangasek> er
[21:28] <slangasek> http://people.canonical.com/~vorlon/mountall-more-asyncer.patch
[21:28] <smoser> more asyncer.
[21:28] <smoser> nice
[21:29] <smoser> slangasek, did you see the complaints in some logs about dbus and ...
[21:29] <smoser> process 329: arguments to dbus_server_unref() were incorrect, assertion "old_refcount > 0" failed in file ../../dbus/dbus-server.c line 749.
[21:30] <slangasek> smoser: there's a bug report about that; I don't yet understand that one either
[21:31] <slangasek> there's only a single call to dbus_server_unref() in all of mountall, so I'd love to know who is unreffing it for us :P
[21:36] <arosen> If you do apt-get installl <blah> apt will automatically resolve the dependencies for you. Is there anything like that if you do dpkg -i foo.deb?
[21:36] <ion> arosen: Offtopic for this channel, but anyway: dpkg --unpack foo.deb && apt-get -f install
[21:37] <slangasek> or just 'gdebi foo.deb'.
[21:37] <ion> If i’m on a desktop, i just use xdg-open, which opens it in software-center nowadays.
[21:45] <slangasek> smoser: can I ask you to open a new bug report about this mountall race?
[21:47] <smoser> are you not happy with the one i have open ?
[21:48] <smoser> https://bugs.launchpad.net/ubuntu/+bug/1078926
[21:48] <slangasek> smoser: no, because you didn't file it against mountall ;)
[21:49] <slangasek> moving it
[21:50] <smoser> slangasek, i should spin the kernel issue off to another bug though.
[21:50] <slangasek> ok
[21:54] <urgodfather> hello room, does anyone have experience in the gma500 (Poulsbo) chip? im looking to update to the 3.7 kernel, as it has additional bug fixes, but cannot get much insight as to update it properly
[22:07] <ScottK> barry: Did the pykde4 import fail because someone had pushed to it before the upload?
[22:07] <slangasek> smoser: ok, bug paperwork done.  do you expect to be able to test that proposed patch today?
[22:07] <barry> ScottK: i'm only guessing on that one.
[22:08] <ScottK> OK.
[22:08] <ScottK> Because that package even has Vcs-foo pointing elsewhere ...
[22:10] <slangasek> smoser: btw, I guess this is a regression vs. 2.42, but I think it's probably just a partial reintroduction of the same problem we had before 2.41?
[22:12] <smoser> slangasek, well, this is probably critical or greater at the moment.
[22:12] <slangasek> so is that a yes? :)
[22:12] <smoser> as anyone who upgrades on a running cloud instance is looking at a boot time of 260 seconds
[22:12]  * slangasek nods
[22:12] <cjwatson> "critical or greater" - signs of severity hyperinflation
[22:12] <smoser> and if we were to promote a daily we'd have 50% failure boot.
[22:12]  * cjwatson looks for the wheelbarrows full of bug comments
[22:12] <smoser> megahyper inflation
[22:13] <smoser> cjwatson, what would you call it if you regressed grub to having new installs boot only 50% of the time.
[22:13] <smoser> and existing installs take 4 minutes additional
[22:15] <smoser> slangasek, i've just built a image with that new deb.
[22:15] <smoser> i will launch a bunch of instances later.
[22:15] <slangasek> smoser: ack, thanks
[22:16] <smoser> and see if it seems fixed.
[22:19] <cjwatson> smoser: "critical", not "critical or greater" :P
[22:19] <slangasek> supercritical
[22:24] <ScottK> hypocritical
[22:24] <ScottK> No, wait.  That's something different.
[22:24] <ScottK> ;-)
[22:50] <mlankhorst> tjaalton: ohey new patch to make libgallium shared on mailing list btw
[23:25] <slangasek> smoser: have trivially proven to myself that this patch is broken; fixing now
[23:45] <slangasek> smoser: ok, have pushed a better patch now, this time it survives my own test. http://people.canonical.com/~vorlon/mountall-more-asyncer.patch
[23:45]  * mlankhorst pokes slangasek 
[23:46] <slangasek> smoser: I'm sufficiently convinced of it's correctness that I'm going to push it to Debian unstable now so we can get it on its way; but please test and let me know if I've missed anything