=== cpg|away is now known as cpg === Ursinha-afk is now known as Ursinha === Ursinha is now known as Ursinha-afk === sraue_ is now known as sraue === cpg is now known as cpg|away === Ursinha-afk is now known as Ursinha === cpg|away is now known as cpg [07:52] good morning [08:09] hi [09:12] pitti, bdmurray: https://wiki.ubuntu.com/ErrorTracker/ReducingRetracerDiskUsage [09:16] Good morning [09:37] pitti, mpt: https://docs.google.com/a/canonical.com/document/d/1BWOIRdJvUueJa4cWV68UEY3nlmHgAVJY6aNGo6YSCQg/edit === mcclurmc_away is now known as mcclurmc === henrix_ is now known as henrix === mcclurmc is now known as mcclurmc_away [10:01] ev, bdmurray: bug 1084996 [10:01] Launchpad bug 1084996 in Apport " StacktraceAddressSignature is missing" [High,Fix released] https://launchpad.net/bugs/1084996 [10:03] pitti: so with that SRU also bug 1084296 should be fixed [10:03] Launchpad bug 1084296 in apport (Ubuntu Quantal) "possible for a 2nd occurrence of a crash not to be sent to errors" [High,Triaged] https://launchpad.net/bugs/1084296 [10:11] ev, bug 1056061 [10:11] Launchpad bug 1056061 in Errors ""First seen" version is wrong in release-specific table" [High,Triaged] https://launchpad.net/bugs/1056061 [10:15] ev, bug 1020988 is fixed now, right? :-) [10:15] Launchpad bug 1020988 in Errors "The Whoopsie Daisy infrastructure should be charmed (juju) for testing and development" [Medium,In progress] https://launchpad.net/bugs/1020988 [10:18] ev, bug 1068060 looks like another example of a typo-test-level bug with a decent-sized benefit [10:18] Launchpad bug 1068060 in Errors ""the past week" is missing from table period menu" [Medium,Confirmed] https://launchpad.net/bugs/1068060 [10:20] ev: is bug 1021326 fixed too? [10:20] Launchpad bug 1021326 in Daisy "Only show version numbers from official Ubuntu sources in "First seen" and "Last seen" fields" [High,Fix committed] https://launchpad.net/bugs/1021326 [10:24] ev, bdmurray: reported bug 1085853 [10:24] Launchpad bug 1085853 in Errors "Errors code revision number not shown on the Web site" [Undecided,New] https://launchpad.net/bugs/1085853 === Tonio_ is now known as Tonio_aw [10:33] slangasek: is a rebuild of the non-ddx reverse-depends of x11proto enough? === pcarrier_ is now known as pcarrier === amitk is now known as amitk-afk [11:09] slangasek: https://launchpad.net/~mlankhorst/+archive/ppa/+packages pushed there, libxcb on armel failed on uploading, building went fine and it wasn't a rdepends anyway, just thought I'd test it [11:10] mvo: have you ever used python-apt to construct an apt cache for a different architecture? [11:10] mvo: I tried to call apt.apt_pkg.config.set('Architecture', 'armhf') before apt.Cache(rootdir=...), but it isn't taken into account === sraue_ is now known as sraue [11:13] mlankhorst: I think that's possibly transient, as it seems to just have fallen over trying to upload to the librarian - retry that build? [11:15] yeah it's fine, armhf worked so no need to try again.. === mcclurmc_away is now known as mcclurmc === dmart_ is now known as dmart === salem_ is now known as _salem === Tonio_aw is now known as Tonio_ [11:51] cjwatson: FYI, as you didn't attach any bugs to https://code.launchpad.net/~cjwatson/frame/cross/+merge/137457, I think that tomorrow's frame daily build will only be "Automated snapshot from rev <…>" (nothing wrong with that personnaly), just a warning :) === _salem is now known as salem_ [11:51] didrocks: Ah, fair enough, I wasn't sure how to interact with it from a patch submitter's point of view [11:51] didrocks: Not a big deal, it's one of hundreds ... [11:51] cjwatson: exactly :) [11:52] So, what, it uses the bug title by default? [11:52] yeah, the bug title [11:53] Would it be helpful if I filed a bug now and attached it? [11:53] it's trying to search for an attached bug for all merges branches (whatever the deepness of the merge is), and as well, trying to fetch bug #<...> and a tons of variations of it in the commit message if you didn't attach it [11:53] cjwatson: hum, now that the upstream jenkins merger is doing its job, I think it's too late TBH [11:53] OK [11:54] I'll live [11:54] same for me ;) [11:54] I just prefer to warn beforehand :) [12:06] pitti: I used this before, yes, let me check whats going on, maybe multiarch oddness [12:06] mvo: or does root= reset this? [12:06] pitti: and its generally a important and useful use-case [12:06] pitti: could be, give me a minute to create a test [12:07] mvo: danke! [12:08] doko__: Have you had any opportunity to consider Debian #693220 yet? [12:08] Debian bug 693220 in build-essential "Add crossbuild-essential support" [Wishlist,Open] http://bugs.debian.org/693220 [12:10] doko__: My cross-build environment has some corner-case difficulties with including wookey_'s bootstrap archive, and I'd like to get to the point of being able to work in plain raring ASAP [12:10] cjwatson, yeah, will look at it this week [12:10] Thanks === doko__ is now known as doko === yofel_ is now known as yofel [12:14] pitti: http://paste.ubuntu.com/1407600/ <- that seems to be doing what needs to be done (modulo that archive does not have armhf but if I use ports there it seems to be fine [12:15] mvo: odd, that's pretty much what I did [12:15] pitti: do you have anything in your rootdir that migh reset something in the config? a apt.conf or apt.conf.d snippets? [12:15] mvo: ok, if that works for you, then I probably just did something wrong [12:15] mvo: a sources.list with http://ports.u.c/, just like you [12:16] mvo: (currently discussing stuff with ev, getting back to this soon) [12:16] mvo: thanks! [12:16] pitti: odd, happy to dig more with you once you have a spare minute :) [12:19] Get:3 http://ports.ubuntu.com precise/main armhf Packages [1258 kB] [12:19] mvo: ok, your test case works here [12:19] mvo: ooh [12:19] pitti: \o/ [12:20] apt.apt_pkg.config.set('Architecture', 'armhf') [12:20] mvo: spot the error :) [12:20] yay, works in apport now [12:20] * pitti hugs mvo, sorry for disturbing then; just blindness [12:20] pitti: :) [12:20] oh, when did we drop the requirement for the /ubuntu-ports dir ? [12:20] pitti: no worries, the config synatax is archane [12:21] it always needed to be http://ports.ubuntu.com/ubuntu-ports in the past [12:23] hm, I guess not since I got my first arm device :) [12:32] pitti: https://wiki.ubuntu.com/ErrorTracker/MapReduce - what I just explained in written form === amitk-afk is now known as amitk === Tonio_ is now known as Tonio_aw === cpg is now known as cpg|away === Tonio_aw is now known as Tonio_ [14:26] pitti: https://dev.launchpad.net/LEP/BugsToFixedBinaries [14:26] mpt: http://bazaar.launchpad.net/~daisy-pluckers/whoopsie/trunk/view/head:/src/whoopsie.c#L87 [14:27] ta [14:32] cjwatson: can you please look at https://code.launchpad.net/~logan/ubuntu/raring/partman-basicfilesystems/debian-merge/+merge/135032 ? [14:33] I'm going to reject it because it's against the wrong branch, but OK ... [14:34] pitti, mpt, bdmurray: https://bugs.launchpad.net/whoopsie/+bug/1085987 - created this from our batches discussion [14:34] Launchpad bug 1085987 in Whoopsie "Submitting batches of error reports to daisy.ubuntu.com" [Undecided,New] [14:35] micahg: can you look at the SRU at bug #1027977 (you sponsored the quantal fix)? [14:36] Launchpad bug 1027977 in valgrind (Ubuntu Precise) "strstr() function produces wrong results under valgrind callgrind" [Low,Triaged] https://launchpad.net/bugs/1027977 [14:38] xnox: can you look at the SRU in bug #1027977 (you sponsored the fix to raring)? [14:38] Launchpad bug 1027977 in valgrind (Ubuntu Precise) "strstr() function produces wrong results under valgrind callgrind" [Low,Triaged] https://launchpad.net/bugs/1027977 [14:39] bdmurray, bug 1084979 [14:39] Launchpad bug 1084979 in Apport "Submitting error report asks confounding questions" [Medium,Confirmed] https://launchpad.net/bugs/1084979 [14:43] bdmurray, https://bugs.launchpad.net/ubuntu-error-tracker [14:43] bdrung: Can you explain how you calculate micahg for quantal & me for raring? [14:45] bdrung: micahg sponsored it into quantal, when quantal was the dev release. I was looking at the bug, but didn't review it for sru. [14:45] xnox: hm, there is a copy-paste mistake [14:46] xnox: bug #973014 is for you [14:46] Launchpad bug 973014 in gst-plugins-bad0.10 (Ubuntu Quantal) "gstreamer0.10-plugins-bad, (libgstvideoparsersbad.so), causes a failure to decode many common video files encoded as AVC 1 Baseline - L2.1, Baseline - L1.1 & others" [High,Confirmed] https://launchpad.net/bugs/973014 [14:47] xnox: my calculation was to look at the sponsor who uploaded the fix to the development version [14:48] bdrung: I see now what you did there =) [14:49] xnox: :D [14:49] * bdrung tries to be a load balancer [14:55] xnox: Any thoughts on http://paste.ubuntu.com/1407904/ ? (still testing) [14:57] ogra-cb: IIRC you used qemu-user in the past to run arm binaries on x86, didn't you? [14:58] pitti, sudo apt-get install qemu-user-static && sudo qemu-debootstrap .... [14:58] bdrung: sorry, not this week [14:58] ogra-cb: oh, I just found the -L switch [14:59] ogra-cb: I was trying LD_LIBRARY_PATH=`pwd`/lib qemu-arm /tmp/sandbox/usr/bin/gdb [14:59] cjwatson: looks ok. As long as it boots & still shows ubuntu logo with christmas lights going on and off again, this should be fine. [14:59] ogra-cb: but that tried to use /lib/ld-linux-armhf.so.3 still; but with -L it seems to work [14:59] pitti, better use qemu-debootstrap and create an arm chroot ;) [14:59] ogra-cb: nah, I want this for the apport retracers [14:59] xnox: seems to in kvm -snapshot at least [15:00] pitti, ah, k [15:00] ogra-cb: they already build a sandbox with all hte necessary debs and ddebs unpacked; I just want to rum armhf's gdb in there [15:00] ogra-cb: and that actually seems to work [15:00] yeah, as long as it finds the libs [15:01] you shouldnt need to call qemu-arm explicitly though [15:01] binfmt shoould take care [15:02] read: LD_LIBRARY_PATH=`pwd`/lib /tmp/sandbox/usr/bin/gdb ... should just work as well [15:02] ogra-cb: ah, maybe I didn't install that properly [15:03] qemu-arm-static should handle it in its postinst [15:03] or qemu-user-static as it is called nowadays === slank_away is now known as slank [15:03] pitti: can you look at bug #1070581 (you are the last uploader of that package)? [15:03] Launchpad bug 1070581 in langpack-locales (Ubuntu) "Name of the language spoken in Bangladesh" [Undecided,In progress] https://launchpad.net/bugs/1070581 [15:05] tjaalton: would it be possible to get a debdiff for bug 1012900 so we can get it through SRU? [15:05] Launchpad bug 1012900 in sssd (Ubuntu Precise) "Using SSSD, PAM error when exiting su session" [High,Confirmed] https://launchpad.net/bugs/1012900 === Ursinha is now known as Ursinha-afk [15:06] stokachu: I'll push 1.8.5 for precise [15:07] it's maintained in git [15:07] tjaalton: perfect thank you [15:07] i'll pull some packaging fixes too [15:08] that landed raring on saturday [15:08] tjaalton: even better! :D [15:08] +in [15:08] I'll finalize the package tomorrow [15:09] tjaalton: thats perfect, ill look for an update tomorrow, thanks again [15:14] xnox: Righto, pretty sure this is right, so uploading; wanted you to have a heads-up though :-) [15:14] cjwatson: ok. i will reboot today =) [15:19] ogra-cb: eek -- we have no armhf ddebs for quantal and raring [15:20] ogra-cb: I fixed the ddeb-retriever configuration, but urgh.. [15:20] seems nobody was missing them so far === Quintasan_ is now known as Quintasan [15:43] ogra-cb: how would qemu-user integrate itself with binfmt-misc? [15:43] ogra-cb: shouldn't it ship a file in /var/lib/binfmts/ the or so? [15:44] qemu-user-static does [15:44] oh, -static [15:44] Which is almost more useful since you can copy it into chroots and such [15:47] hm, but I guess I still need to explicitly call qemu-arm, for the -L option (LD_LIBRARY_PATH doesn't seem to work) [15:50] pitti: https://bugs.launchpad.net/daisy/+bug/1086014 [15:50] Launchpad bug 1086014 in Daisy "Generate reports for testing programmatically" [Medium,Confirmed] [15:53] pitti: also, can you track your progress in the ARM retracer work in https://bugs.launchpad.net/daisy/+bug/1044437 [15:53] Launchpad bug 1044437 in Daisy "Retracer ARM support" [Low,Confirmed] [16:08] mlankhorst: yes, that's fine for a test [16:16] slangasek: good enough then? [16:16] mlankhorst: yep, looks like it - I'll get those processed today [16:25] pitti: Is there a reason to leave jockey in the archive? [16:25] ScottK: did kubuntu stop using it? [16:25] Package: jockey-kde [16:25] Task: kubuntu-desktop, kubuntu-full, kubuntu-active-desktop, kubuntu-active-full, kubuntu-active [16:25] Hmmm. [16:26] ScottK: if you don't need it any more, I'll happily kick it out [16:26] Let me make sure. [16:27] ScottK: muon doesn't yet replace it [16:27] JontheEchidna needs more poking [16:27] Ah. OK. [16:33] cjwatson: Hello! Could you point me to the scripts that generate the "/efi" folder on Ubuntu ISOs? I can't find anything relevant in lp:ubuntu-cdimage [16:35] shnatsel: tools/boot//boot-amd64 [16:35] shnatsel: in lp:~ubuntu-cdimage/debian-cd/ubuntu [16:35] normally x11proto won't break anyway, ddx depend on xserver abi and xserver abi just doesn't use the extra fields unless available. [16:36] infinity, hi :), i hope you can cherry pick the mentioned fix for eglibc here: https://bugzilla.gnome.org/show_bug.cgi?id=687265#c9 [16:36] Gnome bug 687265 in general "deadlock in libcanberra-pulse with gstreamer 1.0.2" [Normal,Unconfirmed] [16:37] cjwatson: thanks a lot! [16:37] found it :) [16:38] slangasek: ah, so I did try gdb-multiarch, but didn't set a target so it coulnd't decipher the core properly; do you happen to know the option for this? [16:41] ah, 'set architecture' [16:42] slangasek, around? [16:44] slangasek: hm, with value "arm" I still get "/tmp/tmpQ9sZpI" is not a core dump: File format is ambiguous", and arm-linux-gnueabihf or armhf are not valid === yofel_ is now known as yofel [16:45] and "show architecture" from arm's gdb is just "arm" [16:46] oh, one more question: I see Precise seeds have been converted to using 3.5 kernel backported from Quantal by default for x86 architectures. I wonder if I should use 3.2 or 3.5 for a derivative distro? I'm primarily concerned by two things: for how long will it be supported and will hardware support patches be backported to 3.5 like they are being backported to 3.2? [16:46] The point of the 3.5 kernel is to provide improved hardware enablement [16:46] Though I think the plan is to supersede it with whatever's in raring after raring's out [16:47] cjwatson: ah, so Ivy Bridge works OOTB, etc? I hear it doesn't work in 3.2 [16:47] No idea about that [16:47] Not a kernel guy, I just do what I'm told with the seeds :) [16:47] cjwatson: thanks, I'll ask in the kernel channel then :) [16:48] it would seem that mountall is likely the difference between pretty green dots at https://jenkins.qa.ubuntu.com/view/ec2%20AMI%20Testing/view/Overview/job/quantal-server-ec2-daily/153/ and red ones at https://jenkins.qa.ubuntu.com/view/ec2%20AMI%20Testing/view/Overview/job/quantal-server-ec2-daily/154/ . the difference in manifests of those 2 builds are http://paste.ubuntu.com/1408166/ . seems bug 1059471 to have regressed. [16:48] Launchpad bug 1059471 in mountall (Ubuntu Quantal) "2.41 fails to mount root partition" [High,Fix released] https://launchpad.net/bugs/1059471 [16:53] ev: http://tldp.org/HOWTO/LVM-HOWTO/ [16:53] ev: http://tldp.org/HOWTO/LVM-HOWTO/commontask.html [17:01] ricotz: I'll have a look at it. [17:03] slangasek: just leaves the rest of the stack then :) [17:04] infinity, thank you [17:09] pitti: can you toss me a sample core file so I can poke at it? the documentation on 'set architecture' is lousy IME, I generally find I have to do this by feel [17:10] slangasek: I was using the one from bug 1070952 [17:10] Launchpad bug 1070952 in nux (Ubuntu) "unity_support_test crashed with SIGSEGV in omap_device_del()" [Undecided,New] https://launchpad.net/bugs/1070952 [17:10] smoser: hey there - otp, but go ahead :) [17:10] slangasek: we don't have debugging symbols for quantal/raring, but at least a stack of ?? without error messages would do fine [17:11] slangasek, i did. see above. [17:11] and comment added to bug 1059471 [17:11] smoser: right, reading now [17:11] Launchpad bug 1059471 in mountall (Ubuntu Quantal) "2.41 fails to mount root partition" [High,Fix released] https://launchpad.net/bugs/1059471 === deryck is now known as deryck[lunch] [17:11] and i apologize [17:11] for not testing as solid as you woudl have liked. but sometimes it doesnt fail, and I *did* test. === jtechidna is now known as JontheEchidna [17:23] smoser: unfortunately, the tests being run here aren't very diagnosable, "assertTrue" doesn't give us any other debugging information; can you reproduce these failures? [17:23] slangasek, yes. [17:23] they fail > 50% of the time on canonistack and on ec2 [17:25] smoser: oh, the console output looks more interesting though... seems you're getting 'Permission denied'? Do the cloud-init jobs need any tweaking to work correctly with the changed mountall? [17:25] where do you see permission denied? [17:26] smoser: https://jenkins.qa.ubuntu.com/view/ec2%20AMI%20Testing/view/Overview/job/quantal-server-ec2-daily/154/ARCH=i386,REGION=eu-west-1,STORAGE=ebs,TEST=simple-user-data,label=ubuntu-server-ec2-testing/console [17:26] smoser: in response to the ssh attempts [17:26] well, if fails to ssh in [17:26] because the instance failed to import hte keypair [17:26] becaus the instance failed to find the datasource [17:26] ie, there was no way into that instance. [17:27] the data you want is actually [17:27] ah; where's the datasource failure? [17:27] https://jenkins.qa.ubuntu.com/view/ec2%20AMI%20Testing/view/Overview/job/quantal-server-ec2-daily/154/ARCH=i386,REGION=eu-west-1,STORAGE=ebs,TEST=simple-user-data,label=ubuntu-server-ec2-testing/artifact/None/i386/m1.small/ebs/i-aa125fe1/uec2-20121128-1030-297e755409fb4e-terminated.console.txt [17:27] wait [17:27] thats a bad one. [17:27] well, sor of a bad one [17:28] it demonstrates the "plymouth ate output" bug [17:29] slangasek, the "good news" is that when i add --verbose to kernel cmdline it gets much harder to reproduce [17:29] :) [17:30] heh [17:30] (ie, i cant reproduce with that on just yet) [17:30] i'm 0/5 [17:30] slangasek, can you confirm that you want me to try to get failure with --verbose ? [17:30] because if so, i'll work on that. [17:31] but i can give you access to an instance to poke around on and just look if you'd like [17:31] your choice of raring or quantal [17:32] ok. i have it failing. console output is very limited, but i'll get it and let you into instance. [17:33] smoser: thanks [17:36] slangasek, console log at: http://paste.ubuntu.com/1408297/ . [17:39] smoser: thanks. which part shows me the failed access to the datasource? [17:40] :) plymouth ate that output [17:40] hmm [17:40] but its in /var/log/boot.log and /var/log/cloud-init.log [17:40] ok [17:40] and you can correlate timestamps [17:41] which is one is the expected data source? [17:41] EC2. [17:41] ok [17:42] slangasek, there is a big hint in that the EC2 datasource didn't run until uptime of 136 seconds. [17:42] barry: ping [17:42] cloud-init-nonet waiting 120 seconds for a network device. [17:42] cloud-init-nonet gave up waiting for a network device. [17:43] it would appear to me that the race condition that we're seeing stops the network device added hooks from firing and bringing up network. [17:46] smoser: eth0 was seemingly not found until 5 minutes after boot... any idea there? [17:46] smoser: /var/log/udev confirms that the kernel didn't turn up eth0 until 265.527315 after boot [17:47] so this is triggered by mountall, yes, but the real problem is that something's preventing the network interface from being found in a timely manner by the kernel [17:47] well, the reason for so long after is proably that cloud-init-nonet blocked for 120 seconds, and then cloud-init (which was blocking) looked for the datasource for probably the remaineder of that time. [17:48] cloud-init-nonet shouldn't be blocking udev [17:48] i agree. [17:48] but it is [17:48] :) [17:48] how? [17:48] well some race there i think. [17:48] udev is 'start on virtual-filesystems', udevtrigger is 'start on startup and started udev and not-container' [17:49] so as soon as virtual-filesystems is emitted, udev runs; and the virtual-filesystems event does not block on the rootfs, which is where cloud-init-nonet hooks in [17:49] 265 = 10 seconds of boot time + 120 timeout + 126 seconds of "look for datasource" [17:49] ok [17:49] so process list confirms udev didn't start until 17:35 [17:49] I think I need mountall --verbose output to diagnose further :/ [17:50] slangasek, ok. i'll work on that. [17:50] you have any ideas on the "plymouth ate my output" issue ? [17:50] (which i'm opening a separate bug for now) [17:50] not really [17:51] other than that I'm not sure 'console output' is actually a good idea [17:51] (*because* it's probably incompatible with plymouth) [17:52] well thats a bug in plymouth/upstart [17:52] if console output doesnt work [17:52] i need some way to get stuff to the console [17:52] (which i can see remotely) [17:59] smoser: it may be a bug in plymouth, but not the bug you're thinking [18:00] plymouth itself is supposed to replay output to the console [18:00] right [18:00] it does the ioctl to /dev/console that says "give me all that stuff" [18:00] and then nis supposed to replay it [18:05] slangasek, you want --verbose or --debug for mountall [18:05] smoser: --verbose is sufficient [18:06] smoser: you're familiar with the dance for /etc/init/mountall.conf to capture the output off somewhere? === mcclurmc is now known as mcclurmc_away [18:07] pitti: 'set gnutarget elf32-littlearm' [18:08] slangasek, i don thtink so. but it *should* go to console i think :) [18:08] smoser: yeah, if plymouth doesn't eat it ;P [18:08] slangasek: oh, how obvious! merci beaucoup [18:08] slangasek: this goes on top of "set architecture arm", I figure? [18:08] but it stlil gets into /var/log/upstart/mountall.conf i think [18:09] smoser: I generally disable 'console output' and do '> /run/mountall.log 2>&1' in the script when I'm debugging mountall; you can also just omit 'console output' and it'll capture to /var/log/upstart automatically, yeah === glebihan__ is now known as glebihan [18:09] pitti: well, seems that it gives me the same output with or without the 'set architecture arm' [18:10] pitti: that output is not particularly pretty though :) [18:10] slangasek: I still get "warning: wrong size gregset struct in core file" [18:11] yeah, I get that too; not sure what it means [18:11] slangasek: with qemu-arm gdb it is slightly bigger than just one line [18:11] digging further [18:11] slangasek: and it at least gets addresses [18:11] * slangasek nods [18:11] slangasek: thanks [18:13] pitti: maybe the original binary is also needed? [18:13] slangasek: I do have that [18:13] hmm [18:13] slangasek: I have a complete apport-like sandbox from apport-retrace [18:13] can you give me a pointer to it? [18:13] ah ok [18:13] $ du -hs /tmp/sandbox/ [18:13] that's not too bad, I could tar it up [18:14] OMGupload speed from the office [18:14] :) [18:14] http://people.canonical.com/~pitti/tmp/sandbox.tar.bz2 [18:16] slangasek: http://paste.ubuntu.com/1408422/ [18:16] slangasek: these are my two commands (slightly hacked in-place, as I was running them out of apport-retrace) [18:17] slangasek: (the sandbox has armhf's gdb package and dependencies unpacked) [18:18] * slangasek nods [18:21] slangasek, http://paste.ubuntu.com/1408434/ [18:23] slangasek, appears mount of /run was blocked [18:23] smoser: where do you see this? I'm not seeing any of the mountall --verbose output in the console log [18:23] its not there. [18:23] in instance, /var/log/boot.log [18:24] ah, ok [18:24] I do however see 'mounted-run' running at 15s, so that doesn't seem to have been blocked [18:24] right, so /run is *mounted*, but something prevents 'mounted MOUNTPOINT=/run' from returning [18:29] smoser: there are only three jobs on this system which start on that event: resolvconf, container-detect, and mounted-run. container-detect doesn't do anything that should block, and in any case the timestamp on its log shows that it completes quickly; that means we need to look at the other two to see what's blocking [18:29] smoser: in theory the only long-running code in mounted-dev shouldn't block because it's backgrounded, but you could try commenting out the 'run-parts' line [18:30] slangasek, sure. [18:33] slangasek, it hink you meant mounted-run [18:35] smoser: yes :) [18:45] xnox: thanks for porting gnome-menus to py3 [18:45] I'm trying to modify an existing package; if I do apt-get source ; how can I recreate the package using those files? [18:46] arosen: dpkg-buildpackage [18:49] slangasek, i think i've reproduced with "# [ -d "/etc/update-motd.d" ] && run-parts --lsbsysinit /etc/update-motd.d > /run/motd &" [18:49] (ie commented out) [18:49] i will wait a bit to be sure. [18:49] yeah, definitely reproduced there. [18:53] mpt, bdmurray, pitti: http://www.toptable.co.uk/opentables.aspx?m=72&p=6&d=03/12/2012%2019:00:00&lat=51.5063116&lng=-0.0998713999999836&di=0.5&ula=Bankside,%20London%20Borough%20of%20Southwark,%20London%20SE1%200SU,%20UK&t=loc&fsr=1&rp=opentables.aspx [18:55] smoser: ok, that seems to leave only resolvconf as a possible culprit, hmm [18:57] http://www.blackandbluerestaurants.com/page/home/11/11 [18:57] totes workin hard ;) [18:58] #ubuntu-devel, The Food Channel [18:58] slangasek, 10.55.60.136 if you want to check my work [18:59] Are we serving Raspberry Pi? /me ducks [19:02] smoser: looks sane to me; resolvconf seems to be the only possible explanation [19:06] smoser: how about a set -x in /bin/resolvconf for debugging? [19:06] smoser: sorry, /sbin/resolvconf [19:07] slangasek, [ 14.223470] init: resolvconf state changed from post-start to running [19:08] that is the last occurance of 'resolvconf' in the console log [19:08] yep [19:08] shouldn't we see a stopped or killed ? [19:08] no [19:08] though, the fact that it goes to 'running' that early means that the pre-start script has finished [19:08] hmm [19:09] ok, this may be a straight up bug in mountall then [19:10] oh yeah. you're right. about the running [19:10] unfortunately we have asymmetric debug output here, so we don't actually see when the mounted event is emitted [19:10] http://paste.ubuntu.com/1408556/ is console output [19:10] but given that none of the three dependent jobs seem to be buggy, I suspect that 'mounted /run' isn't being emitted until after the rootfs is mounted rw [19:10] and that's a bug [19:11] well, that is one of the resonss i put utc and kernel uptime in cloud-init output [19:11] so you can correlate if you have symettric logs [19:11] err... if you only have separate logs [19:11] anyway. [19:12] smoser: could I have you test a patch to mountall to add more debugging, while I start looking for this bug? [19:12] remember when we had /dev/console, and writes there were not masked by some complex daemon ? [19:12] that was cool [19:12] :P [19:12] slangasek, sure. [19:12] yes, I believe that was before we had an event-based init system and you could actually rely on writes to the console happening in some meaningful order [19:14] smoser: http://people.canonical.com/~vorlon/mountall-better-debugging.patch [19:29] smoser: oh, at least I have a clue now why this change triggered the problem: the initramfs is mounting /run with the device name "tmpfs" whereas /lib/init/fstab lists "none", so I guess before this fix we were maybe not emitting mounted MOUNTPOINT=/run at all? [19:30] that should have manifested as broken resolvconf handling in the cloud images though, AFAICS [19:30] well, i've not seen broken resolvconf handling in images. [19:30] certainly not at the 50% failure rate that we're seeing this. [19:31] yeah, it seems that would've been a 100% failure rate, not 50% [19:31] hmm [19:31] smoser: you could try changing "none" to "tmpfs" in /lib/init/fstab, see if it's reproducible then [19:34] slangasek, sure. [19:34] smoser: (would still like to see debug output with that mountall patch first, however, to make sure this isn't a red herring) [19:36] slangasek, ok. i'm building that, and will build an image with it. [19:36] cheers === Tm_Tr is now known as Guest3774 [19:39] sbuild-build-depends-mountall-dummy : Depends: plymouth-dev (>= 0.8.5.1) but it is not installable [19:39] that happened when trying to build patched mountall. :-( [19:39] in schroot [19:40] rly? [19:41] E: Unable to locate package plymouth-dev [19:42] That's a more helpful error message. [19:42] should be plymouth-dev (>= 0.8.5.1) | libplymouth-dev [19:42] and we have the latter [19:42] probably some user error. http://paste.ubuntu.com/1408628/ [19:42] I'd appreciate a review on the following trivial crash fix at some point; https://code.launchpad.net/~ubuntu-treblig/ubuntu/raring/xpilot-ng/bug-1033250 [19:42] slangasek: plymouth-dev is a virtual... [19:43] infinity: shouldn't be? [19:43] infinity: it's the divergent Debian package name [19:43] Oh, it's a virtual in Ubuntu only. Ick. [19:43] you can't do version comparisons like that on virtuals can you? [19:43] smoser: I think there's a toggle you have to set somewhere to get sbuild to behave correctly and not ignore alternate build-deps [19:44] i.e., to get the Ubuntu buildd behavior [19:44] Pretty sure that versioned build-deps on virtuals explodes the apt resolver, hence the buildd sbuild is fine, the one on my system, not so much. [19:44] Oh, or maybe it's just the alt toggle. [19:44] infinity: it's not "virtual", it's *non-existent* [19:44] yeah if only i could do versioned virtuals :/ [19:44] the only reason apt reports it as "virtual" is mountall's own reference to it [19:44] slangasek: I know for php I have to use the aptitude build dep reosolver [19:45] slangasek: Oh, so it is. That helps. [19:45] really seems like we should set this toggle by default in the Ubuntu sbuild package [19:46] slangasek: the man page claims that the aptitude resolver is not the default because of performance concerns [19:46] I can't imagine it adds more than a few seconds to any given build [19:47] what is the toggle ? [19:47] No need to use aptitude. [19:47] smoser: Add "$resolve_alternatives = 1;" to your ~/.sbuildrc [19:47] gracias [19:47] right, there it is [19:47] slangasek: If we toggle it by default, it breaks my sid builds, mind you. I'm not sure there's a right answer here. [19:48] infinity: I guess you could be fancy and make it vary with build target? :) [19:48] Yeah, which is the glory of config files that happen to be perl code. [19:49]