=== bjf is now known as bjf-afk === asac_ is now known as asac [09:51] Anyone around to take a look at a boot chart output from the ARM live-cd? Casper is the obvious cause of major slowness and I've documented it at https://wiki.ubuntu.com/ARM/LiveCDSpeedup#Casper, after Casper I can't see many worth-while wins though. Link - https://wiki.ubuntu.com/ARM/LiveCDSpeedup#BootChart [09:53] looks like adduser and the keyboard stuff are the worst [09:54] ogra: yeah, all due to debconf-communicate [09:54] adduser is taking 10sec ! [09:54] err [09:54] 30, sorry [09:55] JamieBennett: Did you ever track down precisely why debconf-communicate is acting so slow? [09:55] JamieBennett, try adding a sscript that calls debconf-coomunicate [09:55] ogra, persia: done and the findings are on the wiki page I liked to. [09:55] something that sets only one value and see if that also uses 30sec [09:56] i suspect thats a fixed value we see there [09:56] JamieBennett: Hrm? In the "Conclusion" section, or somewhere else? [09:56] i.e. each call to debconf adds 30sec [09:56] I'm seeing clear indication that it *is* debconf-communicate, but I still don't understand why there is a performance skew when compared to other architectures. [09:57] ogra: each call to debconf-communicate takes 4 seconds due to loading templates.dat via perl code [09:57] what about the other 26 ? [09:57] ogra: where are you getting 30 secs from? [09:58] oh, i only looked at the kbd stuff, adduser simply calls it multiple times [09:58] Yes - https://wiki.ubuntu.com/ARM/LiveCDSpeedup#19keyboard [09:58] well, twice actually [09:59] ogra - 5 times - casper-preseed calls debconf-communicate [09:59] i only see it twice in adduser [09:59] at least in the chart [10:00] ogra: The detailed breakdown of the adduser case is above. [10:00] above what ? [10:00] a 7-sec call to debconf-communicate, wondering about in user-setup-apply (which talks to debconf a lot), and then a final 4 seconds in another debconf-communicate. [10:00] Above the chart on the wiki page. [10:01] ogra: are you talking adduser or keyboard? [10:01] adduser [10:01] only looking at the bootchart [10:01] ah, yes thats two calls [10:01] keyboard is 5 [10:01] the chart shows only one very long one [10:02] JamieBennett: Are you *sure* there's no debconf-communicate in user-setup-apply? I thought I remembered a fair bit. [10:02] for kbd [10:02] persia: maybe, I can investigate. [10:02] ogra: persia: so continuing discussion on the our ACTION to document what needs to be backported from .32 [10:02] right [10:02] to .31 kernel [10:02] i see aufs and apparmor as critical [10:02] JamieBennett: What I'm most curious about is *why* perl is so slow at reading that cache on armel. I suspect there's a deeper bug that would be nice to address with lots of improvements all over. [10:02] how should we do this [10:02] ? [10:03] do we already know the areas where we need backports? [10:03] make a list of kernel features we knoe [10:03] *know [10:03] akk? [10:03] all? [10:03] that feels bad ;) [10:03] no [10:03] persia: lool had some thoughts about perl slowness, I'll talk to him to see if he has investigated it at all [10:03] all that are developed inhouse [10:03] ok brainstroming :) [10:03] you say aufs and apparmor ... [10:03] Could we simply compare the kernel-generated header files for .31 and .32, and backport stuff where the API has changed? [10:03] i remember persia mentioned ureadahead or something [10:04] ogra@osiris:~/Desktop/uboot/package/uboot-imx-2009.08$ ls /lib/modules/2.6.31-14-generic-pae/kernel/ubuntu/ [10:04] aufs compcache dm-raid4-5 drbd iscsitarget lenovo-sl-laptop lirc misc ndiswrapper rfkill [10:04] Mine were more userspacey, like ALSA and bluetooth. [10:04] i guess we can ignore the lenovo stuff :) [10:04] and ndiswrapper [10:04] so why are we looking at kernel/ubuntu only? [10:05] we dont [10:05] asac: How do you mean? [10:05] but thats the easy and obvious part [10:05] yeah [10:05] ok [10:05] we surely need to add stuff that happens inline in the kernel, plus what persia said [10:06] though i think the alsa HW specific stuff isnt that intresting [10:06] we dont need intel fixes on arm :) [10:06] What is interesting is keeping sync between userspace and kernelspace interfaces. [10:06] persia: we could check the header approach, and see if we get useful results. [10:06] right [10:06] I don't care about the individual bits of hardware, but I do care if something moved in ALSA, and sound doesn't work. [10:07] Most ARM hardware should be using ASoC anyway. [10:08] asac: To do that, should we generate linux-headers-2.6.31-foo out of the ARM kernel sources, or do you think there are other sources of API changes? [10:08] hmm. are linux-headers really the headers exported to userspace? [10:09] not linux-libc-dev ? [10:09] Well, both are exposed, actually. [10:09] really? [10:09] * persia goes to read up on stuff [10:09] i thought linux-headers was just for kernel modules [10:10] apt-cache show linux-libc-dev [10:10] stop ... [10:10] Right. Sorry. I was confused by "linux-kernel-headers" which was an old name for linux-libc-dev [10:10] This package provides headers from the Linux kernel. These headers [10:10] are used by the installed headers for GNU glibc and other system [10:10] libraries. They are NOT meant to be used to build third-party modules for [10:10] your kernel. Use linux-headers-* packages for that. [10:10] we had that discussion multiple times [10:10] thats a kernel team job [10:11] ogra: what do you mean? [10:11] * ogra tries to find the mail with the discussion about linux-headers vs libc-dev [10:12] asac, armel is a special case [10:12] and it took us weeks to agree on a specific setup everybody was happy with [10:13] * persia isn't entirely convinced it's safe to special-case headers against which userspace is built. [10:13] i know apw summarized it, i cant find it atm [10:14] persia, i think we need to use linux-kernel-headers-*SoC* for our stuff iirc [10:15] but i need the notes [10:15] the email was titled [10:15] Subject: arm headers issue [10:15] apw: To which list was it sent, and about when? [10:15] Anyway, I thought linux-libc-dev *replaced* linux-kernel-headers some time back. [10:16] it was sent to a few people direct [10:16] Ah, tricky those. [10:16] apw, gracias, got it [10:16] asac, forwarding to you [10:16] i don't recall it being contentious, just not interesting to the general reader in the raw form it was [10:17] yeah [10:17] but important knowledge if you work on armel [10:17] and surely important to know for asac who became our fearless tech lead now ;) [10:17] But the summary was that there are linux-kernel-headers-${SoC} packages being built that are just like linux-libc-dev except for the different ARM kernels? [10:18] persia, forwarded to you too [10:18] * persia likes smart search tools that automatically update [10:20] * persia is still confused [10:20] Is this just about arm-specific headers, or *all* headers? [10:21] arm [10:21] its presumably about apps needing vile arm specific driver headers [10:21] That's completely different from what I'm looking for. Let's call that solved for now (for all that I think it's a bit of a mess) [10:21] right [10:22] I was looking to compare linux-libc-dev from the SoC kernels to linux-libc-dev in the master kernel. [10:22] Where those differ, we have a candidate problem. [10:22] persia, that counts in SoC specific userspace API libs [10:22] Right, I don't care about backporting any of that stuff. [10:22] right, you cant [10:22] right :) [10:23] since linux-libc-dev on armel comes from the main source [10:23] linux-libc-dev for armel ... what he said [10:23] Right, so like I said, shall we have the SoC kernel sources provide linux-libc-dev-${SoC} for comparison purposes? [10:23] apw, hmm, how do you handle that with the .32 vs .31 issue btw ? [10:23] And then if we find discrepancies, backport what we need. [10:23] persia, no [10:23] Why not? [10:23] read the mail :) [10:24] colin will slay you [10:24] ogra, we don't [10:24] This is *completely* unrelated to the mail. [10:24] The mail is about SoC-specific headers. I'm talking about general headers. [10:24] I don't expect them to be used by anything: I just want them for reference. [10:24] If anything builds against them, we failed. [10:24] apw, evil ... what if vendors build stuff based on linux-libc-dev ? it will likely be boroken [10:25] We can put ***DON'T USE THIS*** **YOUR APPLICATION WILL FAIL*** in the description. [10:25] arm is in a world of pain by refusing to be at the same version as the rest of the system [10:25] we are assuming some level of compatibility between releases and literally crossing everything [10:25] apw: Well, we're hoping to avoid some of that with backporting. For instance, backporting ALSA should be relatively painless. [10:26] its already backported for non-arm so i assume so [10:26] Right :) [10:26] apw, well, i'm more concerned about things like aufs [10:26] The trick is to determine what needs to be backported to avoid pain. [10:26] we need it for image builds [10:27] i don't think there is any headers for aufs [10:27] how about apparmor [10:27] ogra: So, what objection do you have to comparing the generated linux-libc-dev for each kernel type? [10:27] included in the linux-libc-dev package [10:27] persia, there is no linux-libc-dev for each kernel type [10:27] But there could be :) [10:27] not sure we include those either [10:27] ther is only one and that comes from the upstream source [10:28] and the archive admin team is strictly against having more than one [10:28] Fine. [10:28] read the IRC discussion in the mail [10:28] we can't have more than one, as the archive can only be built for one armel form [10:28] I did. You aren't understanding what I'm saying. I'm not talking about the same thing from the mail. [10:28] and its that process that consumes the package [10:28] But I don't care to argue that: I'll find a workaround. [10:29] have we hit an issue with our assumptions here, or are we worryng about the future [10:29] apw: We've lost track of the discussion. [10:29] The agenda item was to try to identify what shoud be backported into .31 kernels to make sure lucid works with them. [10:30] Our inital guess was aufs compcache dm-raid4-5 drbd iscsitarget lirc misc rfkill + alsa + bluetooth [10:30] whateve misc is [10:30] * ogra takes a look in the dir [10:31] /lib/modules/2.6.31-14-generic-pae/kernel/ubuntu/misc/fsam7400.ko [10:31] description: SW RF kill switch for Fujitsu Siemens Amilo M 7400 [10:31] my understanding is that aufs is 'compatible' between the two semantically [10:31] not for us [10:31] apparmor userspace i believe is aware it may be using two differenet kernels [10:32] apw: We've had a lot of issues with aufs version skew in the past, and have little trust. [10:32] apw, right, but we need to make sure that all the inhouse developed features the kernel team develops work [10:32] thats why my main issues are aufs and apparmor ... but you answered for both, so i personally am happy [10:32] compcache and dmraid are surely also important to have working [10:33] not sure about lirc rfkill or iscsitarget [10:33] rfkill is likely useful, as lots of devices have wireless. [10:34] but do they have the HW that supports rfkill ? [10:34] lirc falls into the alsa/bluetooth category for me: if we don't do it, classes of devices won't work. [10:34] the babbage doesnt have wireless per se [10:34] and we actually only talk about babbage here [10:34] dove is on .32 [10:35] did we lose asac ? [10:36] * persia installs wireless-tools to see if rfkill works on at least one i.MX51-based device [10:37] no [10:37] reading thta long mail atm ;) [10:37] so from what i understand comparing headers doesnt make sense [10:38] its supposed to be not broken in that way [10:38] so lets focus on the core ideas we had: [10:38] a) figure out which "ubuntu" features were added in .32/lucid [10:38] -> review and make a list of what parts we want from those [10:39] Erm, what? [10:39] right [10:40] b) try to find some good targets to look closer: [10:40] e.g. what kernel improvements were done for boot perf. [10:40] which of them didnt go upstream [10:40] -> and check if we need those -> [10:40] we surely do [10:41] unless we stop caring about bootspeed suddenly [10:41] ogra: didnt go upstream in .31 ... given that we had .31 in karmic, nothing went upstream for us (fsl) [10:41] that was done in lucid [10:41] right [10:42] i double checked with dtchen back at UDS ... he said, that sound wont be a problem in that we will see regressions over karmic experience [10:42] ...however, we might not see improvements obviously ... [10:42] but i think we can live with that [10:42] yeah [10:42] as long as noise still comes out of the speakers :) [10:42] so lets scratch sound from the backporrting candidates unless we find real bugs [10:42] right [10:43] QA would hopefully catch that [10:43] Note that he has already backported the latest ALSA stuff to .31 in a PPA, so it should be fairly painless to integrate. [10:43] if fixes are available we are probably happy to pull them in [10:43] -> lets note that as an option/action [10:44] do we know anything besides boot performance kernel changes and sound? [10:44] apw, what looks important to me wrt bootspeed are the patches from csurbhi [10:44] apw, do you see any issues with these to go into a .31 tree ? [10:45] cooloney: there? [10:45] cooloney: what ubuntu specific kernel stuff that is in .32 wont be in our .31 kernel? [10:46] maybe we automatically get everything? [10:46] ogra, those were pretty simple patches, but do we have a bootspeed issue on arm? [10:47] apw, bootspeed is always an issue, yes [10:47] Well, looking at the bootchart we were discussing earlier, we're running at somewhere under 195 seconds right now. [10:47] apw, if we have any improvement in the main distro they should definately go into arm [10:47] persia, thats livefs :) [10:47] ogra: So? :) [10:47] my babbage boots lucid in about 50sec [10:47] Still, a fairly long time. [10:47] or 45 [10:47] yes, but not 195 :) [10:48] lets stay with apples vs apples [10:48] But yeah, we'd like that faster. [10:48] i agree that the kernel tweakage currently done for i386 is not really our major concenr [10:49] but i hope we get the big time consumers removed and then it might be more important [10:49] asac, even if it gains us several seconds in bootspeed ? [10:49] so being prepared is good [10:49] My concern is things not working because of kernel/userspace interface changes, more than small improvements. [10:49] bootspeed is definately a vendor concern we har all the time [10:49] Small improvements are nice, but the kernel is outdated, and deserves to suffer a bit. [10:49] *hear [10:50] persia: i dont think we can understand that up-front fully. [10:50] we can only figure that out through bugs later on [10:50] ogra: yes. but we have 195 secdons. most of that is probably not related to anything done in the .32 kernel [10:51] which is tuning it to get the last few seconds removed from i386 [10:51] asac, yes, but the 195sec are live images [10:51] asac, has nothing to do with this discussion [10:51] asac: Well, hrm. I have suspicions about stuff like bt, alsa, lirc, ieee1394, etc. We had lots of issues with intrepid when the version skewed unexpectedly. [10:51] But I'm not sure our test coverage is broad enough to catch all that (because it wasn't for i386 for intrepid) [10:51] persia: dtchen said alsa isnt a problem for older kernels [10:51] asac, the 45-50 sec for instealled systems *are* an issue for this discussion though [10:52] ogra: yes. but live image is similarly important. thats the first impression users get [10:52] asac: Does that presume a separate alsa-modules package? [10:52] ogra: not really sure where the problem is. i am all for looking at what was done and what can be pulled into .31 for bootspeed [10:52] asac, but they are no kernel related issues [10:53] asac, right ... i'm only talking about the installed system here ... the live issues are all casper related [10:53] ogra: right. but did we analyze our 45 seconds on installed images yet? [10:53] nope [10:53] maybe its half of the time in something similar [10:53] nope [10:53] Um, those two "nope"s are mutually exclusive: without analysis there's not enough information for the second answer. [10:53] you say that userspace is well tuned? [10:53] we dont call things like debconf-communicate anywhere in installed systems [10:54] ogra: still. for me its currently a blackbox [10:54] also on liveimages only 50 seconds or so are deconf-communicate [10:54] http://people.canonical.com/~ogra/babbage2-karmic-20090916-3.png [10:54] there is a karmic bootchart [10:55] Well, more than that. there's about 35 seconds in just 10adduser and 19keyboard [10:55] speed issues there are totally unrelated to the speed issues on the live boot [10:55] anyway. thats not the point [10:55] No. [10:55] right [10:55] but what i say is that we want kernel improvements backported if they can gain us a few secs [10:55] ogra: how can you say that its unrelated without having analysed it [10:56] anyway, as you might know JamieBennett is our boot speed master this cycle ;) [10:56] I think we should be focusing on SoC-specific performance enhancements with SoC-specific kernels, rather than worrying about backporting. [10:56] i.e. if we can get from 41 to 31 secs through them ... plus gain another 5 secs through ureadahead that is definately something we want [10:56] :) [10:56] so maybe he should take the action to work on it ... including figuring what kernel parts would be beneficial [10:56] asac, ++ [10:56] ok. lets do that then. doesnt make sense to speculate [10:56] Indeed. [10:57] asac, i have analyzed it in so far that i took bootcharts every release [10:57] but didnt have any time to look for possible improvements or to dig deeper [10:57] Do we run any perl during boot? [10:57] so bootspeed is covered ;) ... JamieBennett will work with whoever want to peer on that on a) identifying kernel patches to backport, b) identifying userspace bottlenecks in both: install and live image [10:58] OK [10:58] /me scrambles to read backchat to see what I have volunteered to do [10:58] JamieBennett, so please talk to csurbhi, if it makes sense to backport their patches to us [10:58] JamieBennett: basically the last line i said ;) [10:58] s/their/her/ [10:58] ogra: OK [10:58] 11:57 < asac> so bootspeed is covered ;) ... JamieBennett will work with whoever want to peer on that on a) identifying kernel patches to backport, b) identifying userspace bottlenecks in both: install and live image [10:58] JamieBennett: IN summary: analyse bootspeed of installed armel systems, compare against other architectures, and optimise. [10:58] yep [10:58] thats a tough comparison :) [10:58] Yep :) [10:59] given your HW is likely slowing it down [10:59] like disk etc [10:59] but well [10:59] Well, but simple HW issues scale, but if most stuff is 2x and one thing is 10x, that thing needs attention. [10:59] lets go back to the actual discussion [10:59] i dont think we can really compare it [10:59] maybe to identify stuff that takes relatively long [10:59] ok ... so lets move on ... bootspeed is covered ;) [10:59] Right. That's the extent of the potential for comparison. [11:00] now ... areas of potential userspace breakage: [11:00] mentioned were: [11:00] For *-modules, we just have to make sure that it can be built properly. [11:00] as apw said, aufs and apparmor should just work [11:00] alsa, bt, lirc, ieee1394 [11:00] dmraid [11:01] i can take the action to sync with jdstrand on what innovation we might want for apparmor [11:01] (and whoever does that on kernel side ) [11:01] compcache (very important) [11:01] whats your concerna bout compcache? [11:01] rfkill drdb ? [11:01] general breakage i presume? [11:01] yes [11:02] we are currently at "user space brekage";) [11:02] we should have the same upstream version in .31 lucid as in .32 lucid x86 [11:02] isnt compcache covered by kernel/ubuntu investigation? [11:02] since the live images 100% rely on it working [11:02] yes, its in kernel/ubuntu [11:02] which i had hpoed we could put on cooloney's plate ... mission: get as many goodies for ubuntu specific stuff from .32 to .31 as possibe [11:02] right [11:03] ok .... so lets assign this action to him. [11:04] a) make a list of stuff that changed for kernel/ubuntu in .32 [11:04] ok, that only leaves sound and apparmor [11:04] apparmor i took the action to check what innovation we might want [11:04] right [11:04] sound is likely mostly safe to leave alone. People who *really* care can get modules separately. [11:04] sound was already covered imo, but i can also check with dtchen directly what he thinks should be done [11:05] if he has patches ready for consumption that are safe, i dont see why we should not pick them [11:05] Heh. 590 headers files differ between linux and linux-fsl-imx51 [11:05] ;) [11:06] right, so: bootspeed->Jamie, kernel/ubuntu -> cooloney, sound-> dtchen, apparmor -> asac ... [11:06] anything we forgot ? [11:06] Lots of drm changes: we don't care about any of that, right? [11:06] unlikely [11:06] bt, lirc [11:06] lirc is in kernel/ubuntu [11:06] ogra: sound -> asac (with dtchen) [11:06] right [11:06] persia, so do yu take bt ? [11:07] Sure, although I'm going to be limited to source-inspection. I may have to lean on others to help with any testing. [11:07] well, grab a kernel guy for help :) [11:07] for now we want to investigate [11:07] and then we meet after alpha-2 to discuss the final actions i guess [11:08] and present results [11:08] yep [11:08] well, some bits need to happen pre A2 [11:08] is there any area that we know is currently breaking us? [11:08] at least aufs needs to work [11:08] no, we dont [11:08] e.g. is anthing of the above urgent for a2 [11:08] since we havent seen te3h .31 kernel yet [11:08] ok... aufs is kernel/ubuntu [11:08] anything else? [11:09] we can do definitive stuff only if we actually have the new kernel [11:09] ogra: why arent we seeing issues with aufs now if its needed for a2? [11:09] well, as apw said, it shouldnt have issues [11:09] for now, lets hope that stuff that works atm, will work with the new kernel [11:09] right [11:10] but you cant say until you see the real thing :) [11:10] we are talking about actions and urgencies, so fixing aufs is not urgent until we know its broken :) [11:10] sure [11:10] we are talking about hypothetical stuff atm [11:10] Seems that linux-fsl-imx51 headers are rather different for bluetooth :( [11:10] but i would hope that all stuff urgent enough for a2 target would be breakage of images [11:10] and would be covered by all other mechanims [11:11] ogra: yes. thats why i dont think that anything of the stuff we talked about has to happen for a2 atm [11:11] I'd agree that urgent stuff would be covered in other ways. [11:11] until we get to know of breakage ;) [11:11] right [11:11] This is more that we need to be solid pre-beta, and need a head start. [11:11] what we are doing here is an proactive effort to improve the overall quality of our final release [11:11] yep [11:12] persia: want to send out actions we deferred from this with me/ogra/jamie CCed? [11:12] and colooney [11:13] Why CC, rather than addressee? [11:13] hehe [11:13] yes. TOed [11:13] Sure, although I want to dig out of my current stack of stuff first. [11:14] * asac wonders what that is ;) [11:14] * ogra hands persia a ladder [11:14] my current stack of stuff == all the clothes that wait for a laundry here ;) [11:15] or rather: my board buried under heaps of dirty socks ;) [11:15] * persia has ~20 windows opened during this discussion that need to be closed after extracting the key bits while knowledge is fresh: actions are logged in lots of places to grab in some minutes. [11:15] * ogra fights with uboot [11:15] most likely those 20 windows opened on the sharp thing ;) [11:15] lol [11:16] No, only 4 windows on the Netwalker during the past hour discussion :p [11:16] (mostly terminals to check stuff) [11:19] anyone running lucid already? [11:19] on main desktop? [11:19] is sound working? [11:19] * asac scared that voip/softphone will be completely brokenish when i upgrade now ;) [11:19] Should be: lots of people report that it is. [11:20] Try a liveCD first :) [11:20] asac, cjwatson upgraded on monday i think [11:21] no idea if he uses any VoIP though [11:25] most likely not. i have the feeling i am the only one, given how broken all softphones are ;) [11:25] at least for real callout voip [11:25] most apps seems to close the sound stream and open quickly when the call gets dispatched ... this yields random on/off for output/input for the final connection [11:26] if you use pulse [11:26] ogra: Am I reading backscroll correctly that you're not researching anything? [11:26] ogra will be peer for boot performance and the kernel/ubuntu stuff [11:26] i said i didnt [11:26]