/srv/irclogs.ubuntu.com/2011/07/19/#ubuntu-kernel.txt

=== jk-- is now known as jk-
=== jjohansen is now known as jj-afk
chrismsnzHi guys, I'm running 10.04 LTS on a group of servers and I'm having some trouble when it comes under high load02:28
chrismsnzthe system becomes unstable and will sometimes go into a state where it responds to basic network access (i.e. ping, accepting connection) but nothing further and for all intents and purposes is down/dead02:29
chrismsnzI have a system in this state at the moment, and it's fairly easy to trigger the problem (it just takes a trip out to the colo to kick them in the pants when it happens).02:30
chrismsnzHow would I start collecting data to file this as a bug report?02:30
chrismsnzis ubuntu-bug -p linux the best way to get underway?02:31
RAOFchrismsnz: Yes, that's generally a good start.02:32
chrismsnzso rebooting the server and running ubuntu-bug will pick up the old dmesg and everything?02:32
brycehchrismsnz, check dmesg for error messages02:33
brycehchrismsnz, no you have to run it while system is frozen02:33
brycehchrismsnz, if it is a gpu hang, you should also gather intel register dumps as well, although frankly it's unlikely gpu hang fixes will get done for lucid.  But who knows.02:34
brycehEOD, cya.02:34
chrismsnzright02:35
chrismsnzI actually had installed a newer kernel on this particular system to see if the problem was solved - but it was not02:37
chrismsnzI will include in bug report, thanks guys02:37
=== anubhav__ is now known as anubhav
=== smb` is now known as smb
smbmorning07:22
aboganismb: morning07:26
* ppisati -> out for lunch10:59
RAOFapw: Could I trouble you for a kernel build with the patch from bug #753994 ?11:40
ubot2Launchpad bug 753994 in linux "[arrandale] Display is slanted when using 1360x768 resolution" [Medium,Confirmed] https://launchpad.net/bugs/75399411:40
apwRAOF, sure12:01
apwRAOF, for what series12:01
apwRAOF, ok found it, assuming natty, will let you know when its done12:08
ohsixherton: hi, i'm the guy on #793796, is this patch related to it? https://patchwork.kernel.org/patch/837692/ it's in the -11 package that just hit proposed; i won't have time to test it for a bit12:18
hertonohsix: hi. yes, the patch is related, but it wasn't sufficient to fix your case. you need also the 0004 and 0005 patches from http://people.canonical.com/~herton/lp793796/r5/12:20
ohsixok, i read the one that was in r6 and that looked like the same, but percolated up12:21
hertonohsix: upstream did another patch trying to not having to add a lot of tests for dead queues on elevator, it was the last kernel you tested, but it didn't work. I reported your test on upstream bug report, but there is no response yet12:21
ohsixnod, saw that12:22
ohsixjust saw it in -11 and figured some wizard might have just bypassed it all :}12:22
hertonunfortunately no. still you shouldn't see the issues on -11 kernel as we kept reverted the patches that caused the problem for you, so you can use the new kernel12:23
ohsixthose are just reverts right? who committed the patches? shouldn't be too hard to get them to try unplugging a usb drive :]12:25
hertonyep, it just reverts the commit we found on bisect, plus also reverts the followup fixes to this first commit reverted12:27
hertonohsix: the reverts are being tracked on bug 80298612:30
ubot2Launchpad bug 802986 in linux "Revert upstream scsi run queue changes" [Undecided,In progress] https://launchpad.net/bugs/80298612:30
ohsixhm, didn't know about that one, thanks12:34
ohsixi'm surprised i haven't heard of a thread on the lkml about it or something12:34
hertonI think there was a thread only in linux-scsi, from where the patch in r6 you tested came from, and bug https://bugzilla.kernel.org/show_bug.cgi?id=38842 is in the regressions list that is frequently posted on lkml12:38
ubot2bugzilla.kernel.org bug 38842 in Other "panic in elv_completed_request on safe remove usb hard drive" [High,Needinfo]12:38
ohsixi hope people look at the lp bug for the simple reproduction steps, instead of getting the virtual machine image D:12:43
apwRAOF, ok kernels build and published in the bug12:53
apwsmb, hey where do you get old isos from 12:59
smbold isos of what exactly?13:00
apwhardy for instance13:00
smbhttp://release.ubuntu.com 13:00
apwi happen to have other releases lying around13:00
smberr13:00
smbreleases13:00
smbapw, If course that only gives you latest point release and release for lts. if that is ok13:02
apwyeah just something old enough :)13:02
smbapw, could probably ship you gutsy... ;-P13:03
apwsmb, so very kind13:06
smbapw, Heh, though you would be appreciating. Btw, where you want to install hw-wise. Usually hardy fails on NICs or wireless (if recent)13:08
apwon a netbook, any chance ?13:08
smbhm. could already be questionable as they always slam in latest stuff... it seems ok on the dell 1521 which has some broadcom... 13:10
smb(not that you could call that recent)13:10
apwsmb, well i guess i'll give it a shot and see what happens13:10
smbYeah, you will notice quite early when the install complains about the network setup failed13:11
jwibugs related to -virtual kconfig: 720644, 769527, 771855, 794570, 761809, 658461 </braindump>13:18
apwbug #720644 bug #769527 bug #771855 bug #794570 bug #761809 bug #65846113:27
ubot2Launchpad bug 720644 in linux "linux-virtual kernel does not allow network configuration via kernel command line" [Undecided,Confirmed] https://launchpad.net/bugs/72064413:27
ubot2Launchpad bug 769527 in linux "Missing rpcsec_gss_krb5 module" [Undecided,New] https://launchpad.net/bugs/76952713:27
ubot2Launchpad bug 771855 in linux "reiserfs module missing in linux-image-virtual" [Undecided,New] https://launchpad.net/bugs/77185513:27
ubot2Launchpad bug 794570 in linux "igbvf driver is missing from virtual-flavored kernel" [Medium,Triaged] https://launchpad.net/bugs/79457013:27
ubot2Launchpad bug 761809 in linux "Quota modules are missing from the package" [Undecided,Incomplete] https://launchpad.net/bugs/76180913:27
bjf##14:00
bjf## Ubuntu Kernel Team Meeting - Today @ 17:00 UTC - #ubuntu-meeting14:00
bjf##      agenda: https://wiki.ubuntu.com/KernelTeam/Meeting14:00
bjf##14:00
bjfJ14:00
apwbjf thanks for the reminder14:03
bjfapw, and this one only once :-)14:04
apwits hard to remember its tue when its your first day of the week14:04
ogasawarabjf: I liked the morning bug report14:04
ogasawarabjf: sending it 50 times really made sure I looked at it :)14:05
bjfogasawara, heh14:05
* smb was already worried somebody sent a lot of patches...14:06
bjfhey! we take our bugs seriously!14:06
ogasawarasmb: hehe, I had the same feeling for a second14:06
* ogasawara back in 2014:47
ogasawarawhoa, he put himself back to sleep.  lets see how long this lasts...14:49
apwogasawara, not long :)14:52
=== cmagina_ is now known as cmagina
ogasawarashort lived indeed15:00
* ogasawara back in 2015:00
vanhoofherton: heya15:30
hertonvanhoof: hi15:30
vanhoofherton: quick q15:30
vanhoofherton: bug 77494715:30
ubot2Launchpad bug 774947 in linux "[Lenovo Edge 11 AMD] system locks up completely running the "stress" tool" [Medium,Fix released] https://launchpad.net/bugs/77494715:30
vanhoofthat fix came through 2.6.38 -stable, and landed in last weeks 2.6.38.10 kernel15:30
vanhoofanything I need to do with it at this point? (just looking at the most recent update that was popped in there)15:31
hertonvanhoof: in fact it was applied by hand, doesn't have a buglink to a stable update tracking bug15:31
hertonlet me check here if it was included in a stable update15:32
vanhoofherton: ah ok, thought it was from the dialog in the bug15:33
apwvanhoof, why have you closed the natty task for that bug ?  you say its cause 2.6.38-10 was released but that only contains 2.6.38.715:33
vanhoofapw: based on the testing that was done on -10, so I assumed this trickled down in 2.6.38.715:34
apwvanhoof, but you just said some 10 lines back that it was only in last weeks 38.1015:35
apwwhich isn't in 38-1015:35
vanhoofapw: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/774947/comments/3115:35
ubot2Ubuntu bug 774947 in linux "[Lenovo Edge 11 AMD] system locks up completely running the "stress" tool" [Medium,Fix committed]15:35
vanhoofthats what I'm going with here15:35
vanhoofif its in 2.6.38.8, I'll flip it back to fix commited15:35
apwvanhoof, well if the fix is in that kernel then all is good, but thats the right reason to change the state to released15:36
vanhooflooks like it was just those fixed on-top of -1015:37
hertonvanhoof: right, same fix came with 2.6.38.8, I'll update the bug there, you don't need to verify15:37
vanhoofherton: cool, i flipped it back to fix commited then15:37
* vanhoof mis-interpreted the testing comment, and thought it was on the -proposed kernel15:37
apwapw@dm$ git describe --contains 9ee653dce0efc6bad29f0d68b4ac74dbed09313115:38
apwUbuntu-2.6.38-11.47~15915:38
apwas far as i can see the commit is not yet released, it is Fix Committed only15:38
vanhoofright15:38
vanhoofwhich I just set it back to15:38
apwand as it is correcelyt marked15:38
apwit should close itself when it releases15:38
vanhoofapw: we get a few bugs where it's just waiting on some fix in -stable, like this one, didn't know if it would be automagically closed out15:39
vanhoofin any event, we're good15:39
apwvanhoof, no often it won't get picked up if its not known before we apply stable15:40
vanhoofapw: yeah I assumed this was the case, and mis-interpreted the test results that were posted as 'fixed in -proposed' :)15:40
bjf##15:56
bjf## Kernel team meeting today @ 17:00 UTC15:56
bjf##15:56
ckingi'm still confused by bug 774947 - do we need to test or not? the commit is in the -proposed kernel right?16:08
ubot2Launchpad bug 774947 in linux "[Lenovo Edge 11 AMD] system locks up completely running the "stress" tool" [Medium,Fix committed] https://launchpad.net/bugs/77494716:08
hertoncking: you don't need to test, yes the commit is on proposed16:19
bjf##17:00
bjf## Meeting starting now17:00
bjf##17:00
* bjf -> dr. appt.17:30
=== bjf is now known as bjf[afk]
=== cking is now known as cking-afk
=== manjo` is now known as manjo
keesapw: none of the hardy linux CVE fixes show up in the tracker with a specific version...18:04
apwkees, got an example18:04
keesgrep ^hardy_linux: CVE-2011-1170 CVE-2011-1171 CVE-2011-1172 CVE-2011-1173 CVE-2011-2534 CVE-2010-4649 CVE-2010-4073 CVE-2010-4238 CVE-2011-2484 CVE-2010-4165 CVE-2010-4249 CVE-2011-1010 CVE-2011-0711 CVE-2011-109018:04
ubot2kees: net/ipv4/netfilter/arp_tables.c in the IPv4 implementation in the Linux kernel before 2.6.39 does not place the expected '\0' character at the end of string data in the values of certain structure members, which allows local users to obtain potentially sensitive information from kernel memory by leveraging the CAP_NET_ADMIN capability to issue a crafted request, and then reading the argument to the resulting modprobe process. (http://18:04
ubot2kees: net/ipv4/netfilter/ip_tables.c in the IPv4 implementation in the Linux kernel before 2.6.39 does not place the expected '\0' character at the end of string data in the values of certain structure members, which allows local users to obtain potentially sensitive information from kernel memory by leveraging the CAP_NET_ADMIN capability to issue a crafted request, and then reading the argument to the resulting modprobe process. (http://c18:04
ubot2kees: net/ipv6/netfilter/ip6_tables.c in the IPv6 implementation in the Linux kernel before 2.6.39 does not place the expected '\0' character at the end of string data in the values of certain structure members, which allows local users to obtain potentially sensitive information from kernel memory by leveraging the CAP_NET_ADMIN capability to issue a crafted request, and then reading the argument to the resulting modprobe process. (http://18:04
ubot2kees: The econet_sendmsg function in net/econet/af_econet.c in the Linux kernel before 2.6.39 on the x86_64 platform allows remote attackers to obtain potentially sensitive information from kernel stack memory by reading uninitialized data in the ah field of an Acorn Universal Networking (AUN) packet. (http://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2011-1173)18:04
keesaaarggh18:05
ubot2kees: Buffer overflow in the clusterip_proc_write function in net/ipv4/netfilter/ipt_CLUSTERIP.c in the Linux kernel before 2.6.39 might allow local users to cause a denial of service or have unspecified other impact via a crafted write operation, related to string data that lacks a terminating '\0' character. (http://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2011-2534)18:05
ubot2kees: Integer overflow in the ib_uverbs_poll_cq function in drivers/infiniband/core/uverbs_cmd.c in the Linux kernel before 2.6.37 allows local users to cause a denial of service (memory corruption) or possibly have unspecified other impact via a large value of a certain structure member. (http://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2010-4649)18:05
ubot2kees: The ipc subsystem in the Linux kernel before 2.6.37-rc1 does not initialize certain structures, which allows local users to obtain potentially sensitive information from kernel stack memory via vectors related to the (1) compat_sys_semctl, (2) compat_sys_msgctl, and (3) compat_sys_shmctl functions in ipc/compat.c; and the (4) compat_sys_mq_open and (5) compat_sys_mq_getsetattr functions in ipc/compat_mq.c. (http://cve.mitre.org/cgi-bi18:05
ubot2kees: The vbd_create function in Xen 3.1.2, when the Linux kernel 2.6.18 on Red Hat Enterprise Linux (RHEL) 5 is used, allows guest OS users to cause a denial of service (host OS panic) via an attempted access to a virtual CD-ROM device through the blkback driver.  NOTE: some of these details are obtained from third party information. (http://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2010-4238)18:05
ubot2kees: The add_del_listener function in kernel/taskstats.c in the Linux kernel 2.6.39.1 and earlier does not prevent multiple registrations of exit handlers, which allows local users to cause a denial of service (memory and CPU consumption), and bypass the OOM Killer, via a crafted application. (http://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2011-2484)18:05
ubot2kees: The do_tcp_setsockopt function in net/ipv4/tcp.c in the Linux kernel before 2.6.37-rc2 does not properly restrict TCP_MAXSEG (aka MSS) values, which allows local users to cause a denial of service (OOPS) via a setsockopt call that specifies a small value, leading to a divide-by-zero error or incorrect use of a signed integer. (http://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2010-4165)18:05
ubot2kees: The wait_for_unix_gc function in net/unix/garbage.c in the Linux kernel before 2.6.37-rc3-next-20101125 does not properly select times for garbage collection of inflight sockets, which allows local users to cause a denial of service (system hang) via crafted use of the socketpair and sendmsg system calls for SOCK_SEQPACKET sockets. (http://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2010-4249)18:05
ubot2kees: Buffer overflow in the mac_partition function in fs/partitions/mac.c in the Linux kernel before 2.6.37.2 allows local users to cause a denial of service (panic) or possibly have unspecified other impact via a malformed Mac OS partition table. (http://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2011-1010)18:05
ubot2kees: The xfs_fs_geometry function in fs/xfs/xfs_fsops.c in the Linux kernel before 2.6.38-rc6-git3 does not initialize a certain structure member, which allows local users to obtain potentially sensitive information from kernel stack memory via an FSGEOMETRY_V1 ioctl call. (http://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2011-0711)18:05
ubot2kees: The __nfs4_proc_set_acl function in fs/nfs/nfs4proc.c in the Linux kernel before 2.6.38 stores NFSv4 ACL data in memory that is allocated by kmalloc but not properly freed, which allows local users to cause a denial of service (panic) via a crafted attempt to set an ACL. (http://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2011-1090)18:05
blissAAHHHHH18:05
keesapw: anyway, all of those CVEs are just "pending" without versions18:05
apwkees, hardy_linux: pending (2.6.24-29.92)18:05
* kees does an update....18:06
apwapw@dm$ grep ^hardy_linux: active/CVE-2011-117018:06
ubot2apw: net/ipv4/netfilter/arp_tables.c in the IPv4 implementation in the Linux kernel before 2.6.39 does not place the expected '\0' character at the end of string data in the values of certain structure members, which allows local users to obtain potentially sensitive information from kernel memory by leveraging the CAP_NET_ADMIN capability to issue a crafted request, and then reading the argument to the resulting modprobe process. (http://c18:06
apwpending without a version should mean Fix Commited but not uploaded yet18:06
blisskees: make sure that xfs fix comes with the fix to the regression it caused18:06
keesapw: oh, ha-ha, we're out of sync with you. fixing, please ignore me...18:07
* bliss bows his head in shame18:07
apwkees, np18:07
keesbliss: do you have a pointer for it?18:07
blissi'll find the commit18:09
keesbliss: do you mean CVE 2011-0711 ?18:09
ubot2kees: The xfs_fs_geometry function in fs/xfs/xfs_fsops.c in the Linux kernel before 2.6.38-rc6-git3 does not initialize a certain structure member, which allows local users to obtain potentially sensitive information from kernel stack memory via an FSGEOMETRY_V1 ioctl call. (http://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2011-0711)18:09
keesgah, I can't avoid the bot18:09
blissyes, that's the bug18:09
blissmy fix was broken18:09
blissi content their code was broken and my fix revealed it, but that's an argument for another day18:09
blisscontend*18:09
keesbliss: heh. it's just a memset though?18:09
blissyeah18:10
blissbut they cast a structure to a larger size in a function call18:10
blissso the memset caused an overflow18:10
kees*ew*18:10
blissyes, i wrote the patch, but that was clearly their fault18:10
apwthat rings a bell, we may have gotten both but worth checking18:11
keeshttp://git.kernel.org/?p=linux/kernel/git/torvalds/linux-2.6.git;a=commitdiff;h=af24ee9ea8d532e16883251a6684dfa1be8eec2918:11
keesbliss: yup, looks like we've got it.18:12
apwthose sha1s even look familiar18:12
apwyeah we have both marked in the tracker as needed, so we are good18:13
* kees nods18:13
keeshggdh: I have a question for you about q-r-t and updates... I want to add a test for something that hasn't been fixed yet. since we no longer fix CVEs in lock-step across all releases, I don't have a good way to avoid having the test fail. any thoughts on how to add a test that isn't a regression "yet"?18:38
hggdhhum18:40
hggdhthe easy way out: let it fail -- I always verify the results 18:40
hggdhanother option, bit more involved -- set up a table of fixed releases (and versions) and compare against18:41
hggdhkees: ^18:41
keeshggdh: yeah, we're going to need this going forward, so probably the table is the best solution. I will ponder how to best implement it. thanks!18:42
hggdhthe only thing is this is probably going to explode, too many CVEs against too many different packages :-(18:43
keesheh18:44
keeshggdh: well, we have some of the data in the CVE tracker, so I'm just going to ponder how to avoid duplicating it.18:44
hggdhif the cve tracker is API-accessible by anyone... 18:45
hggdhwe could just grab the data on run time. If not accessible, I am not sure it is a good idea18:46
keesjhunt_: hello!18:54
jhunt_kees: hi.18:55
keesjhunt_: so, we were trying to figure out exactly what the changed behavior was so we could reproduce it.18:55
keesjhunt_: without that, we were kind of stuff. we've been looking at another set of related changes, but I couldn't find fd/-specific stuf18:55
kees*stuff18:55
jhunt_well, to be clear, I'm not 100% sure what's happening yet. So, I'd really like to know what is expected behavior wrt a root process calling setuid and attempting to open /proc/self/fd/* directly.18:56
apwi'd have expected /fd/ to continue to work, as you already have those fds open18:57
* kees nods18:57
kees"self" should always be able to read its own fds I would imagine18:57
jhunt_I've got a noddy test program that attempts this and on lucid, the process can only open /proc/self/fd/0 whereas on natty+oneiric, you can open /proc/self/fd/[012]. Any fd higher than 2 gives EPERM.18:58
jhunt_apw: right. And I still think I might be going made, but the upstart session code that falls into this scenario *used* to work (on natty). however, I now cannot make the same code work on either natty or oneiric which made me think maybe a kernel or security change might be affecting things.18:59
jhunt_s/made/mad :)18:59
jhunt_(see, I'm mad me :)18:59
apwjhunt_, perhaps you could send us the test code so we can look it over18:59
jhunt_will do...19:00
apwkees, this fix:     UBUNTU: SAUCE: proc: hide kernel addresses via %pK in /proc/<pid>/stack19:00
apw    UBUNTU: SAUCE: proc: hide kernel addresses via %pK in /proc/<pid>/stack19:00
apwkees, whats the gen with that19:00
kees"gen" ?19:00
apwkees, the history, the background?19:01
apwit looks like a security related change, which we have from mainline in oneirc, backport from you in natty, but not in maverick19:01
keesapw: right, so, it's an upstream improvement for continuing to lock down things that should be hidden with %pK19:02
keesapw: it doesn't need backporting as far as I'm concerned. (i.e. I already backported it to natty which was the first the have %pK)19:02
* bliss cheers19:03
apwkees, ah ok, fair enough, i'll drop teh K in the backport before that then19:03
jhunt_apw, kees: code should with you now.19:04
keesapw: I'm not sure what you mean?19:06
keesjhunt_: thanks19:07
jhunt_kees: np19:07
apwkees, just trying to introduce a %pK too early is all19:09
keesapw: ah-ha, okay19:09
=== bjf[afk] is now known as bjf
* jjohansen -> lunch19:22
bjfogasawara, can you see if you can still get to the "site admin" page on voices.canonical.com? i'm not able to any more so no more blog postings for me19:23
ogasawarabjf: lemme check, just a sec19:23
ogasawarabjf: works for me, want me to post the meeting minutes?19:24
bjfogasawara, sure, i'll pastebin them 19:24
bjfogasawara, http://pastebin.ubuntu.com/647561/19:25
apwbjf what error you get ?  internal server error?19:26
bjfapw, You do not have sufficient permissions to access this page.19:27
apwbjf odder and odder19:27
ogasawarabjf: cool, posted19:27
bjfogasawara, thanks19:28
ogasawaraapw: weren't you getting some error as well?19:28
apwbjf, nice ... i am still getting internal server error whenever i try19:28
apwwe'll run out of people who can soon19:28
bjfapw, jjohansen is the community guy now, maybe only he can :-)19:29
apwbjf he will be so pleased to hear that19:29
=== bjf changed the topic of #ubuntu-kernel to: Home: https://wiki.ubuntu.com/Kernel/ || Natty Kernel Version: 2.6.38 || Ubuntu Kernel Team Meeting - July-26 - 17:00 UTC || If you have a question just ask, and do wait around for an answer!
jjohansenoh yeah so very pleased19:43
jhunt_apw, kees: just seen http://bit.ly/nRONfB which suggests the problem I'm seeing cannot be entirely caused by the /proc/self/fd/ behaviour. I'll do some more debug tomorrow and keep you posted...19:47
keesjhunt_: fwiw, even maverick shows this as a perm failure19:47
jhunt_kees: ok, thx.19:50
sconklinmanjo: I'm sorry I snapped at you yesterday.  20:01
manjosconklin, hey no worries man20:01
manjosconklin, reason I asked you the Q was ... the patch is in 3.1... I was not sure if there was a scheme to get patches down before it hit stable (which might take a few weeks)20:02
manjosconklin, I will push thro sru process once it gets in the 3.1 tree, its got acks from most mainitaners now 20:03
sconklinno, there's no scheme, and in general it's always better to just go ahead and get it out onto the mailing list instead of discussing it on IRC then having to rehash it in email20:03
manjoright .. will do 20:04
sconklinand I owe you a beer. Now, who wants to be snapped at next ;-)20:06
manjosconklin, :) no worries.. you don't 'owe' me... 20:06
hacksawI was just doing some maintenance, and therefore performing basic package updates on my 10.04 box. The explanation of why the kernel is being updated is "Bump ABI". Can anyone explain why I want to take my machine down as a result of this message? Does it mean something?20:44
apwhacksaw, that sounds like a bug in the changlog generator for update-manager ... whats the version you are being offered20:45
hacksaw2.6.32.33.3920:46
=== yofel_ is now known as yofel
bjftgardner, when do you want to get into the office?21:20
tgardnerbjf, I should be by 080021:20
tgardnerbe here*21:20
bjftgardner, sconklin is trying to setup a mumbe meeting, you see that?21:21
tgardnerbjf, yeah, but 0700 seems a bit early21:21
tgardnerbjf, speaking of mumble, I'd better make sure it works on my laptop. I've been having some problems.21:22
hacksawOkay, apw, I'm going to assume someone will look at the potential changelog bug, and make corrections in time.21:43
bjfogasawara, we are going to try to get into the office by 8:00 a.m., that work for you?21:51
ogasawarabjf: I'll probably come in a little later then since Kai gets up around 8am.  be in likely by 9am.21:55
bjfogasawara, sounds good21:55
=== kentb_ is now known as kentb-out

Generated by irclog2html.py 2.7 by Marius Gedminas - find it at mg.pov.lt!