/srv/irclogs.ubuntu.com/2011/04/12/#ubuntu-kernel.txt

hallynfeh.  btrfs just went readonly for no good reason01:00
RAOFIt's trying to save you :)01:01
hallynno doubt01:03
hallyni guess that rsync from a debootstrapped image into a subvolume was just a bit too fast01:03
RAOFHm.  Now that we run fsck.btrfs, booting takes a *loooong* time.  In an hour or so that system might be ready!01:17
Jerubg'day. this bug is marked as closed/fix-released: https://bugs.launchpad.net/ubuntu/+source/linux-firmware/+bug/63074804:19
Jerubbut i can replicate it on the same hardware as the original bug reporter's hardware with the very latest natty debs04:19
Jerubit looks like from the notes made by 'Tim Gardner' that he was told by an intel engineer that it should be fixed, but no report of anyone saying that any changes effected a change.04:28
jjohansenJerub: hrmm update the but with your information04:51
Jerubi did. i'm the last two comments on the ticket. but it's still marked as 'fix-released', instead of 'confirmed'04:52
Jerubi guess what i've said isn't very clear, i could spam the ticket more if you want.04:54
jjohansenJerub: it would be good, to be explicit has to which kernel (version # not latest), and what exactly you are seeing.05:16
jjohansenMaybe even doing an apport-collect 63074805:17
jjohansento add your information05:17
Jerubokay, doing that now05:18
Jerubi can't do apport-collect because the bug is closed.05:19
Jerubi've added specific version numbers and my uname -a to the ticket, and noted that i cannot run apport-collect due to the bug being marked as closed.05:22
=== _LibertyZero is now known as LibertyZero
=== smb` is now known as smb
magciusI have two support questions: one about FS not working, one about network not working. If there's a better place, please tell me :D09:26
magcius1. Sometimes the FS causes a complete failure where "sync" refuses to work.09:27
magciusIf I run "sync", and then do a cat /proc/$(pidof sync)/stack09:27
magciusI get "call_rwsem_down_read_failed" at the top of the stack. There is a fixed kernel bug about this, but it's still happening.09:28
magciusI have a somewhat borked external, and it's happened across two different hardware computers, so I'm not sure what's happening.09:28
magcius2. e1000e fails to work if I don't do proper shutdown when I reboot09:28
apwmagcius, what filesystem format is the drive wh09:29
apwwith the issue09:29
apwand with which kernel for that matter09:30
smbThe main problem with those kind of question is the lack of information. What fs, what drive, what release...09:30
magciusNTFS09:30
magciusI believe09:30
magciusI have no idea, the stack trace doesn't give me much info.09:30
magciusI have about 4 or 5 filesystems on this thing09:30
magciusI'm running Ubuntu 11.04 right now, but I've had this problem for a while.09:31
apwcan you get a picture of the problem?09:31
magcius"picture"?09:31
magciusAnd I'm trying to dig up the exact error message from e1000e, but not having much success since my dmesg log has been scraped09:31
smbI guess it is an external usb drive, but thats only guessing09:31
magciusIt was something like "probe failed with error code 5"09:31
magciussmb, indeed.09:31
magciusThe NTFS drive, at least.09:31
magciusI have no idea *if* it's that filesystem that's failing.09:31
smbThe stack only says that sync wanted a semaphore which is already taken09:32
magciusRight.09:32
smbThis can be a communication problem with the external usb case09:32
magciusOK.09:32
smbSo something tries to write and does not complete09:32
magciusRight.09:32
smbthen the next access just locks up waiting09:33
ohsixi've had fuse do some funny things like that :D09:33
magciuscould be fuse or ecryptfs09:33
smbgivien that it is ntfs there is likely fuse involved here too09:33
smbProbably one thing to look for is usb reset messages in dmesg09:34
magciusWhen it happens the next time, is there anything I can or should do to check for the filesystem?09:34
magciuswhat should I grep for?09:34
smbNot sure about the exact message but something about reset of usb device bla09:35
smbthen there could be the automatic softlockup message. maybe it triggers for the other process too. Though with writes going into the background those usually doent show09:36
magciusjust check /var/log/dmesg the next time it happens?09:37
smbmagcius, Its dmesg (as a command) 09:37
magciusIs there a difference? :P09:38
smbThat does not depend on the filesystem working. /var/log is the written output. dmesg directly reads the buffer09:39
magciusaha09:39
apwsmb, any feel for where in his cycle gregkh is?  for stable-38?09:50
smbapw, The was a larger dump between yesterday and today. He might be preparing to fire the next review/release09:51
apwgood09:52
fairuzHi, if I want to build packages, can I cross compile?10:33
apwmost packages in debian/ubuntu are not really cross compile capable10:36
apwthe kernel is partially cross-compilable10:36
fairuzapw: So the manual way is to cross compile the kernel, cross compile modules, and do modules_install on target machine? 10:37
apwno, i was trying to say that some of the build doesn't work, iirc the tools package doesn't build well in a cross compile environment10:38
apwwe use cross-compilation for build verification and architecture boot strappng so we haven't cared to fix these remaining issues10:38
magciusdebian multiarch?10:50
apw?10:50
magciusisn't that the cross-compilation work?10:50
apwnope, multiarch is cross arch installation10:51
magciushuh, I thought they were at least planning to do cross-compilation10:51
apwmultiarch is all about installation, as far as i know, it would be a handy tool as a basis for cross compilation but i don't beleive that is its function10:54
=== yofel_ is now known as yofel
=== diwic is now known as diwic_afk
bullgard4Is the Natty kernel thread aio generated thorough the source code file /usr/src/linux-source-2.6.38/fs/aio.c?12:24
bullgard4-o12:24
ohsixbullgard4: look for something that creates the kthread, like create_singlethread_workqueue12:26
bullgard4'~$ locate create_singlethread_workqueue' does not produce any output.12:28
tayyabali1how to learn kernel compilation any resources ?14:30
apwhttps://wiki.ubuntu.com/Kernel/BuildYourOwnKernel14:30
JFotayyabali1, https://wiki.ubuntu.com/Kernel/BuildYourOwnKernel14:30
apwjinx14:30
JFo:P14:31
apwJFo, not on mumble ?14:34
JFonot yet, charging my headset14:35
JForealized a bit ago that it was dead14:35
JFoshould be ready soon, it has been on a while14:35
hallynapw: fwiw on the kvm soft interrupt bug just got a more affirmative reply from avi, so he might be sending a patch soon.  14:39
hallyn(bug 747090 that is)14:40
apwhallyn, ok keep me in the loop, on the patch, sounds like its a nasty one ... any idea what the trigger for 'sometimes' is14:40
JFowhere in the heck is ubotu?14:41
hallynapw: i think the trigger is just when you notice :)14:41
apwneeds inviting in again i suspect14:41
hallynapw: it must always happen on softint14:41
apwahh that kind of sometimes14:41
hallynright.  for the most part 'who cares' if you do the interrupt twice :)14:41
apwi thought you could just invite ubottu, but it seems not14:43
JFohmmm14:43
smbI sort of had hoped that this one was "employed" by #is and would not leave with somebody leaving14:44
JFoit looks to still be in other channels14:44
* JFo goes to investigate14:44
smbhallyn, Hm, is that the two backport patches or something different? I was wondering about the other things just today14:45
hallynsmb: ?14:51
hallynsmb: you mean the kvm bug?14:51
apwhallyn, is that conversation on the original thread or elsewhere>14:51
hallynapw: in the kvm mailing list14:51
smbhallyn, I think I was messing things up in my memory. Yes, some kvm bug with clock going backwards, I just sent a nag mail round14:52
hallynsmb: <shrug>  i don't know enough about softints to know if it could cause that.  but it sure should cause some funky stuff14:52
apwhallyn, whats the subject on the contination ... the archives of the kvm list only show your original and one replay14:53
apwreply14:53
hallynsmb: so every 'INT 0xYY' will be called twice14:53
apwhallyn, now how is it that its twice and not thrice?14:53
apwor 10000 times14:53
hallynapw: Subject: Re: buggy emulate_int_real14:53
smbhallyn, Oh its surely not the only kvm bug. Just got reminded of the other one14:53
hallynapw: because I assume after the first time, regular instruction stepping happens.14:53
hallynapw: i did wonder that myself, and am not sure14:53
hallynall right, since there's no patch from him yet, i'll start one now14:54
apwahh its there on gmane not other archives, odd14:54
apwyeah it seems its something like if it page faults taking the int, it then runs the int again14:55
* ogasawara back in 2015:29
apwJFo, looking at the key bugs list we have a large number in New and Undecided status, arn't those supposed to get fixed by the scripts15:29
* apw grins at ogasawara 15:29
JFoapw, what do you mean?15:30
JFooh the arsenal scripts15:30
apwyeah15:30
apwat least the New bit?15:30
JFoI stopped running the 'new' one because it is improperly tagging some bugs as found by myself and bjf15:30
apwand i thought we were concentrating on ensureing there was no Unspecified priorities on that list15:30
JFohe has a new one that he is testing15:31
JFothat does the basics and gets them to confirmed15:31
apwok cool.  the priorities we should be fixing15:31
=== herton is now known as herton_lunch
JFoyes, indeed15:31
apwelse how can we tell which to care about15:31
JFoI just haven't gotten to those yet.15:31
JFothey are next on my list15:32
apwJFo, chat time ?15:32
JFosure, whenever you like15:32
moustafaHi, sorry to pop in, but I was wondering if it would be possible to backport the intel i915 video modules to Lucid?  I'm sure a number of users would enjoy the improvements to the driver code on their LTS machines15:50
apwmoustafa, we do offer lts backports kernels on lucid, offering the maverick, and soon the natty kernel for lucid15:51
apwthough they are only officailly supported for servers, they are complete15:51
moustafaapw: So, enabling the backports should technically allow a user to update to a more recent version of the kernel as well as the available open source drivers?15:52
apwmoustafa, they are separate from the -backports, they are elective installs via a separate meta package15:53
apwlinux-image-generic-lts-backport-maverick ... -natty (once natty releases)15:54
moustafaapw: Since I haven't tried this myself, is there any place I can read more about doing that?15:55
apwnow that is a good question, i am unaware of any speciic documentation other than the announcement15:55
apwJFo, can you remember if there is any specific docs on the LTS backports kernels?15:55
JFonot that I am aware of apw16:00
JFoand I can't find anything in the brief search I did16:00
moustafaapw , JFo so how would it work exactly?  just type in linux-image-generic-lts-backport-(name of release)?  Or am I missing something?16:01
moustafaI remember reading an annoucement, but didn't get the details of how far along it was16:01
JFoI recall the announcement, but I thought we documented the method16:01
JFoI'm just not finding it16:01
apwmoustafa, apt-get install linux-image-<flavour>-lts-backports-<release>16:03
JFofound the blueprint, but still no docs16:03
apwwhere flavour is the one your machine normally uses, and release is the release for the kernel16:03
moustafaapw JFo : Perfect, thanks 16:04
JFoapw, was this always the description for this blueprint? https://blueprints.launchpad.net/ubuntu/+spec/kernel-maverick-new-kernel-on-lts16:05
JFooh wait16:06
JFono, that is a feedback request or something16:06
apwthe description is right yes, just hidden16:06
JFoyeah, I just realized :)16:07
JFosorry about that16:07
bjf##16:07
bjf## Ubuntu Kernel Team Meeting - Today @ 17:00 UTC - #ubuntu-meeting16:07
bjf##      agenda: https://wiki.ubuntu.com/KernelTeam/Meeting16:07
bjf##16:07
=== Sarvatt is now known as Sarvatt|2
=== Sarvatt_ is now known as Sarvatt
apwbjf, yo ... i hear you have a new New bug frootle script for the arsenal ?16:08
bjfapw, for generating regression reports16:09
apwbjf not for handling the triage phase ?16:09
bjfapw, oh, right now it is just looking at bugs that have the required logs and bumping them from "New" to "Confirmed"16:10
bjfapw, it's not running via cron yet, but will be soon16:10
apwbjf, ahh ok, so that is the one i mean16:10
apwif they don't have the required logs will it ask?16:11
bjfapw, right now no, i'm taking it one step at a time, but that would be the next step16:11
apwbjf, ok cool, as long as its coming, as our bugs are getting in a mess :)16:14
bjfapw, tell me about it16:15
apwbjf, i would say, if it has the required logs, then it hsould be going Triaged shouldn't it ?16:15
apwconfirmed means more than one person has seen it, triaged means it has the needed data16:15
bjfapw, according to "our rules" we need to ask for "upstream testing" first, 16:16
apwoh i see ... hmm16:16
bjfapw, you need to look at our bug triage status wiki page16:16
apw(i try not too :))16:16
bjfapw, not sure i agree with what is says, just following it for now16:16
bjfapw, this is the page i'm referring to : https://wiki.ubuntu.com/Kernel/BugTriage/BugStates16:17
apwbjf, something odd is happening with apport ... which you might need to be watching out for ... like bug #75840016:17
apwhas needs-upstream-testing as a tag on it, but ... its never been requested in the bug16:18
apwand i think apport may have done it16:18
apw(and it may be an error in fact for it to have done so)16:18
bjfapw, http://people.canonical.com/~kernel/reports/24.html16:19
smbapw, FYI Review for 2.6.38.3 was more or less just sent out, ending on Thursday16:19
apwsmb, cool. that'd be good timing16:20
bjfapw, yes, those are apport, been doing it all along, you just haven't noticed16:20
bjfapw, we talked sunday about gutting our part of apport16:21
apwbjf, i think the now broken scripts also used that tag too, but either way i think that we should be reinforcing it via a direct comment in the bug16:21
apwas it would only have been in a dialog box and without any hint as to how to actually do it16:21
bjfapw, i'm pretty sure the scripts, when they add it, add a comment, apport does not16:22
apwright ... and i think it is really a mistake to have it in both places ... my opinion of course16:22
bjfapw, right, it needs to come out of apport and we need to be smarter about it in the scripts16:23
apwbjf, so where are you accumulating the new smaller leaner scripts ?16:24
bjfapw, me thinks we need someone on the team to "own" our part of apport16:24
* apw looks at you significantly16:25
bjfapw, wanted to talk to you about that, we said something about a structure under kteam-tools but I couldn't remember what we agreed on16:25
apwbjf, i think we were at that vague level yes16:25
bjfapw, i have it in a local kteam-tools under "bugs" directory16:25
apwbjf, i have one here which shortly will handle the 'New' and 'Incomplete (with response)' onces and puts them back into the right state16:25
apwbjf, that or arsenal seem logical to me16:26
bjfapw, have you looked at http://people.canonical.com/~kernel/reports/ ?16:27
apwnope, they are all new since i was off i think16:28
bjfyes16:28
bjfthey are running automatically via "cron" from the "kernel" user on people16:28
apwdo i have access there?  and are we going to run the arsenal there too ?16:29
bjfapw, you do have access (everyone on the team should)16:30
apwcool16:30
bjfapw, we can run there or on cranberry, the reports run there because they are public and people can talk to LP via the api16:31
bjfapw, my pref would be to do it all in one place16:31
apwseems an appropriate place to run everything then (to me)16:31
apwsnap16:31
* apw notes that this green against grey/black is not good to look at, its hard for me to see the text and the background on the same plane, the letters float oddly16:32
apwwhy are all our pages not the normal -verse16:32
bjfapw, you are the first to complain, others have like it and it works for the author16:33
* apw will have to persuade you to generate both -verses so i can view the other one :)16:34
* JFo reads back16:48
* smb wonders whether we got our meeting warning already17:00
smbOtherwise it might be over before we realize it started17:01
JFolooks like bjf got dropped17:01
smbUsual conference quality17:02
JFo:)17:03
jjohansenheh well the sever team looks to be using our meeting time any ways :)17:08
smbjjohansen, ? Isn't ours not due in an hour?17:09
jjohansensmb: oh cripes so it is, /me is so confused by daylight savings time still17:09
smbjjohansen, Wow I thought we now all changed for two weeks now. :-P I was somewhat surprised there had been a relative smooth transition17:10
jjohansensmb: well that doesn't mean /me is used to it.  /me still expects meets at 9am local and now they are at 10am.  By the time I get used to it, it will switch back17:11
=== herton_lunch is now known as herton
apwjjohansen, it does change only twice a year17:12
jjohansenapw: yeah I know, only 6 months to adjust :)17:13
apwbjf, how does one tell that a bug has had upstream testing ?17:15
apw(programatically)17:16
bjfapw, that's a really good question17:16
apwi suspect we cannot tell at the moment, as we ask them to remove needs-upstream-testing17:16
bjfapw, i don't have a good answer and am wondering if we can really blanket spam bugs17:16
apwhmmm, as i am already looking at the activity log to work out the 'previous state' i might be able to work out if it was on there and removed17:17
apwand use that as indicator17:17
bjfapw, but if they don't remove the tag but have done the testing what do you do ?17:17
apwbjf, how are you detecting 'has apport data'17:17
apwbjf well in that case the automation should assume they have done it, and it'll end up in a state saying we should look17:18
bjfapw, i'm looking to see if the original bug submitter has attached, specific logs17:18
apwand if we find it missing we ask again and re-add the tag17:18
bjfapw, bot comes along, changes status to "Incomplete", adds "needs-upstream-testing" tag and adds comment asking for upstream testing17:20
bjfapw, user does upstream attachment, adds comment, may not change from "Incomplete" to "Triaged/Confirmed" and may not remove "needs-upstream-testing" tag17:21
bjfapw, s/upstream attachment/upstream testing/17:21
bjfapw, is the bot going to look at all "Incomplete" status bugs to see if testing was done? how does it know ?17:22
bjfapw, if the status has been changed but the tag still exists, how do you know if the testing was done ?17:22
apwwell look at this way round ...17:23
apwi am looking at things which were imcomplete and want to change, ie incomplete (with response)17:23
apwi would think it should occur as part of that processing17:23
apwas from there it should go to 'new' (with no data), 'confimed (with apport data), or triaged with apport and with testing done17:24
apwso the trigger to care is incomplete (with response) or 'new'17:24
apwas those are the only places it can go to if they say something17:24
apwbjf mumble ?17:25
bjfat ELC17:25
apwoh yeah, no chance17:25
apwas you say we cannot rely on moving or not moving to New17:25
apwbut we can handle that i think17:25
bjfi think your approach is the right one, i'm just wondering how a bot is going to look at the last comment and decide that the testing was done17:26
apwand i thiknk if they test but don't remove the tag (which we explicitly ask them to do) then we lose them, and they get expired17:26
apwi think we only look at the absence of the tag17:26
apwie, was it added and then removed17:26
apwie. was it on there, and is not now17:27
apwthat says they removed it17:27
apwif the history says it was tagged, and bug.tag('needs....') == false17:27
bjfi was thinking the same, but concerned that relying on users to remove tags is problematic17:27
apwit is not clear how we can do it using text17:28
apwas there is no common way to say it17:28
apwthe text the script inserts says how to remove the tag etc, JFo can confirm i think17:28
bjfbottom line, do it the way you are thinking (which i agree with) but we need to keep an eye on it to see if it's "working"17:28
apwclick the pencil, remove the text 'needs....'17:29
apwyep, so i think if they have either apport data, or have removed the tag (ie done upstream testing) but not both we go to Confirmed, if both then to Triaged17:29
apwand keep an eye on things lingering in Confiirme17:29
bjfagreed17:30
apwdo you have a function to determine if we have apport data I can steal ?17:30
* apw notes we need a library of useful snippets to share shortly17:31
bjfyes17:31
bjfwill push soon17:31
bjfworking on pushing, right now17:31
apwbjf, i suspect that this processing i am doing right now is so allied with the 'ask for data' and 'ask for upstream testing' we are going to end up merging them17:32
apwbut ... anyhow :)17:32
apwwe can likely work out how long we have been waiting for needs-xxx to be removed, and add comments to hastle people17:33
* JFo curses xchat17:37
bullgard4What does "MRU" stand for in  /usr/scr/linux-source-2.6.38/fs/xfs/xfs_mru_cache.c?17:38
bjfapw, pushed, look at bugs/proc-new-regressions   and ktl/bugs.py17:39
apwbullgard4, (guesses at) most-recently-used ?17:39
bullgard4apw: hm17:40
JFoI think for the test to see if they actually did test upstream, it may come to a point where we have a script compile for me a list of bugs to look at. I am not above that.17:40
JFoin fact, it will make what I need to be doing a ton easier17:40
apw * inserted at the head of the current most-recently-used list.17:40
JFostill thinking about the 'keeping an eye on confirmed17:41
apwbullgard4, looks like that is what it is17:41
JFo'17:41
JFoI have an opinion, but it isn't making a lot of sense right now. :)17:41
sforsheehas anyone noticed this thread about bluetooth borkage: https://lkml.org/lkml/2011/3/19/8017:42
sforsheesomeone sent a backport to the stable list today, maybe we want to get it into natty17:42
sforsheeunfortunately I have nothing to test with17:43
apwsforshee, is that the version of bluetooth we use?  there are two and i can never remember which is which17:43
apwmanjo normally knows17:43
bullgard4apw: It seems your guess is correct: http://en.wikipedia.org/wiki/Cache_algorithms17:43
sforsheeapw, had no idea there were two versions, and also no idea which one we use17:44
apwsforshee, seems we do use bluez17:46
sforsheeapw, that's the conclusion I was coming to as well17:46
sforsheeI see one bug that might be related17:46
sforsheehttps://bugs.launchpad.net/ubuntu/+source/bluez/+bug/70029217:46
apwsforshee, thanks, adding a linux task in case17:48
bjf##17:55
bjf## Kernel team meeting in 5 minutes17:55
bjf##17:55
* JFo stretches17:55
ppisatio/17:56
* apw thinks ppisati missed17:59
smbapw, me too18:00
ppisatiuat?18:01
ppisatiops, wrong channel :)18:01
=== sforshee is now known as sforshee-lunch
JFo<- lunch18:07
* apw goes find some dinner ...18:48
=== sforshee-lunch is now known as sforshee
GrueMasterJFo: What more data would you like on bug 758961?  The oops text is there and the kernel devs that actually have these platforms can easily reproduce this.19:47
JFowell, the norm is to gather all of the apport-collect data, that way there is (usually) no need to ask for anything further.19:49
* jjohansen -> lunch20:03
skaetJFo, ogasawara - could you have a look at https://launchpad.net/bugs/759115?20:43
skaetits hitting in Ubuntu Alternate and Edubuntu images.20:44
ogasawaraskaet: hrm, and I assume this wasn't the case for Beta 120:45
JFough, LP taking forever for me :)20:46
skaetogasawara, discussion going on in u-release, looks like security update early this month.20:46
lifelessJFo: doing what ?20:48
JFolifeless, it is a connection issue for me, not LP's fault20:48
JFo:)20:48
lifelessJFo: well, that could because of our ISP or whatever choosing poor routes20:49
JFoI'm pulling an ISO at the same time, so my connection is slow20:49
ogasawaraskaet: Author: Kees Cook <kees@ubuntu.com>20:51
ogasawaraDate:   Wed Mar 23 13:17:13 2011 -070020:51
ogasawara    UBUNTU: [Config] packaging: adjust perms on vmlinuz as well20:51
ogasawara    20:51
ogasawara    Since kernel symbols are resolvable internally to the kernel, the kernel20:51
ogasawara    itself has a map of the symbols. Continuing the tradition of frustrating20:51
ogasawara    off-the-shelf kernel exploits, make vmlinuz unreadable for non-root, just20:51
ogasawara    like has been done for System.map, etc.20:51
ogasawara    20:51
ogasawara    Signed-off-by: Kees Cook <kees.cook@canonical.com>20:51
ogasawara    Signed-off-by: Tim Gardner <tim.gardner@canonical.com>20:51
keesI thought alternate was already fixed?20:52
keesstgraber is working on edubuntu, iiuc20:52
skaetogasawara, yup.  LTSP impacted in alternate and edubuntu.20:52
keesskaet: stgraber is working on it (was just talking to him about it in #ubuntu-hardened)20:53
skaetkees,  yup,  he started the discussion off in ubuntu-release, and the question of "why" came up.20:53
keesskaet: for "frustrating off-the-shelf kernel exploits"20:54
stgraberkees, skaet: The easiest, most maintenable and fastest way for me to fix it is to release a new LTSP upstream. I can have one released with a fix in 5 minutes or so. Other upstream diff is bugfix only and minimal.20:54
skaetkees:  lol,  indeed.20:54
keesskaet: less tersely, it is to stop non-root users from being able to trivially resolve kernel symbols when performing local kernel exploit attacks.20:55
keesskaet: it doesn't stop a dedicated attacker, but it does stop script-kiddies, which has real-world benefits.20:55
skaetstgraber,  ok,  let pitti know its coming and why. 20:56
* skaet crosses fingers bug fixes only means bug fixes and no regressions.20:56
skaetthanks kees20:57
keesskaet: np, sorry for the trouble!20:58
ogra_lets just tell the script kiddies to grow up :)21:01
JFoI wish21:05
gondoiI seem to be missing my quota modules in -virtual22:04
gondoiis this expected?22:04
gondoibtw, I'm running maverick22:05
=== emma_ is now known as em
=== BenC__ is now known as BenC
=== sconklin is now known as sconklin-gone

Generated by irclog2html.py 2.7 by Marius Gedminas - find it at mg.pov.lt!