[00:00] <jeeves_Moss> can someone send me a pastebin of a name baised config file please?  for some reason my mind has taken a vacation, and I can't for the life of me figure out WTF is going on
[00:01] <lil_cain> ;named based config for what?
[00:02] <jeeves_Moss> lil_cain, I'm trying to get my v-hosted apache2 install going
[00:03] <lil_cain> ah. I'm no help I'm afraid.
[00:04] <lil_cain> I try and avoid apache where possible.
[00:04] <jeeves_Moss> no worries
[00:04] <jeeves_Moss> lol
[00:04] <jeeves_Moss> well, I think webmin is just messin' up the configs.
[00:05] <lil_cain> !webmin
[00:05] <lil_cain> :-)
[00:06] <jeeves_Moss> yea yea yea
[00:12] <cn1109> I need ideas in terms of outsourcing or in house hosting. I'm running a web server (e-commerce site) which is using Joomla. Traffic is high and many of the in house software (customer service, finance, etc) connects to the internal mysql server which also is used for the public site. My concern is if I should outsource both the web and sql server but again how will this impact the CAP placed by the hosting company. The personnel in th
[00:12] <cn1109> office does use the mysql connection often.
[00:13] <cn1109> so. Should I go with ubuntu server or could computing through Amazon. Any ideas?
[00:13] <erimar77> cn1109: are you setup for disaster recovery, if not, what's that worth to you
[00:13] <erimar77> cn1109: what's that worth to your customers
[00:15] <cn1109> Well..It seems that the company has bot had an IT guy in house for a while. They have many open source OS's that in reality are not needed. Monowall is an example. The customers depend on the site to be up at all times. I would say that by using ESXi and the Ubuntu server edition we would be okay. Again we could be saving some money by focusing on cloud computing and reduce the maintenance cost of in house servers.
[00:17] <lil_cain> I'd be shocked if maintenance costs for servers were higher than costs for AWS.
[00:18] <cn1109> lil_cain: so what would you recommend?
[00:18] <erimar77> cn1109: depends on who you want to blame during outages
[00:19] <cn1109> erimar77: that would be me :)
[00:19] <erimar77> cn1109: not if you threw it all on a hosted server
[00:20] <cn1109> erimar77: true. Just concerned about the bandwidth cap
[00:20] <31NAAPG5D> has anybody use DEltacopy to make bakups from a windows client to an ubuntu server?
[00:21] <erimar77> cn1109: how much bandwidth are you talking about, it sounds like you already have a host chosen
[00:21] <erimar77> 31NAAPG5D: whats your issue
[00:21] <cn1109> erimar77: we are hosting in house with a Sprint T1 line. We are also paying for multiple Ip's we really do not need.
[00:22] <erimar77> cn1109: how much would you save by getting rid of all that, would it justify getting a higher bandwidth cap
[00:23] <cn1109> it's about 1000 a month for internet. Electricity which comes for the AC unit and powering on the servers. That would eliminate a good amount.
[00:23] <31NAAPG5D> erimar77: i whant to make automatics bakups of a windwos folder to a ubuntu server..i am a newbee and have look for a how to and no success
[00:24] <cn1109> rsync?
[00:24] <erimar77> 31NAAPG5D: what are you trying to accomplish?
[00:24] <erimar77> 31NAAPG5D: ok
[00:25] <erimar77> 31NAAPG5D: from the windows comptuer can you use an app like putty to ssh to the ubuntu server?
[00:26] <31NAAPG5D> erimar77: i have at present moment the folder shared and have the folder mount in ubuntu and resync makes the trick very nicely, the problem is that the shared folder on windows is accesibel buy everybody in my network, that is how i heard about deltacopy./....yes i have access thru putty
[00:28] <erimar77> 31NAAPG5D: it sounds like you have it a bit backwards, you shouldn't have to mount the shared windows folder in ubuntu for it to work
[00:30] <31NAAPG5D> erimar77: well you might be rigth since i am a newbee...but so far works nice..probelm is that the share folder is public to everybody..let me show you how i have it
[00:32] <31NAAPG5D> erimar77: //192.168.1.106/Charts /home/rgotten/charts.windows   cifs  credentials=/root/.credentials,iocharset=utf8,file_mode=0777,dir_mode=0777 0 0
[00:33] <erimar77> 31NAAPG5D: give me a sec to install deltacopy
[00:34] <31NAAPG5D> erimar77:
[00:34] <31NAAPG5D> erimar77: thanks
[00:38] <31NAAPG5D> erimar77: i have to take care of somebody in my office , will be back in 30 minutes..it is ok?
[00:39] <erimar77> 31NAAPG5D: i think i'll be around
[00:40] <31NAAPG5D> erimar77: thanks
[00:42] <SirStan> Where can I download ubuntu server directly?
[00:42] <SirStan> vs using the form on ubuntu.com that doesnt work in ie6 : (
[00:42] <erimar77> SirStan: 32 or 64 bit
[00:43] <SirStan> 32, 8.04 lts
[00:43] <SirStan> http://mirror.site.com/directory/ is fine
[00:44] <erimar77> http://mirrors.us.kernel.org/ubuntu-releases/8.04.4/ubuntu-8.04.4-server-i386.iso
[00:44] <SirStan> thx
[00:45] <erimar77> SirStan: if that mirror is slow, http://www.ubuntu.com/getubuntu/downloadmirrors#mirrors
[00:48] <njbair> does ubuntu server cache http output by default?
[00:48] <SirStan> its a little slow.. but 1mb is ok
[00:48] <SirStan> im not in a rush
[01:16] <smoser> hggdh, bug 564924 confirmed here.
[01:22] <RoAkSoAx> smoser, the puppet integration with EC2/UEC is only to register the Instances running on the cloud automatically, correct?
[01:23] <smoser> RoAkSoAx, best to ask mathiaz.
[01:23] <smoser> http://ubuntumathiaz.wordpress.com/2010/03/25/using-puppet-in-uecec2-automating-the-signing-process/
[01:23] <smoser> http://ubuntumathiaz.wordpress.com/2010/03/24/using-puppet-in-uecec2-puppet-support-in-ubuntu-images/
[01:23] <smoser> i must admit to knowing little about it.
[01:23] <RoAkSoAx> smoser, oh ok, thought the cloud-init package was part of this. So what is the cloud-init package for exactly?
[01:25] <smoser> RoAkSoAx, well, cloud-init enables it
[01:25] <smoser> cloud-init provides some hooks, the puppet takes advantage of that and does some puppet-y things for it.
[01:26] <smoser> http://ubuntu-smoser.blogspot.com/ is an entry i wrote about cloud-config
[01:26] <smoser> mathiaz's work provides cloud-config syntax to get puppet up and running
[01:27] <smoser> cloud-init has other functions other than just cloud-config though, it can allow you to hook into the boot process very early.
[01:27] <smoser> from your provided user-data
[01:27] <RoAkSoAx> smoser, I see.
[01:27] <RoAkSoAx> smoser, I'll take a look to your blog post and bug mathiaz about it then  :)
[01:27] <smoser> i need to write another blog entry about what else it does
[01:29] <smoser> hggdh, i'm going to run a single_config job
[01:29] <RoAkSoAx> smoser, will be waiting for it :)
[01:29] <smoser> alright :)
[01:29] <smoser> in the meantime, there is doc (admittedly not well exposed)
[01:30] <smoser> http://bazaar.launchpad.net/~cloud-init-dev/cloud-init/trunk/files/185?file_id=doc-20100127032511-5qfkb7qsglz0tfnn-1
[01:30]  * RoAkSoAx reads
[01:30] <smoser> hggdh, one featur request for that. i'd like the output log to have what emi- was used
[01:30] <smoser> (possibly with its manifest name, to give some human readable context)
[01:44] <minitux> ??
[02:17] <airliasdesign> hello
[02:17] <airliasdesign> is anyone here?
[02:18] <lamont> that depends on your definitions
[02:19] <lamont> the best approach is to just ask your question, instead of asking if you can ask your question.
[02:20] <airliasdesign> I'm looking to run high capacity web servers, but wasn't sure if I should go with a private Amazon EC2 cloud or to just get the most powerful server I can
[02:53] <hggdh> smoser: I will print out the emi code at the start, then
[03:09] <hggdh> smoser: first print-out line states the emi in use. I am running a new battery now, and will check on results tomorrow morning
[03:23] <MTecknology> True or false? Ubuntu's version of kickstart allows you to pass "pre-­seeding" parameters to the debian installer.
[03:24] <MTecknology> Is that False? I thought it was True
[03:24] <MTecknology> I know two of the answers I got wrong and I can't figure out what the third is :(
[03:49] <smoser> hggdh, thanks
[03:51] <smoser> hggdh, well, it seems "fixed" with my changes, now its getting "cant find metadata service", just like dustin and I see.
[03:52] <twb> MTecknology: Ubuntu uses Debian's "preseeding" technique.  This is ortogonal to RHEL's "kickstarting", which was ported to Ubuntu (and Debian) more recently.
[03:52] <twb> *orthogonal
[03:53] <twb> Although that's not really the right word; preseeding and kickstarting achieve the same result through different implementations.
[03:53] <smoser> thats a tricky question
[03:54] <MTecknology> twb: oh - so the correct answer was false - because the same affect is acheived differently?
[03:54] <twb> MTecknology: The question is ill-defined.
[03:54] <smoser> i'd have to look at the implementation to be sure
[03:54] <twb> MTecknology: so really the best answer is 無
[03:54] <smoser> but i had always just assumed that kickstart config came in and was translated to preseed
[03:54] <MTecknology> twb: ok - it was on the serverassessment for UCP
[03:54] <smoser> which would mean 'true'
[03:54] <smoser> but i've never looked at it
[04:32] <bluethundr> I just unmounted a volume known as /dev/sd2a and created a partition on it with fdisk. but when I try to format it with mkfs.ext3 it reports that the partition I created isn't there even tho fdisk reports that it is. I made sure to reboot after using fdisk
[04:32] <bluethundr> http://pastebin.com/a3XQ3EPM
[04:34] <cn1109> is it worth install webmin?
[04:35] <maginot> does ubuntu servr install prompt for a root password on install screen or I'm just confusing with something else?
[04:36] <cn1109> maginot: It did not for me. Asked me for a new username and password
[04:39] <cn1109> so...has anyone used webmin or something similar?
[04:44] <twb> cn1109: we do not approve of web-based sysadmin wrappers here.
[04:44] <cn1109> twb: okay. thx
[04:44] <twb> cn1109: if you MUST use one, you should know that Ubuntu nominally supports ebox and definitely doesn't support webmin at all.
[04:46] <twb> bluethundr: are you mis-typing /dev/sda2?
[04:46] <twb> bluethundr: ah, I see what you're doing.
[04:47] <twb> bluethundr: /dev/sda is a disk.  /dev/sda2 is a partition ON that disk.  By calling fdisk on /dev/sda2, you've created a partition on a partition, which linux/udev won't create a device file for by default.
[04:47] <bluethundr> twb: no, actually I am copying / pating
[04:47] <bluethundr> s/pasting/pating/
[04:47] <bluethundr> ah ok
[04:47] <twb> bluethundr: in the unlikely case that you actually want a partition table on a partition, you can create a device for it using losetup.
[04:48] <bluethundr> twb: good point, thank you so much for the insight
[04:48] <twb> twb knows all!  Tell your friends!
[04:48]  * bluethundr will!!! :-)
[04:53] <bluethundr> hmmm.. I can't seem to fdisk /dev/sda .. but fdisk /dev/sda2 works
[04:53] <bluethundr> Unable to open /dev/sda
[04:54] <twb> Who owns those devices?
[04:54] <twb> Are you running these commands as root?
[04:55] <twb> You should probably be using GNU cfdisk or parted rather than fdisk.
[05:06] <bluethundr> ok, I just attached an S3 bucket to my Ubuntu 6.06 server on AWS.. I am able to run fdisk on this volume and create partitions, but mkfs claims this volume is in ue even tho I can't see it with df -h
[05:06] <bluethundr> /dev/sdf1 is apparently in use by the system; will not make a filesystem here!
[05:08] <Hitiek> can't ssh into machine. can't log in to physical terminal. sudo hangs. existing vnc, screen, irssi all still work. apache is fine. samba share doesn't seem to work.
[05:08] <Hitiek> any ideas what would cause all that?  happens once every few months.  I end up rebooting and everything is fine again for a while.
[05:10] <bluethundr> I get the same result with cfdisk as with fdisk and I am running the commands as root
[05:11] <twb> bluethundr: 6.06 is too old for me to support
[05:13] <bluethundr> k
[05:24] <arrrghhh> hey does anyone use their server to play music out of the local soundcard?  like with MPD?
[07:27] <nhawdge|work> Hey anyone got a moment to answer some questions about hard drive recovery?
[07:40] <SandGorgon> does anybody know if /etc/syslog.conf is even used  (9.10 onwards) ?
[08:25] <_ruben> SandGorgon: sysklogd has been replaced by rsylogd, and thus /etc/rsyslog.conf is used instead
[09:00] <jiboumans> morning folks
[09:52] <lau> ping jdstrand
[10:57] <lau> hello on karmic, if I run /usr/sbin/mysqld --defaults-file=/etc/mysql2/my.cnf --print-defaults I got no parameters
[10:57] <lau> but cat /etc/mysql2/my.cnf works ok, any idea ?
[12:29] <maxagaz> how to check the procs (number, speed, type) of my server ?
[12:29] <alvin> maxagaz: sudo lshw
[12:30] <lil_cain> if it's just the processer you're looking for, cat /proc/cpuinfo
[12:30] <lil_cain> easier to read, and doesn't require root.
[12:32] <maxagaz> alvin, cat /proc/cpuinfo works fine too
[12:33] <maxagaz> alvin, thanks
[12:34] <maxagaz> is it possible to know the brandt of the machine from the console ?
[12:34] <alvin> you're welcome (/proc/cpuinfo is one of the places lshw looks)
[12:34] <lil_cain> not in any portable way that I've ever heard of.
[12:35] <alvin> Yes, lshw :-) (look for 'vendor')
[12:36] <persia> Depends on the hardware: that works for some vendors, and not for others.
[12:36] <persia> Tends to work more poorly for white-label stuff.
[12:37] <alvin> Probably. I'm in luck. (vendor: transtec AG)
[12:41] <lau> maxagaz: you can try dmidecode too
[12:41] <persia> I get mixed results.  One of my two apples lists Apple, the other doesn't.  My Panasonic only lists Intel and Ricoh.
[13:47] <Italian_Plumber> I am searching for an ubuntuforums thread regarding upgrading a server from hardy to lucid. If one exists, I cannot find it.  I'm looking specifically for any discussion regarding recommending not upgrading because hardy is ext3 and lucid is ext4 (by default, I assume).  I have found this post http://ubuntuforums.org/showpost.php?p=8813514&postcount=5 but nothing more.  Do my searching skills suck or should I start
[13:52] <Pici> Italian_Plumber: What sort of issues are you expecting to see from a filesystem change?
[13:53] <Pici> Or rather, and even then it won't be switching filesystems when you upgrade.
[13:53] <alvin> Italian_Plumber: I'm pretty sure ext3 will not be automatically converted to ext4.
[14:06] <Italian_Plumber> I don't know; I just saw this post and panicked. :)
[14:06] <smoser> ttx, copied you on bug 566792
[14:22] <Thkx> hi
[14:25] <alvin> Italian_Plumber: I wouldn't panic. Lucid isn't 'made for ext4'. I do hope ext4 has good support, but ext3 will most certainly be at least equally well supported. There are far more worse things that can happen during or after the upgrade than staying on ext3.
[14:25] <Thkx> i have a question...i have necessity to make a server to manage 3 different email account (for example, @gmail.com, @hotmail.com, etc)...and sort email to each account...how can i do this?postfix + fetchmail?
[14:28] <ttx> smoser: thanks
[14:32] <zul> mathiaz: fyi mysql-5.0 is gone from universe
[14:33] <ttx> smoser: I think you make a good case of pinpointing a bug in eucalyptus, let's see how they react
[14:33] <ttx> smoser: any reason to keep the cloud-init task opened ?
[14:33] <smoser> probably not, since i can't even successfully work around it
[14:33] <smoser> :)
[14:33] <smoser> i tried
[14:34] <ttx> smoser: you reproduce it every single time, or need specific situation ?
[14:35] <smoser> timing
[14:35] <smoser> luck
[14:35] <ttx> ok
[14:37] <ttx> team: please do some ISO testing coverage
[14:47] <zul> ack
[14:49] <zul> ttx:i was going to say only if you say please but you did ;)
[14:53] <smoser> ttx, i will start ec2 tests
[15:04] <zul> ill start the i386 tests
[15:17] <hggdh> ttx: do we need to go thru the UEC tests?
[15:28] <ttx> hggdh: I'll cover the ISO / UEC tests
[15:30] <hggdh> ttx: the point is we have been testing it. Continuously
[15:31] <ttx> hggdh: right
[15:38] <ttx> hggdh: I mean, I'll cover the strict "install from ISo" testcases, you should concentrate on feature/stress testing
[15:38] <ttx> as it showed some problems recently
[15:40] <hggdh> ttx: ack
[15:41] <ttx> smoser: targeted the metadata service issue to lucid, added to watch list
[15:43] <xperia> hello to all. i have installed dancer-ircd but it looks like i have some permissions problems. most of the files and dirs have dancer as a user for accessing the files and dirs
[15:43] <xperia> but i dont know as what user dancer-ircd is executed
[15:43] <xperia> and what for permissions i need exactly
[15:44] <xperia> how can i prove hat such a user "dancer" with limited access exist on my system
[16:04] <JanC> xperia: grep dancer /etc/passwd
[16:05] <xperia> JanC: great thank you. it looks like i have them as users
[16:08] <xperia> JanC: what could be the problem that dancer-ircd report afterthe installing that the MOTD file is missing
[16:08] <xperia> "[16:09] <xperia> somehow dancer-ircd report allways after the install that this file is missing even if the file exist
[16:09] <JanC> maybe the permissions are wrong?  or you spelled the file name incorrect (case-sensitive!)?
[16:12] <eric_f> Can someone point me to a link that describes why EC2 AMIs have an ubuntu user and how that differs from root?
[16:22] <gzmask> Are there any network configuration information/wiki/tutorial for UEC?
[16:22] <hggdh> ttx: bug 565101 sounds bad
[16:26] <ttx> I'm more concerned about the SSh connection failures, which seem to be getting worse as our images boot faster
[16:26] <ttx> I suspect tat's what bug 566792 is about
[16:29] <ivoks> shouldn't kvm-pxe put files into /usr/share/qemu, instead of /usr/share/kvm?
[16:32] <ivoks> it should.
[16:32] <zul> oh bloody hell
[16:33] <ivoks> :)
[16:34] <smoser> ttx, i think you are correct about fast boot causing it
[16:35] <ttx> smoser: it's getting worse as we improve
[16:35] <smoser> but what doesn't seem to collaborate is that attempts to just wait and re-try dont fix
[16:35] <ivoks> soren: welcome back :)
[16:35] <ttx> smoser: but you still get a status 200 ?
[16:35] <smoser> ttx, right.
[16:36] <ttx> an empty status 200 is plain wrong
[16:36] <smoser> its as if an early attempt to get at the metadata service breaks it
[16:36] <smoser> obviously, yes
[16:36] <smoser> but it doesn't recover.
[16:36] <smoser> ie, you get that once, wait 20 seconds, try again, its not fixed
[16:36] <ttx> yes, so it's not just "we ask too early"
[16:36] <smoser> its not like its populating
[16:36] <smoser> and we were just too fast.
[16:36] <ttx> it's we break it by asking too early
[16:36] <smoser> right
[16:37] <ttx> kirkland: could you coordinate a call with eucalyptus and smoser, whenever the west coast wakes up ?
[16:37] <ttx> we need to have them look into that issue.
[16:37] <kirkland> ttx: nurmi and I have a daily SCRUM call in 20 minutes
[16:37] <kirkland> ttx: nurmi and I are doing daily SCRUM calls through Lucid release date
[16:38] <ttx> kirkland: please introduce that issue
[16:38] <kirkland> ttx: okay
[16:38] <ttx> that's quite a show stopper.
[16:56] <eolo999> hi, i'm trying to install ubuntu server LL on a Dell R410 but i cannot boot the root partition which sits on lsi logic 1068e raid controller. Anyone had some hints?
[17:02] <eolo999> A*Anyone faced the same issue?
[17:03] <lil_cain> I've had problems with hardy on Dells before.
[17:03] <lil_cain> Try dropping to a shell halfway through the install, and installing the generic kernel
[17:03] <lil_cain> (rather than the server kernel)
[17:11] <slipper1024> hi Guys
[17:16] <slipper1024> configuration: Ubuntu server 8.04 LTS, running ispconfig, courier pop, postfix mta, dovecot sasl user authentication, self signed certificate for mail.          issue: Some hosting clients get relay access denied when sending mail, while successfully authenticated. Over TLS, SSL and Login same problem. 1 Shared IP, multiple VHosts.
[17:17] <slipper1024> [05:05:20 PM] Mike Hyland: Client host rejected: Access denied', Port: 465, Secure(SSL)
[17:19] <slipper1024> The message could not be sent because one of the recipients was rejected by the server. The rejected e-mail address was 'somebody@gmail.com'. Subject 'test', Account: 'mail.thisdomain.com', Server: 'mail.thisdomain.com', Protocol: SMTP, Server Response: '554 5.7.1 <unknown[196.35.255.4]>: Client host rejected: Access denied', Port: 465, Secure(SSL): Yes, Server Error: 554, Error Number: 0x800CCC79
[17:20] <slipper1024>  'Somebody' on 2010/04/19 07:53 PM
[17:20] <slipper1024>             554 5.7.1 <vc-41-3-207-163.umts.vodacom.co.za[41.3.207.163]>: Client host rejected: Access denied
[17:21] <slipper1024> /var/log/mail.warn output:
[17:22] <slipper1024> Apr 20 18:09:49 myhost postfix/smtpd[11155]: warning: 190.26.17.145: address not listed for hostname adsl190-2617145.dyn.etb.net.co
[17:22] <slipper1024> Apr 20 18:11:26 myhost postfix/smtpd[11155]: warning: 201.86.222.79: address not listed for hostname 201.86.222.79.static.host.gvt.net.br
[17:22] <slipper1024> Apr 20 18:12:08 myhost postfix/smtpd[11155]: warning: 62.41.253.74: hostname static.kpn.net verification failed: Name or service not known
[17:22] <slipper1024> Apr 20 18:19:01 myhost postfix/smtpd[13459]: warning: 88.247.68.249: hostname dsl88-247-17657.ttnet.net.tr verification failed: Name or service not known
[17:23] <slipper1024> Is there anybody that can assist me, have tried everything?
[17:23] <ivoks> let's see
[17:24] <slipper1024> sorry for all the info, hopefully the more the merrier
[17:25] <ivoks> slipper1024: you have unrelated stuff here
[17:26] <ivoks> first one is, i guess, client pasting outlook issue?
[17:26] <ivoks> s/issue/message
[17:26] <ivoks> not here? ok...
[17:28] <ivoks> doh
[17:31] <kirkland> hggdh: hiya
[17:32] <kirkland> hggdh: i think i might have a solution for your heap/memory errors
[17:32] <kirkland> hggdh: how easily can you reproduce that one and test a fix?
[17:34] <ivoks> i love libvirt
[17:34] <ivoks> :)
[17:34] <ivoks> am i the only one? :D
[17:35] <kirkland> ivoks: there's a few of you
[17:36] <RoAkSoAx> i love it too
[17:36]  * RoAkSoAx hates packing and moving out
[17:37] <ivoks> virt-install --help outputs strange manual :)
[17:37] <ivoks> oh, lol, it's broken
[17:42] <hggdh> kirkland: usually after about 300 runs, so ~30min or so
[17:42] <hggdh> smoser: can I use the rig?
[17:42] <kirkland> hggdh: sweet ...  let me know when you get to the point you're ready for the fix
[17:42] <hggdh> kirkland: roj
[17:42] <kirkland> hggdh: should be a one-liner config change in the upstart init script
[17:42] <hggdh> kirkland: increasing the heap for the java machine?
[17:42] <smoser> hggdh, yeah have at it
[17:42] <kirkland> hggdh: basically, we're going to give the jvm running eucalyptus-cloud 384M of mem
[17:43] <smoser> hggdh, goog guess
[17:43] <smoser> good even
[17:43] <hggdh> kirkland: yeah, thought this might be it. I still do not know *why* it does not release the mem
[17:43] <kirkland> hggdh: i talked to nurmi about it ...
[17:43] <kirkland> hggdh: it will expire it over time
[17:44] <hggdh> kirkland: delayed garbage colleciton?
[17:44] <kirkland> hggdh: but there can be a limit of the number of instances that can run in a short amount of time
[17:44] <hggdh> kirkland: yeah. But we are hitting it pretty hard
[17:44] <kirkland> hggdh: right
[17:44] <hggdh> kirkland: OK. give me 10 min to check some few changes I made to the script
[17:44] <kirkland> hggdh: nurmi says that 384M should put that limit much further out of reach
[17:44] <kirkland> hggdh: ack
[17:45] <hggdh> kirkland: I just wonder when this is used on a bug cloud farm. We might have to distribute the walruses
[17:45] <hggdh> s/bug/big/
[17:46] <gzmask> guys, in UEC, how do I change the mac address of my VM instance?
[17:46] <kirkland> hggdh: perhaps, yeah
[17:46] <kirkland> gzmask: don't know that that's possible
[17:46] <kirkland> gzmask: you'd need to hack the XML of the VM
[17:47] <gzmask> kirkland: where is that XML file located?
[17:48] <slipper1024> guys, sorry dropped off there
[17:49] <slipper1024> power failure
[17:49] <kirkland> gzmask: /var/lib/eucalyptus/instances ?
[17:49] <slipper1024> is there a way to recall chat history
[17:52] <gzmask> kirkland: found it at /var/lib/eucalyptus/instances/eucalyptus/cache/, don't think mac address is there or it's encrypted. I though xen can change mac address so does kvm  but apparently not.
[17:53] <kirkland> gzmask: kvm (qemu, actually) can definitely change the mac address
[17:53] <kirkland> gzmask: http://libvirt.org/formatdomain.html
[17:53] <kirkland> gzmask: eucalyptus is just leaving it unspecified
[17:53] <smoser> jjohansen, ping
[17:54] <jdstrand> jjohansen: hey, I moved bug #567334 to linux, since it is the apparmor_parser hang. please adjust if that is not correct
[17:55] <jdstrand> smoser: regarding that bug^ does the guest have the latest lucid kernels?
[17:55] <smoser> ec2 ami tests done. all passed, one new bug (that above)
[17:55] <smoser> latest ec2 kernel as of today
[17:55] <jdstrand> jjohansen: ^
[17:56] <kirkland> gzmask: see: /usr/share/eucalyptus/gen_kvm_libvirt_xml on your Node
[17:56] <jjohansen> hrmm, smoser latest user space too?
[17:57] <smoser> jdstrand, note, this happened once in 142 instances.
[17:57] <smoser> the other 141 are identical
[17:57] <jdstrand> smoser: I saw something yesterday about 94.9% instances out of 1000 came up. is this bug the cause for the ones that didn't come up?
[17:57] <smoser> jdstrand, no. thats eucalyptus specific (eucalyptus failure)
[17:57] <smoser> this is ec2
[17:57] <jdstrand> oh yes
[17:57] <jdstrand> k
[17:58] <slipper1024> can anybody that watched the channel please paste reply's on postfix issue mentioned earlier and any replies
[17:58] <jjohansen> smoser: once in 142?
[17:58] <jdstrand> smoser: 11:56 < jjohansen> hrmm, smoser latest user space too?
[17:58] <smoser> well, the latest everything that was in the archive at ~ 1:30 AM UTC today
[17:59] <jdstrand> we've not done any apparmor uploads in ages, so it should be up to date
[17:59] <smoser> i ran the automated tests that i have.
[17:59] <jjohansen> smoser: okay, and this only happened once in 142 instances, ie not the ec2 kernel bug we talked about yesterday
[17:59] <jdstrand> Tue, 30 Mar 2010
[18:00] <gzmask> kirkland: ah ok, but can I modify that file? It looks like it's a shell script that shows me the xml template and where should this xml template located?
[18:00] <jjohansen> just making sure, I'm not getting mixed up here
[18:00] <smoser> the total of 142 is across 2 arches , 3 regions, 5 instance sizes, 2 root-store types...
[18:00] <smoser> 6 amis total tested, all use same kernel same user space
[18:00] <smoser> jjohansen, i dont remember talking about ec2 kernel bug yesterday.
[18:00] <kirkland> gzmask: sorry, i don't know; i'm just giving you a few pointers
[18:00] <smoser> but yeah, its only 1 in 142 instances that saw this.
[18:00] <gzmask> kirkland: ok. thanks for the info
[18:01] <smoser> and the number of boots is slightly higer than that (some of them are shut down and then started)... making total of (guess) 175 boots or so
[18:01] <jjohansen> smoser: hrmm, I could have sworn it was yesterday but maybe friday.  only failing once in lots and lots of runs
[18:02] <smoser> jjohansen, ah... that is bug 564924
[18:02] <smoser> that is linux-server kernel in kvm guest (eucalyptus)
[18:03] <smoser> and we/I now believe that to be actually bad (virtual) hardware... see links to lkml discussion from that bug
[18:03] <jjohansen> smoser: right, its just that the once in X many boots is a similar symptom
[18:04] <kirkland> hggdh: http://paste.ubuntu.com/419361/
[18:05] <kirkland> hggdh: that's the patch you'll want to apply (or manually add the option) to /etc/init/eucalyptus.conf
[18:05] <kirkland> hggdh: that ups the jvm's heap to 384M
[18:05] <smoser> jjohansen, yeah. sorry that my brain is failing
[18:05] <jjohansen> np, same problem here
[18:05] <smoser> that one seems racey.
[18:06] <smoser> i can't understand why the apparmour stuff would be racy
[18:06] <gzmask> anyone know what is the default acount/password for ubuntu 9.10 32-bit UEC image? I got it running on my cloud but can't ssh in
[18:07] <jjohansen> smoser: well its possible something in the initscripts is broken
[18:07] <smoser> gzmask, there is no password.
[18:07] <smoser> you go in as ubuntu user, but you must specify a '--key' flag to run-instances
[18:08] <gzmask> smoser: that means I can just ssh in with my key file?
[18:08] <smoser> this may ont be the first time i've seen this issue, but it is the first time that i've collected log showing the apparmour message
[18:09] <smoser> gzmask, thats the theory (ssh -i path/to/your/key.pem ubuntu@x.y.z.y)
[18:09] <gzmask> smoser: thanks, that makes perfect sense
[18:11] <slipper1024> Hi guys, having trouble with postfix configuration. Setup: Ubuntu server 8.04, Postfix, Courier mail, Dovecot sasl user authentication, self signed certificate for mail. Single shared IP.     Working: pop over ssl. tls  ,  User Authentication,    smtp over ssl and tls from some domains.   Not Working: SMTP login, tls and ssl from some domains.     Error Message: Pasting below:
[18:12] <smoser> jjohansen, jdstrand when i look more at that, i dont think its apparmor specific
[18:12] <smoser> look at the other "blocked" tasks
[18:12] <slipper1024> Client error:       The message could not be sent because one of the recipients was rejected by the server. The rejected e-mail address was 'somebody@gmail.com'. Subject 'test', Account: 'mail.thisdomain.com', Server: 'mail.thisdomain.com', Protocol: SMTP, Server Response: '554 5.7.1 <unknown[196.35.255.4]>: Client host rejected: Access denied', Port: 465, Secure(SSL): Yes, Server Error: 554, Error Number: 0x800CCC79
[18:13] <ivoks> slipper1024: you again
[18:13] <ivoks> slipper1024: look at mail.log, this is outlook message (we don't use and it doesn't provide any info)
[18:14] <ivoks> slipper1024: i'd bet your client didn't authorize to postfix
[18:14] <jdstrand> smoser, jjohansen: I think you might be right
[18:14] <jdstrand> init: console-setup main process (62) terminated with status 1
[18:14] <jdstrand> [  240.805451] INFO: task apparmor_parser:182 blocked for more than 120 seconds.
[18:14] <hggdh> kirkland: ready
[18:14] <slipper1024> Apr 20 18:48:50 mycomp postfix/smtpd[16781]: warning: 201.19.75.90: address not listed for hostname 20119075090.user.veloxzone.com.br
[18:14] <kirkland> hggdh: can you ping me here, and update the bug at https://bugs.edge.launchpad.net/ubuntu/+source/eucalyptus/+bug/565101 when you have results?
[18:14] <slipper1024> Apr 20 18:53:26 mycomp postfix/smtpd[17357]: warning: 203.210.156.182: hostname adsl.hnpt.com.vn verification failed: Name or service not known
[18:14] <slipper1024> Apr 20 18:57:29 mycomp postfix/smtpd[17406]: warning: 85.105.244.76: hostname dsl.static.85-105-62540.ttnet.net.tr verification failed: Name or service not known
[18:14] <slipper1024> Apr 20 19:07:11 mycomp postfix/smtpd[17607]: warning: 117.4.141.93: address not listed for hostname localhost
[18:14] <kirkland> hggdh: rock on
[18:15] <jdstrand> let me start over
[18:15] <slipper1024> The above is output of /var/log/mail.warn
[18:15] <hggdh> kirkland: what did you change?
[18:15] <smoser> jdstrand, that one is red-herring
[18:15] <smoser> its "normal"
[18:15] <jdstrand> http://paste.ubuntu.com/419364/
[18:15] <smoser> (console-setup)
[18:15] <jdstrand> ok
[18:15] <kirkland> hggdh: i haven't uploaded anything
[18:15] <jdstrand> still, mount and the flush blocked too
[18:15] <kirkland> hggdh: you need to change /etc/init/eucalyptus.conf:
[18:15] <kirkland> -       opts="-h $EUCALYPTUS -u $EUCA_USER --pidfile /var/run/eucalyptus/eucalyptus.pid -l $LOGLEVEL -L console-log"
[18:15] <ivoks> slipper1024: that isn't related to the error you pasted above
[18:15] <kirkland> +       opts="-h $EUCALYPTUS -u $EUCA_USER --pidfile /var/run/eucalyptus/eucalyptus.pid -l $LOGLEVEL -L console-log -Xmx384m"
[18:16] <kirkland> hggdh: specifically adding: -Xmx384m"
[18:16] <smoser> jdstrand, right. regarding mount and flush blocking.
[18:16] <hggdh> kirkland: on the wlarus, or everywhere?
[18:16] <ivoks> slipper1024: look into mail.log and search for somebody@gmail.com
[18:17] <kirkland> hggdh: on the java components
[18:17] <kirkland> hggdh: so cloud, walrus, sc
[18:17] <jdstrand> unless the other two blocked cause of the first
[18:17] <jdstrand> jjohansen: what do you think? ^
[18:18] <jjohansen> sorry I'm a bit behind have been trying to follow k-t meeting as well
[18:18] <hggdh> kirkland: orj
[18:18] <jdstrand> jjohansen: http://paste.ubuntu.com/419364/
[18:18] <hggdh> roj
[18:18] <jdstrand> jjohansen: mount and flush blocked too
[18:19] <jdstrand> jjohansen: so maybe it isn't apparmor, or it is and that is why the mount and flush blocked
[18:19] <jjohansen> hrmm, yeah that doesn't seem like apparmor
[18:19] <jdstrand> jjohansen: consider yourself caught up :)
[18:19] <jjohansen> thanks
[18:20] <mcas1> hi has anyone tested the preseed test case with virtualbox?
[18:20] <jdstrand> smoser: mount and flush blocking-- doesn't that seem like a disk issue?
[18:21] <smoser> i guess it could
[18:21] <jdstrand> I don't know, I'm asking :)
[18:22] <smoser> jdstrand, i dont know
[18:23] <smoser> jdstrand, but, i wonder, would it be possible that the 2nd and third (mount, flush) are a result of the first (apparmor)
[18:23] <jjohansen> jdstrand: it could be, or it could be a kernel issue, or a hypervisor issue
[18:23] <jdstrand> smoser: I wondered the same, but jjohansen indicated it didn't seem apparmor related
[18:23] <jjohansen> smoser: yes that is possible
[18:23] <jjohansen> but unlikely
[18:23] <smoser> one way or another its a race condition somewhere.
[18:24] <smoser> it could be hypervisor or even faulty disk, but we have no real way of knowing
[18:24] <slipper1024> ivoks: Apr 20 18:37:33 mycomp postfix/smtpd[16281]: NOQUEUE: reject: RCPT from unknown[196.35.255.4]: 554 5.7.1 <unknown[196.35.255.4]>: Client host rejected: Access denied; from=<mike@hisdomain.com> to=<somebody@gmail.com> proto=ESMTP helo=<hylandmnb>
[18:24] <slipper1024> Apr 20 18:44:36 itdirect postfix/smtpd[16996]: NOQUEUE: reject: RCPT from unknown[196.35.255.4]: 554 5.7.1 <unknown[196.35.255.4]>: Client host rejected: Access denied; from=<mike@hisdomain.com> to=<somebody@gmail.com> proto=ESMTP helo=<hylandmnb>
[18:24] <slipper1024> Apr 20 19:06:51 mycomp postfix/smtpd[17810]: NOQUEUE: reject: RCPT from unknown[196.35.255.4]: 554 5.7.1 <unknown[196.35.255.4]>: Client host rejected: Access denied; from=<mike@hisdomain.com> to=<somebody@gmail.com> proto=ESMTP helo=<hylandmnb>
[18:24] <jjohansen> not yet anyway, we need to probe into it more
[18:24] <ivoks> slipper1024: don't paste so much output
[18:24] <slipper1024> sorry
[18:25] <ivoks> slipper1024: now look for 196.35.255.4 before first line
[18:25] <jdstrand> smoser, jjohansen: for now, I removed the apparmor tag and adjusted the title slightly to more accurately reflect the current thinking
[18:25] <jdstrand> smoser, jjohansen: feel free to adjust if apparmor is the culprit
[18:25] <jjohansen> thanks jdstrand
[18:25] <ivoks> slipper1024: paste it on pastebin.com
[18:26] <smoser> oh. i did the same jdstrand
[18:26] <smoser> :)
[18:27] <jdstrand> smoser: do you have any insight into the underlying machine? like an id number or something? I ask cause if you ran your tests again, it might be interesting if the same host had the same problem
[18:28] <ivoks> slipper1024: grep 'Apr 20 18' /var/log/mail.log | grep 196.35.255.4
[18:29] <smoser> jdstrand, you get no such information
[18:29] <jdstrand> bummer
[18:30] <slipper1024> ivoks: http://pastebin.org/162495
[18:31] <slipper1024> pasted connect-disconnect info
[18:32] <ivoks> slipper1024: so, as you can see
[18:32] <ivoks> slipper1024: there's no sasl_username
[18:32] <ivoks> slipper1024: your client didn't set up outlook to authenticate to smtp
[18:33] <ivoks> slipper1024: and smtp won't relay host for anyone, only for authenticated users
[18:34] <slipper1024> I see, Anonymous
[18:34] <mcas1> i need some help with the preseed testcase... i can boot the machine but the ntptime command gives me an error
[18:34] <ivoks> slipper1024: http://pastebin.com/aJ22CJz9
[18:34] <ivoks> slipper1024: you should have something like this
[18:35] <mcas1> what can i do what is going wrong here
[18:35] <mcas1> ?
[18:35] <ivoks> ntptime?
[18:35] <ivoks> ah...
[18:35] <slipper1024> he is using Outlook express and i watched him enter his login details
[18:36] <slipper1024> weird, gonna re-do
[18:36] <slipper1024> account setup
[18:36] <ivoks> this is not account setup
[18:36] <ivoks> this is smtp (outgoing) setup
[18:36] <ivoks> it's not offered in wizzard, iirc
[18:36] <slipper1024> sry that is what i mean
[18:37] <ivoks> check again
[18:37] <slipper1024> ivoks: thanks man, those errors in mail.warn posted earlier
[18:37] <ivoks> slipper1024: http://dnet.net/techSupport/smtpAuthOE.asp
[18:37] <slipper1024> any idea?
[18:37] <ivoks> slipper1024: ignore them
[18:37] <ivoks> those aren't errors
[18:38] <ivoks> those are warrning, hence the name mail.warn
[18:38] <slipper1024> Ok thought it might be a dns misconfiguration on my part
[18:39] <ivoks> slipper1024: those are spamers
[18:40] <ivoks> slipper1024: if configured right, postfix won't accept mail from hostname that's not resolvable
[18:40] <ivoks> slipper1024: so, if say 'helo i'm wickedname.google.cn' to you postfix
[18:40] <ivoks> slipper1024: it will laugh at me and drop the connection
[18:41] <ivoks> and then it will add notice to mail.warn, so you could laugh with it
[18:41] <stephank> Hello! I'm on Lucid and trying to set up OpenLDAP with TLS and mutual authentication. Whether I use ldaps or StartTLS, the server closes the connection right after the client hello, and logs: “TLS: can't accept: The request is invalid..”. The client and server are the very same machine. I'm not sure where to go from here?
[18:41] <ivoks> stephank: self signed cert?
[18:42] <stephank> ivoks: Yes, but I specified TLS_CACERT, TLS_CERT and TLS_KEY, and it's reading the files (seen in strace).
[18:42] <ivoks> stephank: try adding TLS_REQCERT allow to /etc/ldap/ldap.conf
[18:42] <ivoks> stephank: just to be sure...
[18:43] <ivoks> this is, of course, client related
[18:43] <stephank> ivoks: I had it set to demand, but just tried allow and no dice either.
[18:43] <ivoks> but this is very common mistake, so let's try
[18:43] <ivoks> ok
[18:43] <ivoks> maybe your client isn't trying ssl?
[18:44] <ivoks> i like python for debuging ldap connection :)
[18:44] <stephank> Well, I'm convinced it's my mistake, on the other hand, it stops at the client hello, so all that's been said so far is which cyphers to use. :)
[18:44] <smoser> jdstrand, http://paste.ubuntu.com/419377/
[18:44] <stephank> I'm trying with "ldapsearch -ZZ"
[18:44] <stephank> and wireshark shows the starttls and client hello, then a connection close from the server
[18:45] <ivoks> start openldap in debuging mode
[18:45] <smoser> i see that on a node controller... i think that the too many open files might be a result of bug 567371, and the apparmor errors just fallout
[18:46] <stephank> ivoks: that's how I got the TLS error “TLS: can't accept: The request is invalid..”. I can't find anything on google related to slapd and that error, but it seems to be an error reported by gnutls.
[18:47] <stephank> strange thing is, it's the same machine, and thus the same gnutls library on server and client
[18:47] <jdstrand> smoser: cool
[18:48] <kirkland> jdstrand: have you ever seen: libvirt: cannot create pipe: Too many open files (code=38)
[18:48] <kirkland> jdstrand: this is related to smoser's error
[18:48] <jdstrand> smoser: so it was able to launch virt-aa-helper, but then when it went to fork/exec kvm, it couldn't
[18:48] <ivoks> stephank: slapd -o openldap -g openldap -d 16383 ?
[18:49] <jdstrand> kirkland: sounds like the system is over-taxed, or you need up the ulimit
[18:49] <jdstrand> kirkland: but no, I have not seen it personally
[18:49] <kirkland> jdstrand: yeah, we're taxing it, by running hundreds (or thousands) of euca instances over a few hours
[18:50] <smoser> jdstrand, over time
[18:50] <jdstrand> kirkland: maybe you have an fd leak somewhere... or you just hit the limit
[18:51] <smoser> right.
[18:51] <smoser> in libvirt
[18:51] <smoser> i suspect
[18:51] <smoser> the system is not terribly over taxed at the moment
[18:51] <smoser> as 114 processes in 4G of memory
[18:51] <stephank> ivoks: Here's what I get: http://www.privatepaste.com/d0397d2513
[18:51] <kirkland> jdstrand: yup, <nurmi__> oh yeah, file leak
[18:51] <kirkland> jdstrand: <nurmi__> it's holding on to the VM log files
[18:52] <jdstrand> upstream fixes fd leaks all the time (it seems) -- maybe check git?
[18:52] <kirkland> jdstrand: will do
[18:52] <ivoks> stephank: tls negotiation
[18:52] <ivoks> stephank: bah... have no clue :D
[18:53] <stephank> ivoks: no problem, thanks for thinking along. :)
[18:53] <ivoks> stephank: no-tls works?
[18:53] <stephank> ivoks: I'm going to try with gnutls-serv and the same certificates
[18:53] <stephank> ivoks: well, I've only tried ldapi
[18:56] <stephank> ivoks: well, gnutls-serv and gnutls-cli report the exact same error :'(
[18:56] <ivoks> stephank: try regenerating certificates
[18:56] <ivoks> stephank: use openssl
[18:56] <ivoks> :)
[18:57] <stephank> ivoks: Will try that. Maybe I'm just doing something stupid, because I was trying to use the certificates my puppet installation generated
[18:57] <jdstrand> kirkland: fyi, I just double checked the security driver, and the one place it doesn't use a libvirt wrapper for managing fds, it dtrt
[18:57] <kirkland> jdstrand: smoser spotted http://osdir.com/ml/libvir-list/2010-02/msg00919.html
[18:58] <kirkland> jdstrand: smoser: hmm, that looks openvz specific
[18:58] <jdstrand> kirkland: that should be unrelated (openvz
[18:58] <smoser> yes it does
[18:58] <smoser> yeah, thats all in openvx_conf.c
[19:01] <smoser> bug 567392 is open now
[19:03] <jdstrand> kirkland, smoser: this could be it: http://osdir.com/ml/libvir-list/2010-02/msg00210.html
[19:05] <jdstrand> absolutePathFromBaseFile() is used by virStorageFileGetMetadataFromFD(), which is used in a few places in the storage driver
[19:06] <Hypnoz> is anyone here familiar with dsh?
[19:07] <pwnguin> dancershell?
[19:07] <pwnguin> yes
[19:07] <Hypnoz> do you know how to use a ' in the command string?
[19:07] <jdstrand> that is actually used by the security driver, but in virt-aa-helper, not libvirtd, so that shouldn't be the cause of the problem
[19:08] <jdstrand> but, virStorageFileGetMetadata() is used in other places as well
[19:08] <jdstrand> smoser, kirkland: ^
[19:08] <pwnguin> Hypnoz: are you using dsh -f hostlist -- command?
[19:09] <Hypnoz> yeah. for example, dsh -f machines.list -- 'echo "a ' b"'
[19:09] <pwnguin> well, im only familiar with it
[19:09] <pwnguin> i ui admin our small renderfarm with it
[19:10] <pwnguin> s/ui//
[19:10] <Hypnoz> yeah i use it a lot for admin of a group of servers, but I pass any commands that have a ' in them
[19:10] <Hypnoz> such as things with awk commands
[19:10] <jdstrand> kirkland, smoser: commit 53b7dae139f2dd66866aabedca35aa74dad1579d
[19:10] <smoser> jdstrand, http://libvirt.org/git/?p=libvirt.git;a=commit;h=53b7dae139f2dd66866aabedca35aa74dad1579d
[19:10] <smoser> yeah
[19:10] <smoser> i was just typing that
[19:10] <jdstrand> kirkland, smoser: I've not looked at it, but that seems as good a place to start as any
[19:11] <smoser> yea
[19:11] <pwnguin> Hypnoz: well, i'm interested to find out the answer myself now
[19:12] <Hypnoz> I'm trying things to escape the ' out like dsh -m web100 -- 'echo "a \' b"'  but still no luck
[19:14] <Hypnoz> hooray! dsh -m web100 -- 'echo "a '"'"' b"'
[19:14] <Hypnoz> web100: a ' b
[19:14] <jdstrand> smoser, kirkland: you guys might want to do something like: watch 'sudo ls -1 /proc/`cat /var/run/libvirtd.pid`/fd | wc -l'
[19:15] <smoser> jdstrand, awesome
[19:15] <smoser> i was wondering how iw as going to tell if this fixed the problem
[19:15] <smoser> without waiting till tomorrow
[19:15] <jdstrand> well, shoot
[19:15] <jdstrand> there is a leak
[19:16] <jdstrand> if I do a virsh start the fd increments by 2,then a virsh destroy it only decrements by 1
[19:18] <jdstrand> smoser, kirkland: ^
[19:18] <jdstrand> smoser, kirkland: fyi, I tested with the apparmor driver disabled and it still leaks
[19:20] <jdstrand> smoser, kirkland: it isn't closing the log:
[19:20] <jdstrand> $ diff /tmp/before /tmp/after
[19:20] <jdstrand> 38a39
[19:20] <jdstrand> > lrwx------ 1 root root 64 2010-04-20 13:17 43 -> /var/log/libvirt/qemu/sec-dapper-i386.log
[19:20] <jdstrand> I'll update the bug
[19:20] <smoser> console log ?
[19:20] <smoser> thank you
[19:21] <jdstrand> smoser: not the console log, the debug output log.
[19:21] <smoser> at the risk of sounding like a fool, debug output log ?
[19:21] <smoser> of what
[19:22] <jdstrand> smoser: libvirt
[19:22] <jdstrand> smoser: libvirtd logs stuff like this to files in /var/log/libvirt/qemu/<vmname>.log:
[19:22] <smoser> ah. ok
[19:22] <smoser> yeah, i see it
[19:22] <jdstrand> http://paste.ubuntu.com/419395/
[19:23] <jdstrand> smoser: which bug is it?
[19:23] <smoser> https://bugs.launchpad.net/ubuntu/+source/libvirt/+bug/567392
[19:23] <jdstrand> smoser: I'll update it with what I found
[19:23] <jdstrand> cool, thanks
[19:25] <binBASH> is there any way to get glusterfs packages from lucid into actual ubuntu server release?
[19:26] <jdstrand> smoser: let me check to see if shutdown is also affected
[19:26] <binBASH> I've seen actual release has glusterfs 2 while the lucid has glusterfs 3
[19:26] <smoser> http://libvirt.org/git/?p=libvirt.git;a=commit;h=598a0c00dcc7fa07d226a3f508b517690a9be819
[19:26] <smoser> jdstrand,
[19:27] <smoser> wait
[19:27] <smoser> ignore that
[19:27] <jdstrand> k
[19:27] <smoser> http://libvirt.org/git/?p=libvirt.git;a=commit;h=caad0a878337b52c4453444bb227b74cce6e6d5f
[19:27] <smoser> that one
[19:27] <jdstrand> smoser: that has to be it
[19:28] <jdstrand> I can almost guarantee it
[19:28] <jdstrand> and it is a two line patch!
[19:29] <smoser> yeah. stars aligned
[19:29] <jdstrand> heh
[19:30] <smoser> jdstrand, please do update that bug
[19:30] <smoser> at least confirm it
[19:30] <jdstrand> smoser: is this for SRU?
[19:30] <kirkland> jdstrand: ideally for RC
[19:30] <kirkland> smoser: jdstrand: back from lunch
[19:30] <jdstrand> my vm died and I couldn't shut it down nicely
[19:30] <jdstrand> hold on
[19:30] <smoser> jdstrand, sru for lucid ?
[19:30] <smoser> as opposed to NOW
[19:31] <jdstrand> well, I have an SRU bug I can squeeze in there so I can prepare the upload if you like
[19:31] <JanC> binBASH: wait 2 weeks, then lucid will be an "actual release"  ;-)
[19:31] <jdstrand> smoser: yes, SRU vs NOW
[19:31] <jdstrand> smoser: I can do the NOW if required
[19:31] <smoser> well, lets verify this fixes first
[19:31] <smoser> which we think it would
[19:31] <jdstrand> of course
[19:32] <smoser> i'm so happy that you use quilt
[19:32] <jdstrand> smoser: I'm so confident it will, I'll even do all the work, test it and get back to you :)
[19:32] <smoser> jdstrand, fair enough
[19:32] <jdstrand> heh-- thank debian :)
[19:33] <smoser> ok then, build me a deb and i'll test it.
[19:33] <smoser> so we think that simply bouncing libvirtd will fix my euca problem
[19:33] <jdstrand> k, still checking shutdown. after will update the bug and build a deb
[19:34] <jdstrand> smoser: amd64 or i386?
[19:34] <jdstrand> (or both)
[19:34] <smoser> this is amd64
[19:34] <jdstrand> k
[19:34] <smoser> after restart: ls /proc/$a/fd | wc -l
[19:35] <smoser> went from 1021 files to 33
[19:35] <kirkland> smoser: lsof should help too
[19:36] <smoser> bouncing libvirt (sudo restart libvirt-bin) fixed the euca issue
[19:36] <binBASH> JanC: Well, still a bit long
[19:38] <JanC> binBASH: you can always test with the beta for now
[19:39] <binBASH> JanC: I had very bad experience in the past at least with the desktop versions of ubuntu. :)
[19:40] <_ruben> binBASH: depending on the dependencies (no pun intended), you could either just install deb lucid .deb file, or backport it (and perhaps some/all of its dependencies)
[19:41] <binBASH> _ruben: Yeah I will try it.
[19:41] <binBASH> However I didn't find 64 bit version
[19:42] <binBASH> ahh I found it
[19:45] <binBASH> _ruben: /usr/sbin/glusterfsd: symbol lookup error: /usr/sbin/glusterfsd: undefined symbol: gf_proc_dump_info
[19:45] <binBASH> looks like it failed
[19:46] <binBASH> :/
[19:48] <jdstrand> smoser: building now
[19:48] <JanC> binBASH: I assume that you want to test something like glusterfs before using it for real, so using a beta version for testing shouldn't be much of an issue?
[19:49] <binBASH> JanC: I tested the glusterfs already
[19:49] <JanC> I mean version 3  ;)
[19:49] <slipper1024> ivoks: outlook express and outlook on winxp and earlier doesn't support DHE-RSA-AES256-SHA  encryption
[19:50] <binBASH> JanC: I tested version 3 on CentOS ;)
[19:51] <binBASH> I tested version 2 as well and I know it was not what I need ;)
[19:52] <_ruben> binBASH: which package(s) did you install?
[19:54] <JanC> binBASH: so, try it on ubuntu lucid beta so you can be sure it works well...
[19:55] <JanC> it's not like much will change in the 2 weeks that are left...
[19:55] <smoser> kirkland, so you never hit that issue in libvirt because you had multiple node controllers
[19:55]  * RoyK sends some Icelandic ash in binBASH's directoion
[19:55] <RoyK> direction, even
[19:55] <kirkland> smoser: ah, true, i do have 3 NCs right now
[19:56] <kirkland> smoser: <nurmi__> do you have log files from the CLC in the case where the meta-data service is failing in the way scott was descriving this morning?  We've looked at the code and nothing is jumping out
[19:57] <kirkland> jdstrand: reading scroll back, looks like you've grabbed that libvirt fix, and you're building a libvirt for smoser to test?
[19:57] <hggdh> kirkland: out of memory again. I will add the data in the bug
[19:58] <kirkland> hggdh: hmm, even with the 384M fix?
[19:59] <smoser> kirkland, its readily reproducible
[19:59] <smoser> especially on datacenter
[19:59] <kirkland> smoser: k, will grab logs
[19:59] <smoser> hggdh, maybe we could mark the logs (or rotate them) then start a run
[19:59] <smoser> so you've got clean stuff.
[20:00] <kirkland> hggdh: did you have any of the metadata service style failures in that last run?
[20:02] <smoser> kirkland, the most recent log there single_test.log.2010-04-20_135807
[20:02] <smoser> has loads of them
[20:03] <binBASH> _ruben: glusterfs-client and glusterfs-server
[20:03] <binBASH> JanC: I don't wanna damage my working cloud structure ;)
[20:05] <hggdh> kirkland: yes, a series of them -- the instance waits for the metadata, then barfs when it gets it empty
[20:05] <kirkland> hggdh: great, could you tar up /var/log/eucalyptus on the CLC?
[20:05] <kirkland> hggdh: and put it somewhere I can get to it and send on to Eucalyptus?
[20:06] <kirkland> hggdh: also, could you pastebin /etc/init/eucalyptus.conf for me?
[20:06] <kirkland> hggdh: and (silly question), you did restart all of the eucalyptus services after making that conf file change, right?
[20:07] <jdstrand> kirkland: I am testing the build now
[20:08] <hggdh> kirkland: in reverse order: yes, I restarted the CLC, CC, and SC; I can pastebin, but the 'ps aux' in the bug comments shows the memory allocation
[20:08] <hggdh> for the CLC
[20:12] <\sh> hmmm..does anyone has experience with an HP lefthand iscsi system + HP StorageWorks x5000 as nfs/cifs concentrator?
[20:13] <kirkland> hggdh: got it
[20:13] <kirkland> hggdh: now, just a tarball of the /var/log/eucalyptus from the CLC
[20:16] <hggdh> kirkland: attaching to the bug right now
[20:16] <jdstrand> smoser, kirkland: packages fix it for me. they are on chinstrap in /home/jamie/uploads/lucid/libvirt
[20:16] <kirkland> hggdh: cheers
[20:17]  * jdstrand updates bugs
[20:17] <jdstrand> bug
[20:17] <kirkland> jdstrand: i'll take a quick look
[20:17] <kirkland> jdstrand: i suggest we try to get this into RC ... objections?
[20:17] <kirkland> smoser: ^
[20:18] <smoser> i dont know. it is obviously a but, but not one that would be hit in a week or two by someone doing something other than libvirt testing
[20:18] <smoser> s/obviously a but/obviously a bug/
[20:18] <smoser> but it seems like extremely low chance of regression
[20:18] <jdstrand> I'll attach the debdiff and let you decide
[20:18] <jevidl> In the past I have used mem="2048" on the kernel line in grub to limit the system memory. With the advent of grub2, I don't seem to be finding a way to do it anymore. Google is happy to tell me all about how grub uses memory, etc. but not much about the kernel line option. Is this still possible?
[20:18] <smoser> kirkland, can you get to cempedak ?
[20:18] <kirkland> smoser: yeah
[20:18] <smoser> if so, its in /tmp/eucalyptus-logs.tar.bz2
[20:18] <smoser> euca logs
[20:20] <kirkland> smoser: strike that, ssh denied to cempedak
[20:21] <hggdh> huh? you cannot ssh there?
[20:21] <ttx> kirkland: was bug 567371 something we hit in our stress testing ?
[20:22] <kirkland> ttx: yes, something that dan hit during stress testing
[20:22] <kirkland> ttx: i have merged his fix
[20:22] <kirkland> ttx: i was going to do a brief round of testing, and then propose for upload
[20:22] <ttx> ok, not something we hit in stress testing, but something they hit in theirs ?
[20:23] <kirkland> ttx: yes, but i think i've hit it too
[20:23] <ttx> ok
[20:23] <kirkland> ttx: though i don't have evidence of it on my cloud at the moment
[20:23] <kirkland> ttx: basically, there are 3 bugs we're tracking/trying to fix right now ...
[20:23] <kirkland> ttx: a) a java heap memory error, #565101
[20:24] <kirkland> ttx: b) a libvirt file leak error, #567392
[20:24] <kirkland> ttx: c) dan's NC pthreads error, #567371
[20:24] <jdstrand> kirkland, smoser: debdiff attached, bug updated with my testing (#567392)
[20:24] <ttx> kirkland: hm, what about bug 566792 ? it's pretty high on my list
[20:24] <kirkland> ttx: all 3 have proposed fixes, we're just trying to verify the fixes now
[20:25] <ttx> kirkland: did nurmi comment on that one ?
[20:25] <kirkland> ttx: right, d) metadata sometimes fails, #566792
[20:25] <kirkland> ttx: dan is looking into that one, haven't been able to reproduce it
[20:26] <ttx> kirkland: ok, that matches my list then :)
[20:26] <kirkland> ttx: appears to be hardware dependent ... the faster your hardware, the more likely/often you hit it
[20:26] <kirkland> ttx: which of these do you want in RC?
[20:26] <kirkland> ttx: which of these do you want in SRUs?
[20:26] <smoser> i think its more io speed related
[20:26] <smoser> than just general "fast"
[20:26] <ttx> I think you hit it on slow UEC and fast images
[20:26] <ttx> (fast to boot)
[20:27] <kirkland> ttx: so do you want us to target all of these for RC?
[20:27] <kirkland> ttx: 2 of them are basically "done"
[20:27] <kirkland> ttx: one of them has a proposed fix that didn't quite work on first round of testing
[20:27] <ttx> a b c can be done in SRUs idf needed
[20:27] <kirkland> ttx: and one is still under investigation
[20:28] <kirkland> ttx: i think all of these can be done in SRUs ... i'm just concerned about the effort required to do these in RC versus SRU
[20:28] <kirkland> ttx: i'm looking for your guidance on that
[20:28] <hggdh> eeek. I also see a JavaStackOverflow in the walrus
[20:29] <ttx> at that point the effort is the same. So it depends on the regression risk of the patch. If any, then SRU
[20:29] <ttx> but...
[20:29] <ttx> for (d) we need to come to the bottom of it, make sure its a euca bug and not something we need to fix/workaround in our images
[20:30] <ttx> (we can still issue an image update but I thin it's less easy than a package sru)
[20:32] <ttx> kirkland: for sru vs. RC fix, I trust the release team to make the right choice... if you want me to assess it, I can have a quick look at the patches
[20:32] <ttx> if you point me to them
[20:32] <kirkland> ttx: https://bugs.edge.launchpad.net/ubuntu/+source/libvirt/+bug/567392
[20:32] <kirkland> ttx: patch at the bottom;  i just proposed for RC
[20:33] <ttx> that one sounds rather limited, I'd support it
[20:33] <kirkland> hggdh: for the heap error, did increasing the memory change the behavior at all?  push out the mem errors further?  anything?
[20:34] <kirkland> ttx: thanks; i agree
[20:34] <ttx> if you want it to RC, you need to push it to release team now and ask them to reroll asap
[20:34] <ttx> any other patch ?
[20:34]  * jdstrand is ready to upload
[20:35]  * ttx looks into rev1223 for the NC thread thing
[20:37] <ttx> that one looks slightly more dangerous...
[20:37] <kirkland> ttx: okay
[20:38] <kirkland> ttx: so we should start prepping a Day0 SRU?
[20:38] <ttx> yes, having something in -proposed would be good
[20:39] <ttx> the walrus thing... I'm ok with -Xmx384m, but that doesn't seem to work so well
[20:41] <kirkland> ttx: right, i'm hoping this one is easy to fix
[20:41] <kirkland> ttx: and easy to verify is-fixed
[20:41] <kirkland> ttx: but that didn't work yet
[20:41] <ttx> kirkland: ok, I'll shut down for the day and let all that in your hands
[20:42] <kirkland> ttx: ack, bon soir
[20:42] <ttx> (and the release team ones) :)
[20:43] <ttx> cheers
[20:46] <kirkland> hggdh: could you "grep -ri pthread /var/log/eucalyptus/*" on each of the lab machines?
[20:46] <kirkland> hggdh: tell me if you have any hits?
[20:46] <kirkland> hggdh: i don't see any traces of https://bugs.edge.launchpad.net/ubuntu/+source/eucalyptus/+bug/567371 in my test rig
[20:46] <kirkland> ttx: i have targeted https://bugs.edge.launchpad.net/ubuntu/+source/eucalyptus/+bug/567371 at lucid-updates SRU
[21:17] <_ruben> binBASH: you likely need libglusterfs0 to be upgraded as well
[21:26] <dbernar1> I'm trying to set up samba as a domain controller. I was following https://help.ubuntu.com/8.10/serverguide/C/samba-dc.html and my windows client can not see my domain.
[21:27] <dbernar1> http://pastebin.com/tKCJ4yEW
[21:29] <dbernar1> That's my smb.conf
[21:38] <kirkland> hggdh: around?
[21:43] <hggdh> kirkland: on call
[21:43] <kirkland> hggdh: i'm trying to get onto cempedak
[21:44] <kirkland> hggdh: n'mind, i got it
[21:49] <kirkland> hggdh: okay, by changing to 384M, we actually "reduced" the amount of memory (turns out it defaults to 512M)
[21:58] <hggdh> kirkland: I am back, sorry, was on my 1-on-1
[21:59] <kirkland> hggdh: no problem; debugging heap mem issue with dan
[21:59] <hggdh> kirkland: will check for the pthreads now
[22:00] <hggdh> heh. less mem to alleviate memory starvation is fun :-)
[22:00] <kirkland> hggdh: did it poop out soon/later/same when you did that?
[22:03] <hggdh> kirkland: it changed the behaviour, actually -- instead of refusing any, and all, new requests, it would fail *sometimes*
[22:03] <hggdh> kirkland: no pthread in any of the logs
[22:03] <kirkland> hggdh: thanks
[22:03] <hggdh> kirkland: but, OTOH, I have been unable to register a new UEC image
[22:04] <kirkland> hggdh: can you bump it up to 1024 on each of the clc, walrus, sc and restart those services?
[22:04] <hggdh> kirkland: roger wilco
[22:04] <kirkland> hggdh: don't know if that's going to be acceptable though
[22:05] <binBASH> _ruben: Thanks the message dissapeared ;)
[22:06] <dbernar1> I'm trying to set up samba as a domain controller. I was following https://help.ubuntu.com/8.10/serverguide/C/samba-dc.html and my windows client can not see my domain. http://pastebin.com/tKCJ4yEW is my smb.conf
[22:07] <hggdh> kirkland: it's been a time since I worked on Java, but I have a dim memory it was not acceptable. Will find in a few ;-)
[22:07] <kirkland> hggdh: heh
[22:09]  * RoAkSoAx feels like coding...
[22:09] <binBASH> _ruben: Thanks a lot again.
[22:12] <hggdh> kirkland: swallowed hook, sink, and bait. Will do a new sequence of tests
[22:33] <hggdh> from an instance console: WARNING:INSTANCE i-3C9E0644:rm: cannot remove `/var/lib/urandom/random-seed': Read-only file system
[22:33] <hggdh> what does that mean? random-seed is static?
[22:34] <hggdh> smoser: which image has the debug on metadata?
[22:55] <kirkland> hggdh: if your filesystem has gone readonly, that's globally bad
[22:56] <hggdh> kirkland: I see it on some of the failed instances
[22:58] <hggdh> kirkland: I only really noticed it due to a completely different (and unrelated) issue involving virtual machines
[22:58] <hggdh> kirkland: now, if this should not happen, then we may have yet another issue to look at
[22:59] <mathiaz> jdstrand: hi - looking at bug 566803
[22:59] <mathiaz> jdstrand: is there a default location for the iptables save file?
[23:00] <jdstrand> mathiaz: using standard iptables-save?
[23:00] <mathiaz> jdstrand: yes
[23:00] <mathiaz> jdstrand: I don't see an init script or upstart job
[23:00] <jdstrand> mathiaz: no-- it just dumps to stdout. You have to redirect it somewhere
[23:00] <mathiaz> jdstrand: right
[23:00] <mathiaz> jdstrand: so there is a standard location for the file
[23:02] <jdstrand> mathiaz: the iptables tuturial mentions /etc/iptables-save
[23:02] <mathiaz> jdstrand: what would be recommendation for Ubuntu?
[23:02] <jdstrand> mathiaz: Debian used to put it somewhere, I don't recall otoh
[23:02] <jdstrand> mathiaz: ufw
[23:02] <jdstrand> :)
[23:03] <jdstrand> mathiaz: tbh, there are a bunch of iptables frontends. they all do stuff differently
[23:03] <jdstrand> mathiaz: /etc/iptables-save seems reasonable enough if you aren't going to use ufw (or shorewall)
[23:03] <mathiaz> jdstrand: right - /etc/sysconfig/ is definetly not the correct place though
[23:04] <jdstrand> mathiaz: no
[23:04] <jdstrand> mathiaz: I wouldn't put it in /etc/default
[23:04] <jdstrand> either
[23:04] <jdstrand> mathiaz: what is this for?
[23:04] <mathiaz> jdstrand: bug 566803
[23:04] <mathiaz> jdstrand: the location of the default file for augeas
[23:05] <jdstrand> mathiaz: well, technically, it is saving the state of the firewall
[23:05] <jdstrand> mathiaz: as such, it should really by in /var/lib
[23:06] <jdstrand> mathiaz: however, since this is presumably starting early on boot, /var may be on another partition
[23:06] <jdstrand> mathiaz: and thus not available
[23:07] <jdstrand> mathiaz: so, you can try to be FHS compliant and put it in /lib/...somewhere (though some interpretations of the FHS wouldn't like that either), or just say /etc/iptables-save
[23:08] <jdstrand> mathiaz: since /etc/iptables-save is most discoverable, and in the iptables tutorial, I recommend it
[23:09] <mathiaz> jdstrand: great - thankls
[23:09] <kirkland> hggdh: yeah, if instance filesystems are going readonly, that's another bug :-/
[23:09] <kirkland> hggdh: possibly an fsck one?
[23:11] <gzmask> question: in UEC, using SYSTEM mode in networking, I start an instance with --addressing private, and I got the private IP 172.19.1.2 from euca-describe-instances, but I can't ping or ssh onto the instance. Is it something wrong here? Also, when I euca-run-instances without using --addressing private, it gives a address resource error which is expected since I need to submit MAC address to get assigned an IP here on campus network.
[23:12] <jdstrand> mathiaz: sure, np
[23:17] <kirkland> mathiaz: are you planning to take hggdh's recent improvements to the test scripts?
[23:19] <mathiaz> kirkland: yes
[23:19] <mathiaz> kirkland: I should create a project for that
[23:20] <kirkland> mathiaz: twould be nice
[23:25] <mathiaz> kirkland: are you also running into bug 567396 while doing your UEC test install?
[23:25] <Rafael_>  i am new and try this comand and got the following error, anybody can help: rgotten@myplasticare:~$ sudo /etc/init.d/rsync restart
 write: Resource temporarily unavailable
  * Restarting rsync daemon rsync                                                                                                                                                  write: Resource temporarily unavailable
  * rsync daemon not running, attempting to start.
 write: Resource temporarily unavailable
[23:26] <Rafael_> this was done on my ubuntu server
[23:27] <kirkland> mathiaz: i think i just had to add "d-i partman/confirm_write_new_label boolean true"
[23:28] <kirkland> mathiaz: well, hmm, i see you have that
[23:28] <kirkland> mathiaz: this preseed worked for me yesterday: http://mirror/uec-auto/preseed/preseed-00-24-e8-da-a2-11
[23:29] <\sh> guys....I have to say "Thank you" to you all...doing jaunty updates (security and updates) and everything works like a charm..no flaws :) thanks for that :)
[23:32] <hggdh> kirkland: OK, seems the -Xms1024m did the trick. I have reached 890M on RES, and it is still running
[23:39] <mathiaz> kirkland: hm - could you post your preseed somewhere that I can access to it?
[23:41] <erichammond> jiboumans: Has any progress been made on getting Amazon's Perl package for SimpleDB into CPAN/Debian/Ubuntu? http://developer.amazonwebservices.com/connect/entry.jspa?externalID=1136&categoryID=148
[23:43] <g-hennux> hi!
[23:43] <g-hennux> where does ufw store its configuration, in case i want to migrate it from one host to another?
[23:44] <g-hennux> or what's the preferred way of backing up my ufw rules, if you want?
[23:55] <hggdh> kirkland: I am in doubt. Against which package should the bug be (for the readonly messages I saw in the instance console output)
[23:56] <kirkland> hggdh: hmm, great question :-/
[23:56] <kirkland> mathiaz: oh, sorry
[23:56] <kirkland> hggdh: start with cloud-init, i suppose
[23:57] <RoAkSoAx> mathiaz, I have a quick question. Is the puppet EC2/UEC integration is only the ability to register the instances running to a puppet server automatically?
[23:57] <RoAkSoAx> \sh, btw... where you able to setup pacemaker/corosync with puppet?
[23:57] <hggdh> thanks
[23:58] <\sh> RoAkSoAx, nope...we are still working on that
[23:59] <\sh> .oO(and more)
[23:59] <mathiaz> RoAkSoAx: there other alternatives
[23:59] <mathiaz> RoAkSoAx: you can always create your script to do that