[00:00] justizin: well, 4 would be fine, 2700 is trouble. [00:00] even 4 seems like 3 too many ;d [00:00] justizin: any idea what process is doing that? :) [00:00] i'm not even sure how to investigate [00:00] those are system mounts [00:00] these aren't like SHM segments [00:01] justizin: naaah they're nearly free to setup, 4 vs 1 is probably only a few kilobytes difference in the end [00:01] right, but i think it indicates a bug that gets out of control and creates 2700 on some systems.. [00:01] 4 is an incorrect number. ;) [00:01] just not harmful [00:01] justizin: darn. I was hoping you'd have some idea, because my idea is ugly -- if you install auditd, you can use auditctl to log every 'mount' system call, and collect data on what is going on that way. [00:02] anyway just sniffing it out, i tried changing the mount to /run/shm and rebooted the box (non-prod), we'll see if it has a pile of mounts tmrw :) [00:02] * justizin re-adds #ubuntu-server to auto-join [00:03] justizin: I think it'd be something like this: auditctl -a exit,always -S mount -F success=0 [00:03] * justizin nods [00:03] justizin: oh, maybe success=1. Or leave that off. meh. poor documentation. :) hehe [00:08] TJ-: I'm back. Unfortunately, TTY7 didn't display anything of note. Generic status of the speed of the RAID algorithms available, etc. [00:08] However, I looked at /tmp and noticed something interesting. [00:09] There were two soft links in there: ./mountroot-fail-hooks.d/20-lvm2 ./mountroot-fail-hooks.d/10-mdadm [00:09] garrettk: Yes, they are created by the initrd scripts [00:10] I'm pretty certain that both of those scripts failed to execute. I tried manually running "set -x" and then running the mdadm script, but all I got was an error complaining that the particular link in /tmp already existed. I guess -x doesn't propagate down the call stack. [00:12] garrettk: What's the content of the system's /etc/mdadm/mdadm.conf ? It should have something like "ARRAY /dev/md0 UUID=..." [00:14] TJ-: ARRAY /dev/md0 level=raid1 metadata=0.90 num-devices=2 UUID=5c92f0d9:9cf5be95:03611c5e:a540b92f [00:17] garrettk: OK, aside from UUIDs nothing different to my test setup [00:18] hallyn: still around? [00:20] TJ-: So should I start looking to stuff 'set -x' everywhere possible and see what happens? === freeflying_away is now known as freeflying [00:23] garrettk: I think it needs it in "/scripts/local" at the start of the mountroot() function === freeflying is now known as freeflying_away [00:34] TJ-: Rebooting/testing. Back shortly. [00:41] hallyn, stgraber sent mail on my comment above to lxc-devel. [00:45] TJ-: Well, I attempted the boot. Unfortunately, the kernel decided to panic, complaining that it couldn't mount the rootfs. [00:46] FWIW, I stuck "set -x" in several locations. [00:46] garrettk: maybe you caused a syntax error in your placements? [00:47] Possibly relatedly, the kernel decided that there was a problem with the SATA link to one of the HDD, so now my array is rebuilding. [00:47] Maybe. Any good way to test that? [00:48] It sounds to me like there's something more going on with that newer kernel version. [00:48] TJ-: Maybe. [00:48] have you tried reinstalling it (apt-get --reinstall install linux-image-3.2.0-54-generc-pae) ? [00:49] I don't see much point - the busybox shell has happened with every kernel since I've upgraded. I've gone through lots. [00:49] You can still boot with older kernels though? [00:49] Yeah, prior to the upgrade. [00:50] That's how I'm able to talk to you now. :-) [00:50] which kernel version is the latest that starts? [00:50] 2.6.32-46 starts. 2.6.32-47 fails. [00:51] I've tried diffing the initrds, but there is a *lot* which changes between them. [00:51] Could it be that the MD is a red herring? maybe something *before* that is causing the failure... such as kernel modules that are supposed to be loaded [00:51] -47 got created as a part of the automatic "sync to head-of-line of 10.04 before upgrading" [00:52] TJ-: I'll buy that. [00:52] Anything special on that system, device-wise, that'd need a kernel driver module in the initrd to boot? [00:52] Nope. [00:53] It's an ETX board with an add-on NIC. 1 HDD, 1 SDD, mirrored/RAID1. [00:54] No encrypted filesystem. AMD board (x86_64 instruction set) [00:55] the hard disk and solid-state disk are the two halves of the mirror? [00:56] TJ-: Yup. [00:58] how well did that work out, before things stopped working entirely? [01:00] Erm. I didn't test it for long. My previous motherboard up and died, so I put this one in. The old harddrives were PATA connection, and the new MB only has SATA, so I pulled one of the old HDDs out (as a backup), PATA/SATA adaptor on the other, and plugged the SSD into the MB. I powered things up, fiddled with mdadm for about an hour and everything was up and working. [01:01] Then I discovered that the ethernet chips on the MB and the NIC have known driver issues which cause them to drop link every now and again. [01:01] So after a day or so I performed the upgrade to 12.04. [01:02] Everything seemed to work between the hardware upgrade and the software upgrade. [01:02] But it didn't get a lot of "soak" time because I wanted to be able to maintain TCP connections for longer than 20 minutes ... [01:04] Anything else I can fill in? [01:06] I don't think so... I need to go sleep, but I'll think about this and return to the test VM if I have any inspiration [01:07] TJ-: Thanks for all of your help. Rest well. I'll poke some more. [01:13] garrettk: oh man :/ not even an hour with it functional. bugger. :/ [01:14] sarnold: No - it worked for about a day or so before I upgraded. [01:14] It took me an hour to get mdadm to do what I wanted. [01:14] garrettk: ah [01:14] garrettk: still, not much time to get to know how the mixed raid thing worked.. I was just curious if you got ssd read speeds and hdd write speeds or what :) [01:16] sarnold: Something like that. It's a router/personal mail server. I've done a bit of benchmarking, but not much. I'm limited to the slowest device for write, but reads fly. :-) [01:16] So it works well for my needs. [01:16] garrettk: cool :) [01:16] The next storage device I'll buy will be another SDD, completing the slow (but financially sound) transition. [01:16] *nod* [01:17] after switching to SSD for my laptop, I -really- don't want spinning metal speeds again, but the cost per byte is awesome. :) [01:19] :) [01:19] we switched from a 100 15krpm disks to 48 ssd's for our san :) [01:19] patdk-lap: From which manufacturer? [01:19] purestor [01:20] patdk-lap: I don't know them - I work for NetApp. [01:20] patdk-lap: zounds. I drooled over those 15k drives for a loooong time. [01:20] I'm going to try something else. I'll be back in a few. [01:20] patdk-lap: it was hard for me to believe that my consumer ssd makes even those 15k drives look quaint and cute. [01:31] Back. And, I think I have something interesting to go on, too. [01:40] hey guys, can someone confirm if the paragon HFS+ drivers support volumes LARGER than 2TB [01:56] This may be a dumb question but why do the iso releases (http://releases.ubuntu.com/precise/) get built from the week of the point release (8/22/13 - https://wiki.ubuntu.com/PrecisePangolin/ReleaseSchedule) and the cloud images (http://cloud-images.ubuntu.com/releases/precise/release/) get built from the latest week (10/3/13) ? Is there a difference? [02:09] Ha! I think I've figured out what's going on! [02:13] Yup. Just confirmed it. :-) [02:13] The utility wait-for-root does, among other things, return the filesystem type. [02:15] For some reason, it decides to detect the filesystem type of the root device (dev/md0) as LVM2_member. This is true even for the kernels which work. [02:15] However, the newer images have updated the local script so that it passes that value into mount. [02:16] That is, before it was running mount -r /dev/md0 /root [02:16] Now it is running mount -r -t LVM2_member /dev/md0 /root [02:17] Clearly LVM2_member isn't a mountable filesystem type (and is incorrect in any case) so mount now fails. === zexcriz_ is now known as zexcriz === TDog_ is now known as TDog [03:46] sarnold: if this line isn't present in /etc/fstab at boot, it's added: tmpfs /dev/shm tmpfs defaults,ro,noexec,nosuid 0 0 === andol_ is now known as andol [06:08] justizin: yeah, quite a lot of the system won't work without that mount. You still shouldn't have thousands of them, but I expect a fair number of applications would fail without shm segments available.. [06:09] garrettkajmowicz: excellent troubleshooting :) nice to know there's a rational cause.. [06:11] sarnold: but /run is already mounted as tmpfs, what good does it do to mount /run/shm as tmpfs? do you know of any docs on what that's actually for? it comes up a lot in google and stackoverflow for chrome, but this is just a server running nginx, unicorn, and postgresql.. i see this a lot on ubuntu servers running postgresql, but i can't logically connect /run/shm with the shared mem strategy of postgres [06:11] nor do i understand why it would be mounted hundreds or thousands of times, or why if it's necessary it's not just documented and commented, but instead something forces that line into /etc/fstab in startup [06:12] anyway i'll keep digging :) [06:14] justizin: it's used for shmget(), also used by chrome and some X11 modules [06:14] justizin: the different mount points are so that they can get different mount options... moment.. [06:15] justizin: http://paste.ubuntu.com/6208065/ [06:16] justizin: if it were all on a single mountpoint, it'd be too easy for temporary files to collide with shared memory segments, or the other way around -- and system administrators would have no options for configuring how much memory to allow for different uses [06:17] ah fair enough, if they use hashes that can collide or something [06:17] but having 2700 of them, like i did earlier today? [06:17] that was on a test server noone uses and that doesn't have backups running, should be almost no transactions at all [06:17] the mystery remains! :-P [06:17] anyway thanks for being a sounding board, i appreciate it! [06:18] i am only chatting on it here because if we find a misconfiguration, we could help to fix it for everyone. [06:24] justizin: and obviously something is busted somewhere, that really shouldn't happen. :) have you had a chance to throw auditctl at the problem? [06:24] i have i need to dig the logs more [06:24] it's pretty late here, i'll dig them tmrw [06:24] excellent! [06:24] thanks [06:24] but i do think postgres has something to do with it [06:24] it's on all my staging / dev boxen where the whole stack is smashed together [06:24] i don't think it's on my prod postgres, but that's 9.0 on 10.04 [06:25] so i'd love to sort this before i launch fancy new SSD monster prod DBs on 12.04 with 9.2 or 9.3 postgres.. [06:25] and since it sets at boot, i'm not sure it's postgres' fault, maybe it's a dependency postgres installs, who knows.. [06:26] i hate to say, i love debuntu, but this is the sort of thing that has me thinking CoreOS. But if I went Docker I still think Ubuntu would be a better home. [06:29] no doubt, coreos has attractive points. [06:32] during setup on a hp proliant ML310 server I get stuck during the raid configuration of the onboard iscsi. it gives the message that it finds a iscsi raid configuration and it gives the option to activate yes / no when you select yes it request to configure the Iscsi drives by entering a source ip+port + user and password Also need to do this for the destination drive [06:36] it does but i'm just not confident enough about unattended environments to have an OS with zero useful userspace.. [06:36] I want to keep most of my devs, even my CTO if possible, hands off prod, but i want to be able to feel comfy and my lead ruby dev is an emacshead. ;d [06:36] but CoreOS is audacious fasho :) === Gnubie is now known as Guest87897 === Firartix is now known as Fira === Sprockt is now known as Sprocks === Screedo_ is now known as Screedo === Nolar2_ is now known as Nolar2 [07:30] hello [07:30] hello, i can ssh to my server fi,ne from within lan but not wan, i got a hostname from dyndns , made router changes where necessary, but when i connect from internet ip there s no way i can get rid of that "access denied" error with my correct password [07:39] smoser: bug 1236724 [07:39] Launchpad bug 1236724 in uvtool "cloud-localds missing from cloud-utils" [High,New] https://launchpad.net/bugs/1236724 [07:39] smoser: need cloud-localds in the cloud-tools pocket === rbasak_ is now known as rbasak [08:02] ....... === anternat- is now known as anternat [08:04] i cannot login to my ubuntuserver 12.04 via ssh from wan,port forwarded and did most. alwways " access denied" what must i do [08:07] !patience | anternat [08:07] anternat: Don't feel ignored and repeat your question quickly; if nobody knows your answer, nobody will answer you. While you wait, try searching https://help.ubuntu.com/ or http://ubuntuforums.org/ or http://askubuntu.com/ [08:08] anternat: sounds like you have a networking problem, rather than anything specific to Ubuntu Server really. By default, sshd on Ubuntu Server doesn't differentiate where a connection is coming from. Perhaps you're connecting to something else, not your Ubuntu Server? [08:09] not sure === zexcriz is now known as Guest13150 [08:10] but i get the connection and login screen fine just it doesnt accept me "access denied" [08:10] You can see the reason for "access denied" in /var/log/auth.log on your server, assuming that your connection actually got there. [08:12] alreayy looked in there :( [08:14] the way to tell that you see what you think you see is, on the server, tcpdump on the wan port on port 22/tcp. Then try to ssh to it. If you don't see traffic on the server, you're connecting to something else [08:14] if you do see traffic, then it's worth doing some debugging [08:26] how to dump that sgran [08:30] https://gist.github.com/delinquentme/6881477 [08:30] can someone explain what that initial test is checking for?? === dosaboy_ is now known as dosaboy [08:35] tcpdump -i port 22 [08:38] ty sgran but dunno what wan nickname is :( [08:38] i thought it was host name but it wasnt [08:39] anternat: if you run 'ifconfig', it will list your nics and their configuration. You should be able to figure out which nic is your wan port [08:40] hi guys. i'm trying to setup openstack on a few old servers for testing and evaluation. i'm trying to get maas running with it. the servers are all already running 13.04. i can add new nodes but there status is "Failed Tests". Is there a howto to add existing installations to maas? [08:42] oh myyyy [08:42] flooding as hell [08:43] that means you've picked your lan port and you're seeing your own ssh traffic [08:44] yup eth0 was the nick [08:44] ty [08:45] sgran can u have a look at here ? http://pastebin.ca/2464036 [08:48] anternat: that doesn't tell me very much [08:49] does the host fingerprint match your host? Did you see any traffic on eth0 when you were trying to connect? [08:52] sgran sorry bro, almost done, i think i have miscong in routers config that is my roter and not my server, sorry for the mess [09:02] I thought it would be something like that :) [09:08] :) TY very much all the same [09:26] Hi all, I am running apache on a server and I have locked down the directory... so I cannot CD into it... but just wondering how to run the following command from its parent directory... sudo rm apache2/*.gz [09:26] Without it considering the wildcard as part of the filename [09:30] hallyn: please see bug 1236726 - completely breaks the ubuntu-cloud template in Saucy I think. [09:30] Launchpad bug 1236726 in lxc "lxc ubuntu-cloud template is broken due to bogus tar option" [High,Triaged] https://launchpad.net/bugs/1236726 [09:31] Sorry if my question is not for ubuntu-server, I was told in #bash and #ubuntu to ask it here [09:32] d1rkp1tt: you could do something like "sudo sh -c 'rm apache2/*.gz'" if I understand what you want. [09:32] d1rkp1tt: you need a root shell to interpret the *, since your own shell cannot expand it. [09:32] will test that... thanks [09:33] same result [09:34] no such directory [09:34] so, I am in /var/log [09:34] to look into /var/log/apache [09:35] I sudo ls apache2 [09:35] But cannot delete files within that directory at the moment without resetting its perms... CD into it... then run 'rm *.gz' [09:36] I really just want to leave the perms as is, but tidy up.. rotate logs, move logs off for analysis etc [09:39] Can you pastebin an exact copy and paste of what you tried? [09:39] (and the subsequent error) [09:39] Also repeat the same thing but with ls instead of rm, and include an exact copy and paste of that, too. [09:46] ls [09:46] works [09:46] will paste though [09:47] oh wait... [09:47] huh.. [09:47] so the last command looks like it worked, even though it errored [09:47] ... [09:47] Thanks for that [09:48] rbasak, mind If I ask what a root shell is? [09:48] It's a shell running under root privileges === anternat is now known as Guest20618 [09:51] rbasak, Thanks for your help [09:51] np [09:55] rbasak, You led me to this... http://linuxcommand.org/lc3_lts0080.php which is great.. [09:55] on expansion [09:57] d1rkp1tt: that looks pretty good, thanks. [09:57] * rbasak makes a note [09:58] You know anything about apache2 log rotation on Ubuntu by any chance? [10:00] It's probably handled by logrotate. [10:00] See /etc/logrotate.d/ [10:01] Thanks [10:17] hmm. Anyone here responsible for the cloud-repo? The ceilometer in the havana tree is hardly usable [10:35] sgran, are you using -proposed or -updates? === freeflying_away is now known as freeflying [10:36] both [10:36] sgran, OK - so its out of date [10:36] it seems so [10:36] sgran, the staging ppa is the most up-to-date right now [10:37] where is that? [10:37] ppa:ubuntu-cloud-archive/havana-staging [10:37] sgran, once we have glance rc1 fixed up that will be synced through to -updates [10:37] sgran, this is a good report to look at - http://reqorts.qa.ubuntu.com/reports/ubuntu-server/cloud-archive/havana_versions.html [10:38] cool, thanks - that's a help [10:40] now to figure out how to get hold of that repo === freeflying is now known as freeflying_away === freeflying_away is now known as freeflying === yofel_ is now known as yofel === Guest13150 is now known as zexcriz [12:45] Hi all [12:48] * Derbedeu from where i can buy a good shell ? [13:12] adam_g, I think we are probably ready to start with https://code.launchpad.net/~openstack-charmers/charm-helpers/to_upstream/+merge/189838 [13:15] Does using MAAS imply using juju in any way? [13:19] soren, no [13:20] although juju has been the primary consumer and hence driver of maas features to-date [13:20] errg [13:20] gareth@kitsunet:~$ sudo swapon -f -U f831c4d1-bc6f-4d5b-b2e1-09439a4ddbb5 [13:20] swapon: /dev/sda1: swapon failed: Invalid argument [13:20] f-what?! [13:25] nvm, fixed it, had to re-init swap === freeflying is now known as freeflying_away [13:28] Hello, I am seeing segfault at 0 ip 00007fda3efe2362 sp 00007fda27564dc8 error 4 in libc-2.11.1.so[7fda3ef5a000+17d000] in my logs on my asterisk server, I am using asterisk from repo. Is there anyway to convert this into useful information without recompiling ? [13:55] I have multiple 3G data usb cards. I can connect each of them separately and access the internet. All 3g cards are using differents ISP. Is there any way i can aggregate the bandwidth of these cards to enjoy the combined speed? What i mean is simultaneously plugging in all the cards and getting the sum total of the bandwidth. I would like to create a big virtual link with high bandwidth upload (7 Mbps for example) to sent a high quality real time video stream [13:56] jamespage: Cool, thanks. [13:56] soren, np [13:57] zul, adam_g: did either of you request ceilometer and heat be added to the MRE yet? [13:59] plm: yes, with a combination of bonding and VPN to a remote end-point which Masquerades the bonded link [14:04] TJ-: hmm.. more details please =D I was researching about multilink ppp [14:04] plm: it can be done but it's pretty advanced stuff. You generally need help from your ISP(s) unless you control the other end or are sending UDP only. [14:05] rbasak: I don't have any help from ISP, I need to do something trnasparent.. [14:06] plm: see http://lartc.org/howto/lartc.rpdb.multiple-links.html. You can't do something completely transparent without help from the other end, since your three links will have three different source IP addresses. [14:06] rbasak: I can control in other side, will have my server... and about UDP is not problem. I need just sent video streaming , and udp is fine for this [14:06] plm: I use it for bonding multiple links into a datacenter, but as rbask says, its advanced and can take some time to perfect. [14:06] Ah if you control the other end then you can tunnel and you're fine. [14:06] (not that it makes it any easier) [14:07] any suggestions for a union filesystem? im looking at mhdeffs, but it uses fuse and slows down transfers [14:07] rbasak: just for clarify: when I talk about I control in other side, is not other side where ppp are connected, just I will have a server in datacenter where I can access it after 3g ppp established with the ISP [14:08] plm: yeah that's fine. Your ISPs will see three streams; your servers will split and recombine them. [14:09] rbasak: hmmm.. [14:10] rbasak: are you talking about I will not have a combination of bonding, but just split it in box1 and recombine them in box2? [14:28] stgraber, hallyn how do you want me to handle this... [14:29] http://sourceforge.net/mailarchive/forum.php?thread_name=alpine.DEB.2.02.1310072033430.4094%40brickies&forum_name=lxc-devel [14:29] wow, a sourceforge link. that takes you back. [14:30] smoser: hallyn is out this morning I believe. I just acked and pushed your version of the patch upstream, I'm now updating the patch in the Ubuntu package to match [14:30] thanks, stgraber [14:31] rbasak, https://bugs.launchpad.net/uvtool/+bug/1236724 [14:31] Launchpad bug 1236724 in uvtool "cloud-localds missing from cloud-utils" [High,New] [14:31] really you just need to depend on the newer version [14:32] your 'apt-get install' didn't have a reason to upgrade cloud-utils so it didn't [14:32] smoser: can you confirm this looks right: http://paste.ubuntu.com/6209552/ [14:32] stgraber, can you give a bit and i'll actually *test* it ? [14:32] :) [14:33] smoser: sure :) [14:33] it does look right though [14:33] smoser: oh. Sorry. I'll look - thanks. [14:34] rbasak, i'd just depend on >= 0.26 [14:34] or you can just do 0.27 to be safe. [14:39] zul: OK - copying everything apart from xen to -proposed [14:39] sweet [14:40] jamespage: glance built ok? [14:40] zul, yes [14:40] that is weir [14:40] d [14:40] jamespage: huzzah [14:40] run_test.sh always resulted in me pressing the power button before [14:40] jamespage: i think we were a couple of versions behind on dependencies that the testsuite needed [14:41] zul, best cloud archive ever then? [14:41] jamespage: the last one is always the best one [14:41] zul, yeah - you just get good at it then another lts comes along [14:41] lol [14:42] next cycle will be a walk-in-the-park [14:42] lol [14:42] is it ever.... [14:42] * zul checks in a mental insitution [14:42] smoser: in fact just cloud-image-utils (>= 0.27) should do, right? [14:43] * rbasak doesn't need growpart, vcs-run or ec2metadata [14:45] yeah. [14:45] that shoudl be fine. [14:46] stgraber, i'm still trying to test [14:46] but one thing is don't drop the "" around it. [14:46] as thats a bug itself [14:46] i dont know if i did that or you did [14:47] smoser: Serge added the "" in his patch, you didn't add them in yours so they appear as removed in the diff [14:48] when I close vi with ZZ command, it does not close, all I get is ~~~~~~ [14:48] how do i go back to command line [14:48] ? [14:48] sgran, fyi I just started pushing updates into proposed - will take a few hours to build across the board [14:48] stgraber, well, i didn't patch his pathc locally. ijust did it against trunk at the time. [14:48] -updates pocket will take less time [14:48] smoser: right and trunk never contained the "" AFAICT [14:50] jamespage: https://bugs.launchpad.net/ubuntu/+source/python-cinderclient/+bug/1236901 [14:50] ah. ok. well, add them. i cant stand the thought of being git-blamed for code that odesn't deal with spaces in a filename [14:50] :) [14:50] Launchpad bug 1236901 in python-cinderclient "FFE for python-cinderclient 1.0.6" [High,New] [14:51] :) [14:52] jamespage: \o/ was just going to ask if you could do some of that :) [14:52] that will be a big help, thanks [14:52] hallyn: ping [14:52] smoser: http://paste.ubuntu.com/6209640/ [14:52] I'm trying to do some pre-release bug fixing, and having something to work against makes it lots easier :) [14:52] sgran, ~3 hrs or so to build and sync out to the archive [14:53] sgran, ditto - but I can run from PPA - guess you are firewalled right? [14:53] yes, sadly - we have a mirror machine that reserves internally, though [14:53] so I can pull from the ppa, it just needs a bit of faff :) [14:54] sgran, I've hit one big-ish issue [14:54] bug 1236439 [14:54] Launchpad bug 1236439 in neutron "switch to use hostnames like nova breaks upgrades of l3-agent" [High,New] https://launchpad.net/bugs/1236439 [14:54] TJ-: I'm not certain if you saw the posts I made after you dropped off for the evening last night, but I figured out what was going on. [14:54] Data and solution posted: http://askubuntu.com/questions/307509/upgrade-to-12-04lts-dumps-to-busybox-on-boot [14:54] and a minor niggle with nova interface attachment in bug 1236875 [14:54] Launchpad bug 1236875 in nova "unable to hotplug additional network interfaces into instances" [High,New] https://launchpad.net/bugs/1236875 [14:55] sgran, are you testing ceilometer? or heat even? [14:55] (not got that far yet) [14:55] both, yeah [14:56] I'm not using l3 agents - we use hardware routers for the gateway here [14:56] thankfully :) [14:56] I'm trying to get a heat autoscaling group that behaves like an AWS one - instance autorestarting, add to ELB, etc [14:57] I think I have the last set of patches to go in now, once I figure out which metrics from ceilometer indicate an 'unhealthy instance' [14:57] garrettkajmowicz: No I didn't, I disconnect when I leave. I'll go have a read! [14:59] garrettkajmowicz: !!! I was looking hard at "${FSTYPE:+-t ${FSTYPE} }" too! [15:02] stgraber, http://paste.ubuntu.com/6209672/ [15:04] smoser: ok, good, pushing the second patch upstream and cherry-picking that into Ubuntu [15:04] so http://paste.ubuntu.com/6209640/ is fine with me. it has the additional quotes around "tarname" and "imgname" (versus what i tested), but that should not cause problems. [15:05] TJ-: Now I create a Launchpad account to file a bug against initramfs-tools (for wait-for-root detecting the wrong FS type) and go on my merry way. :-) [15:05] garrettkajmowicz: I'll work on it [15:14] smoser: lxc uploaded [15:15] garrettkajmowicz: Looks like it could be a udev issue, since wait-for-root is using the udev db to get the FSTYPE [15:15] jamespage: https://code.launchpad.net/~zulcss/python-cinderclient/1.0.6/+merge/189885 [15:15] TJ-: Interesting. [15:16] Is there any way I can manually test or see that on my system? [15:16] stgraber, thank you. [15:17] smb: ping [15:17] garrettkajmowicz: The udev report? or what /sbin/wait-for-root returns? [15:18] stgraber, you can adjust state of https://bugs.launchpad.net/ubuntu/+source/lxc/+bug/1236577 as you see fit. [15:18] Launchpad bug 1236577 in lxc "container's /home/ubuntu/ spawns with incorrect permissions, preventing SSH access" [Undecided,New] [15:18] (ie, fix-committed in lxc or -released, whatever you consider "committed to trunk") [15:20] TJ-: The udev report. I know what wait-for-root returns. :-) If this is due to udev somehow, we ought to be able to work this backwards and figure out *where* this is coming from. Ideally I'd like to be able to see udev spitting out something incorrect so we know that the problem lies there or below instead of in wait-for-root. [15:20] garrettkajmowicz: Yes, I'm looking at that now. [15:21] garrettkajmowicz: It strikes me that maybe there are some leftover LVM metadata markers on one (or both) halves of the RAID1 array which cause the issue. Have the underlying disks ever had LVM ? [15:21] smoser: is that related to the thing that we just fixed? [15:22] TJ-: I ... don't know. Maybe? [15:22] This system has been in use, with upgrades, for 6 years now. [15:23] garrettkajmowicz: FS detection is an art not a science, it can sometimes depend on the order in which the fs-sniffer tools are called, if other (unused) metadata is also on the raw devices [15:23] stgraber, it is the bug that is "--numeric-owner is necessary" [15:26] TJ-: Running fdisk against the underlying drives shows the partitions to be of type "fd Linux raid autodetect". There's no partition table on the partitions themselves, or /dev/md0 [15:26] garrettkajmowicz: If you boot into the bad initrd with "break=mountroot" you can use udevadm to check out the info on md0 [15:28] garrettkajmowicz: "/sbin/udevadm info --query=all --name=/dev/md0" [15:28] TJ-: Both the good and bad initrd returned an incorrect fstype. It's just that the "good" image didn't pass the incorrect type to mount and thus fail. :-) [15:29] garrettkajmowicz: good point, lets find out what udev reports then... can you pastebin the entire report? maybe redirect it to a file in the rootfs after mounting it, then pastebinit after boot is complete? [15:29] smoser: right, so marking fix released in Ubuntu then [15:29] TJ-: http://pastebin.com/3DJqUwUH [15:30] zul, cinderclient +1 [15:30] garrettkajmowicz: So there's the issue... we've gone from initramfs-tools to udev... and I suspect we'll end up in another package yet [15:30] jamespage: thanks [15:33] ok back in about 40 minutes [15:34] TJ-: I think I've found it. Just a sec. [15:35] TJ-: http://pastebin.com/dy6pi2a0 [15:36] garrettkajmowicz: Yes, that matches what udev reports [15:36] So, there is *some* leftover bits of LVM on the block device, I guess. [15:36] garrettkajmowicz: Yes, which explains everything! [15:36] garrettkajmowicz: No wonder I couldn't reproduce it :) [15:37] So ... outside of reading raw disk blocks, how would I know WTF was going on? [15:38] garrettkajmowicz: Because it didn't boot... that's a big clue :) [15:39] Thanks. :-) [15:39] garrettkajmowicz: You'll need to "pvremove" on that [15:40] I'm attempting to run it remotely. I guess one of the things which through me for a loop was: Incorrect metadata area header checksum. That makes me think that it isn't a LVM volume. [15:42] garrettkajmowicz: At least its sorted and not a major bug in the code! [15:42] Even though it is marked on disk as being one. [15:42] And yet mount has no problem with it. [15:43] Okay - so this is probably a supportability issue of some kind. Any bug you think I should file over this? Maybe for better diagnostics or something? [15:47] garrettkajmowicz: No, its a sysadmin bug! Always ensure devices are correctly wiped when re-assigning [15:48] garrettkajmowicz: My guess would be if lvm wasn't installed in the initrd, it'd have worked. You recall there was a /tmp/20-lvm-mountroot-failed or whatever file? That, in retrospect, was a big clue, alongside the /tmp/10-mdadm file [15:48] garrettkajmowicz: Only now though do we realise the implications [15:58] TJ-: True, though I'd argue that running mke2fsck on a device really ought to, you know, make it that. [16:00] garrettkajmowicz: It did ... but the sectors it writes to won't always coincide with those of a myriad other file-system and container formats and their meta-data, and you don't want to have to zero every block just to format a FS. [16:01] rbasak: http://simonmott.co.uk/vpn-bonding [16:07] TJ-: Right. However, if the blocks are used to identify the other FS. IDK. I'm of the view (and experience) that when something isn't quite right, there should be an obvious indication of this, and what should be done. The wait-for-root was returning LVM2_member for years and at least 2 releases without the system breaking hard. [16:09] garrettkajmowicz: I think it is the difference between wait-for-root delegating to udev, and mount which delegates to the various mount.${FSTYPE) tool helpers that do the detection [16:10] garrettkajmowicz: I agree that it should highlight the reason for the failure more prominently in the mount failed messages, rather than just a whole list of reasons why it may have failed [16:11] TJ-: That makes total sense. And, at the same time, a nice message to the console of 'mount command failed' would have been nice. Even better if mount kicked out something saying explicitly that the fstype specified wasn't recognized or didn't match the fs found. [16:11] jamespage, https://bugs.launchpad.net/neutron/+bug/1156932 [16:11] Launchpad bug 1156932 in python-novaclient "User can't modify security-group-rule via nova-api if there are duplicated security group name" [High,Confirmed] [16:11] how did you triage that as "High" priority ? [16:11] that seems a very simple case of "well don't do that then" [16:11] Rather than having to enable debug mode and hope you can save/read the file somewhere. [16:12] smoser, that needs revising [16:12] at the time we thought it was more impacting that it is now [16:12] k [16:12] well, i just moved to low [16:12] done [16:12] me too [16:12] i'd just leave this to be fixed upstream if someone seems motivated [16:13] garrettkajmowicz: the thing is, there is no fstab to refer to in initrd, only the "root=device" so it had to use auto-detect and rely on it. That's why your fix was to add the rootfstype to the kernel cmd-line. So the tool has no way of knowing what fs-type to expect [16:15] TJ-: What is the "it" we are talking about here? If the script doesn't pass in -t= to mount, everything works fine. There are only a few cases where that value makes any sense. You're trying to mount a filesystem as a different compatible filesystem. Eg. mount ext3 as ext2 to avoid journal replay. [16:15] it == mount [16:16] Why does mount need to be passed the fstype at all? It seems to autodetect everything just fine. [16:16] garrettkajmowicz: I'm not sure but I'd guess the reason for the switch is to drop the requiremernt to include in the initrd all the mount.${FSTYPE} helper binaries, because udev is already there and able to do the job [16:20] Poking around, I suspect it might be for the "treat this blog of random bytes as an undetectible encrypted filesystem". I only see helpers for fuse and encrypted filesystems. [16:26] garrettkajmowicz: revision 151 in Dec. 2009 introduced wait-for-root and the "mount ${roflag} ${FSTYPE:+-t ${FSTYPE} }${ROOTFLAGS} ${ROOT} ${rootmnt}" [16:31] TJ - the -t was passed in to a different section, the one dealing with loopback filesystems. The main mount part did not pass in the fstype. [16:39] on 12.04 anyone filter dhclient message from rsyslog have a howto? === JanC_ is now known as JanC [16:59] hi all. trying to find a scalable NAT solution here, would linux allow me to use several IP addresses for NAT if I have thousands of clients behind it, as to scale better? With only 65k ports and thousands in TIME_WAIT, it won't scale too well [17:03] RoyK: you just give the SNAT iptables target a --to-source range of IPs. [17:08] rbasak: any way to do this dynamically based on load? === psivaa is now known as psivaa-afk === jkyle__ is now known as jkyle === psivaa-afk is now known as psivaa [17:35] smoser: but you didn't add --numeric-owner to the *First* tar xf (from imgfile)? [17:36]