shawn1Could anyone help me?  I'm experiencing a problem similar to what is described here:01:33
shawn1But my ubuntu 12.04 LTS server doesn't seem to have an /etc/rc.conf file01:34
shawn1So I don't know where to implement the answer given in that article or even if that answer is relevant on my version of Ubuntu Server01:34
sarnoldshawn1: you're getting "permission denied" when you're trying to ssh to a specific server?01:36
shawn1yes.  Let me copy and paste my own outputs01:36
shawn1Shawns-MacBook-Pro:~ shawnshipley$ ssh -vvv
shawn1OpenSSH_5.6p1, OpenSSL 0.9.8r 8 Feb 201101:37
shawn1debug1: Reading configuration data /etc/ssh_config01:37
shawn1debug1: Applying options for *01:37
shawn1debug2: ssh_connect: needpriv 001:37
shawn1debug1: Connecting to [] port 22.01:37
shawn1debug1: connect to address port 22: Permission denied01:37
shawn1ssh: connect to host port 22: Permission denied01:37
sarnoldshawn1: I have a wild guess that your laptop and your server do not agree on netmask or their network address01:38
shawn1how can I fix that?01:39
sarnoldshawn1: you'll have to find the correct settings to use, probably from your router, and then ensure that your server and your macbook both have the same settings for netmask, broadcast, and so forth01:40
sarnoldshawn1: (the 'permission denied' here is probably because you're trying to ssh to a broadcast address, for a network and netmask (a /24))01:40
sarnoldshawn1: if you manually assigned an IP address for your server, don't use .255. :)01:41
shawn1I didn't manually do that.  Why would that present a problem, though? (just so that I understand for future reference).01:42
sarnoldshawn1: normally the 'last' address in a network range is a broadcast address that can be used to contact all hosts in a subnet. but it only works with UDP and ICMP, not TCP, and it requires superuser privileges, and most hosts don't respond to broadcast ping requests any more anyway :(01:43
sarnoldshawn1: so, if your network is and the netmask is /16, the broadcast address would be; if the network is and the netmask is /24, then the broadcast is
shawn1.255 is the global address, .5 is the local address, but it won't ping to .501:45
sarnoldshawn1: it's not always 255, that's just convenient with /8 or /16 or /24 netmasks -- but it does stand out clearly here :) hehe01:45
shawn1This is an odd question, but is your name Sarah Arnold?01:45
sarnoldshawn1: indeed no, seth arnold. hehe. :)01:46
shawn1I know a Sarah Arnold, so I was just wondering01:46
shawn1My question:  Do you know a way that I can completely reset the router?01:47
shawn1I configured it in windows and then later had to reinstall windows and it would only let me connect as a guest.01:47
sarnoldshawn1: depends on the router. often holden the power button for twentyseconds or something will zero the memory.01:47
shawn1It'll completely reset everything and let me reconfigure?01:47
ag763Any ideas on why wlan0 wouldn't start on boot when configured in interfaces file but does with 'ifconfig wlan0 up'01:50
sarnoldshawn1: quite often; it varies from vendor to vendor, but that's a good first shot. hehe. :)01:50
sarnoldag763: pastebin your interfaces?01:51
ag7631sarnold, http://pastebin.com/1CGSyeQK01:55
sarnoldag7631: do you need a module loaded first? maybe add the module name to /etc/modules?01:57
sarnoldag7631: hope that helps, it's time for me to bail :) have a good night, have fun01:58
sarnoldshawn1: any luck with 20-seconds? :)01:58
ag7631sarnold, I would expect to see a dmesg or syslog error for that right?01:58
sarnoldag7631: hrm. dunno. seems plausible though :)01:59
ag7631sarnold, looking through both I don't see anything.  I can start it right after booting without any issues, seeing nothing in dmesg or syslog I'm at a loss.02:01
shawn1Netgear can be frustrating....02:04
shawn1Okay.  I think I found what I needed02:04
=== Lcawte is now known as Lcawte|Away
shawn1the netmask address matches02:16
shawn1I was making a very dumb mistake!02:18
shawn1I was accidentally trying to ssh the broadcast address02:18
shawn1I'm connected remotely02:19
shawn1(you can tell that I'm new at working with servers)02:20
shawn1Thanks, Sarnold02:42
anepanaliptoshow do we ceck the health of a filesystem?02:55
virusuyanepanaliptos: fsck /filesystem03:00
virusuya good practice is umount that filesystem first03:00
virusuyand you fsck the partition03:00
anepanaliptoswhat can do you do over ssh03:00
anepanaliptosis there anything just to show the health 'status' so to say?03:00
virusuywell, fsck is the most secure way to see if a filesystem is health or not03:02
virusuydo you have any kind of read/write errors or something?03:02
anepanaliptoswell i just ran the server too many times on/off the harsh way03:03
anepanaliptosand i just want to make sure the disks are ok03:03
virusuyanepanaliptos: uhmm, fsck all th way03:03
virusuybut , yeah, its really importat unmount those filesystem first03:04
anepanaliptosyeah i once did it mounted03:04
virusuyand also there is a way to force fsck before boot in the next reboot / power on03:04
anepanaliptosmade everything brand new03:04
anepanaliptoso yeah? how's that?03:04
virusuyuhmm .. i didn't recall, but let me search for you03:04
anepanaliptosoh no that's ok03:04
anepanaliptosill rtfm03:04
anepanaliptosim not lazy, just unknowlegeable.03:05
virusuythat web page was written in 2011, info could be old .. but i know there is a way to force that03:05
LLckfanI am trying to reconnect a blu-ray player to my router wirelessly and keep get dhcp cannot be acquired. Is there a way to fix this?04:05
anepanaliptoswhere is dhcp provided from?04:08
LLckfanI do not knwo04:08
shawn1can anyone help me?05:04
shawn1I set up port forwarding and made my server IP static in order to connect through SSH to my server over the internet.05:05
shawn1but I don't know what command to use from the client machine in order to access the server.05:05
shawn1well, yes05:05
shawn1but to I just ssh my server login name and external ip05:06
shawn1or do I need to specify port 8005:06
qman__ssh youruser@
qman__absolutely not, port 80 is for http05:06
shawn1well that's the point05:06
shawn1I'm trying to connect over http05:06
shawn1maybe I stated that incorrectly, sorry05:07
qman__you can't ssh over http as far as I am aware05:07
qman__I don't know why you would want to05:07
qman__in fact, quite the opposite is a typical case, http tunneled over ssh05:07
shawn1here's what I did05:08
shawn1in my router settings, I clicked on the 'port forwarding' option05:08
qman__if you want to SSH from the internet, you need to forward port 2205:08
shawn1let me change things here05:09
qman__and I suggest that you either install fail2ban or configure a limiting firewall to defend against brute force attacks05:09
shawn1well sweet05:10
shawn1that was easy05:10
indistyloFolks, Having problem starting Jboss server, it says Jboss home pointing to different installation , Output can be seen in the URL( http://paste.ubuntu.com/5953884/ ) please suggest some solutions05:10
shawn1Thank you very much  :)05:10
indistyloshawn1 Can you resolve my problem, any idea on Jboss server ?05:18
indistyloshawn1: Can you resolve my problem, any idea on Jboss server ?05:18
indistyloqman__: Can you suggest something about Jboss? any idea?05:19
qman__sorry, I don't know anything about it, I try to stay as far away from java as possible05:23
shawn1Sorry, indistylo05:27
shawn1I just got your messages05:28
shawn1and I'm one of those people who are on here for help05:28
indistyloShawn1: No issues05:28
shawn1in questions related to server issues, qman_ will know much more than me  :)05:28
indistyloshawn1: Its alright I am juggling with problem still, Ya i had already asked qman__  : waiting for his reply !05:29
foo357Hm, I want to change my home directory but usermod complains about me being logged in (which is true), but I am the single logon-user available on the machine...05:39
anepanaliptosfoo357: root.05:48
foo357anepanaliptos: I connect to the machien through ssh and login with my account05:50
foo357anepanaliptos: I don't think it's possible to directly login as root that way05:50
ScottKUse sudo -i05:53
indistyloFolks, Eclipse not starting in ubuntu12.04, I installed in /usr/share/eclipse directory and created eclipsed.desktop but its not starting, Kindly suggest solutions05:54
ScottKAsk for help on #ubuntu since that's not a server issue.06:05
shawn1got it!07:02
=== thumper is now known as thumper-afk
=== three18t- is now known as three18ti
_rubenugh .. why is Azure so limited .. they make it really hard to make linux clusters (no floating ips for HA services, etc)08:43
_rubenneed to find something decent to replicate/sync my data between nodes somehow08:44
=== smb` is now known as smb
=== Lcawte|Away is now known as Lcawte
=== Jikan is now known as Jikai
=== Jikai is now known as Jikan
=== matanya_ is now known as matanya
geserstgraber: Hi, can I ask you something about the isc-dhcp-server6 upstart job? I'm trying to figure out how to fix bug #118666211:37
uvirtbotLaunchpad bug 1186662 in isc-dhcp "isc-dhcp-server fails to renew lease file" [Undecided,Confirmed] https://launchpad.net/bugs/118666211:37
stgrabergeser: sure11:38
stgrabersounds like a regression that would have happened when we switched from our own privilege dropping code to upstream's --paranoia option11:38
stgraberit could be that in the past, dhcpd only dropped privileges after all the files were opened but now it happens a bit earlier, causing the -ENOPERM11:39
geserthe lease file itself get updated, but it can't get rotated as dhcpd is running as dhcpd but everything is owned by root11:40
geserso no updates to leases~11:40
stgraberah right, so we probably should just give the dhcpd user ownership of the dir and be done with it?11:41
geseralmost, with chown dhcpd it gets a little bit farther till: dhcpd: Can't backup lease database /var/lib/dhcp/dhcpd6.leases to /var/lib/dhcp/dhcpd6.leases~: Operation not permitted11:41
geserthe apparmor profile needs an update too: kernel: [2773234.120934] type=1702 audit(1375788247.840:13): op=linkat action=denied pid=31888 comm="dhcpd" path="/var/lib/dhcp/dhcpd6.leases" dev="dm-0" ino=275711:42
geserdo I need to specify that dhcpd can write (create new files) to /var/lib/dhcp into the apparmor profile?11:44
stgraberI'm surprised the existing apparmor rule doesn't cover that11:46
geserthere is "/var/lib/dhcp/dhcpd{,6}.leases* lrw" but it seems it doesn't cover creating new files11:48
geserlooking at the code, dhcpd removes the old leases~ and renames leases to leases~ before it creates a new leases to write into11:49
BrixSati need some info, i have 200 servers under my management. Is there any way i can have a key management system, say i want one key to all servers and i dont want to ssh them manualy or in a script, since a server can be now offline and later online11:52
BrixSati need to be able to generate keys and distribute them11:52
geserjdstrand: can you help me to understand why the dhcpd apparmor profile issue? I've in syslog: "dhcpd: Can't backup lease database /var/lib/dhcp/dhcpd6.le12:10
geserases to /var/lib/dhcp/dhcpd6.leases~: Operation not permitted"12:10
geserand later "kernel: [2773234.120934] type=1702 audit(1375788247.840:13): op=linkat action=denied pid=31888 comm="dhcpd" path="/var/lib/dhcp/dhcpd6.leases" dev="dm-0" ino=275712:11
geserisn't "/var/lib/dhcp/dhcpd{,6}.leases* lrw" enough to allow it?12:11
jamespagezul, bah - ordering of package build in havana-proposed is creating installability issues12:14
zuljamespage:  like what?12:16
jamespagezul, "python-keystone : Depends: python-sqlalchemy (< 0.8) but 0.8.2-1~cloud0 is to be installed"12:16
zuljamespage:  grr..12:17
zuljamespage:  hmmmm12:18
jamespagezul, if I rebuilt it now against staging it gets the correct versioned depends12:18
jdstrandgeser: what version of ubuntu are you seeing this?12:18
zuljamespage:  right12:18
zuljamespage:  so we need to rebuild things?12:19
jamespagezul, yeah - I'm just pushing stuff with a ~cloud1 with 'No change rebuild for new version of SQLAlchemy."12:19
zuljamespage:  ack12:20
zuli just uploaded a new version of pbr for trunk packages (as of 5 minutes ago)12:20
geserjdstrand: ubuntu server 13.0412:20
jamespagezul, OK12:22
jdstrandgeser: /var/lib/dhcp/dhcpd{,6}.leases* lrw is not in the 13.04 profile. did you add it yourself?12:22
jdstrandoh, dhcp*d*12:22
jdstrandhold on12:22
StathisAhello, i'm trying to get my head around Tar syntax...i got two folders.../mnt/source & /mnt/destination...how can i tar the contents of /mnt/source to a tar file in /mnt/destination without running the command from neither of them?12:23
geserjdstrand: for the background: I'm trying to fix #1186662, and got this far after changing the owner of /var/lib/dhcp to dhcpd so that dhcpd can write to it again12:26
jdstrandgeser: see 'man apparmor.d'. do the source and target files meet the criteria for 'Link mode'?12:29
jdstrandgeser: it isn't clear to me if the apparmor.d man page is talking about the apparmor permissions or the apparmor+DAC permissions12:32
jdstrandgeser: I need to call in reinforcements12:32
jdstrandjjohansen: I'm not sure what is going on with geser ^ and his quest to fix bug #118666212:33
uvirtbotLaunchpad bug 1186662 in isc-dhcp "isc-dhcp-server fails to renew lease file" [Undecided,Confirmed] https://launchpad.net/bugs/118666212:33
jdstrandjjohansen: backscroll 24 minutes from this timestamp12:33
geserjdstrand, jjohansen: does owner of the file matter when linking? dhcpd is run as dhcpd and dhcpd6.leases is owned by root currently12:34
jdstrandthat's the bit I'm not sure of12:34
jdstrandI would think you would see a dac_override entry if that were the case12:35
jdstrandgeser, jjohansen: I wonder if it has anything to do with 4.2.4-1ubuntu4 and bug #102852612:37
uvirtbotLaunchpad bug 1028526 in isc-dhcp "dhcpd failed to start with apparmor denied: capname="dac_override"" [High,Fix released] https://launchpad.net/bugs/102852612:37
Madkissroaksoax: y0.12:47
hallynlifeless: are you still seeing the libvirt memory leak with virt-manager?  I was running under valgrind to inspect...  but now i can't reproduce it, even without valgrind12:51
Madkissroaksoax: i have afresh pcmk 1.1.10+git here that fixes some nasty bugs.12:57
=== p0wp0w_ is now known as p0wp0w
zuljamespage:  https://code.launchpad.net/~zulcss/cinder/babel/+merge/17874713:20
zuljamespage:  ping i was thinking....shouldnt we building against the -proposed pocket in the openstack-ci lab since we would be catching stuff much sooner13:24
zullike the sqlalchemy stuff13:25
roaksoaxMadkiss: perfect ill get that synced13:51
Madkissroaksoax: where is your stuff?14:05
hallynstgraber: http://people.canonical.com/~serge/lxc-resolve.debdiff14:06
stgraberhallyn: I think it's fine14:08
Techdude1011I am looking for suggestions for snmp trap software. Currently I am using snmptrapd but I would like to add interface descriptions from switches14:14
roaksoaxMadkiss: ill put it in github in a bit14:15
hallynstgraber: cool14:20
jamespagezul, sorry - was OTP14:27
jamespagezul, thats a really good point14:28
zuljamespage:  i could do that if you want14:28
* jamespage thinks14:28
zuljamespage/roaksoax: https://code.launchpad.net/~zulcss/cinder/babel/+merge/17874714:32
zuljamespage:  typo fixed14:38
jamespagezul, someone synced over your python3 changes in stevedore btw (might have been Daviey)14:40
zuljamespage:  grrr...14:40
* zul shakes his fist at Daviey14:41
zuljamespage:  mind if i push this cinder branch i fixed the changelog entry14:43
jamespagezul, +114:43
zuljamespage:  thnak14:44
zuljamespage:  the sqlalchemy fix we had in nova got merged fyi14:46
zuljamespage/roaksoax: https://code.launchpad.net/~zulcss/keystone/oslo.sphinx/+merge/17877814:59
zulSpamapS:  pinger15:13
zuljamespage:  what did you do to fix the autopkgtests with nova?15:15
smosermr hallyn15:19
smoserare you around sir ?15:19
hallynsmoser: yup15:37
hallyn[ERROR] ./stack.sh:698 nova-api did not start15:37
hallynguess i'm really not meant to use this on saucy15:37
smoseri have 2 [recurring] lxc questions.  a.) lxc-start-ephemeral minus the ephemeral . b.) lxc clone hooks.15:38
smoserhallyn, i dont know. i think i used it last on raring fairly painlessly.15:38
smoseri think my ud-devstack worked start to end last time i used it.15:39
zuljamespage/roaksoax: this is really needed to get python-pbr through: https://code.launchpad.net/~zulcss/heat/sqile-fix/+merge/17878615:39
hallynsmoser: what is ud-devstack?15:40
hallynstgraber: this is great, and i can also resolve c1.lxc from c2.lxc from there.  cool.  pushing.15:40
SpamapSzul: pongitola15:44
zulSpamapS:  any idea when heat is going to switch over to neutronclient?15:45
smoserhallyn, ^15:45
smoserhallyn, see my questions above ?15:45
smoseri was poinged by juju team.15:45
smoserthey're interetsed in clone being faser15:45
smoserand those are 2 things that they'd need.15:45
* hallyn reading up15:45
hallynsmoser: what time, which chan?15:46
SpamapSzul: got a bug? I'll grab it and submit a patch now.15:47
SpamapS(or file the bug)15:47
hallynoh i see.  weird15:48
hallynmy eyees just totally glazed over that15:48
smoserhallyn, what?15:49
smoserquestions above.15:49
smoseri have 2 [recurring] lxc questions.  a.) lxc-start-ephemeral minus the ephemeral . b.) lxc clone hooks.15:49
smoserand ud-devstack is : launch instance with --user-data of that 'ud-devstack.yaml' and then wait.15:49
zulSpamapS:  https://bugs.launchpad.net/heat/+bug/119720815:49
hallynsmoser: so those things all work great using the ubuntu-lxc/daily ppa.15:49
uvirtbotLaunchpad bug 1197208 in heat "Migrate Quantum references to Neutron" [Undecided,In progress]15:49
hallyna. lxc-start-ephemeral minus the ephemral, becomes:15:49
hallynlxc-create -t ubuntu -n orig15:49
hallynlxc-clone -B overlayfs -o orig -s -n ephem115:49
hallynb. lxc clone hooks - thye should be there (in ppa), lemme check15:49
hallynyup, lxc.hook.clone is there15:50
hallynnote the saucy lxc should be merged from upstream git in (iirc) august.  stgraber is gonna strangle me soon because i can never remember the dates he has in mind15:50
smoserhallyn, k. thank you.15:51
hallynsmoser: np, shout if you need more15:53
geserjdstrand: I did some more tries: after chowning both the directory and dhcpd6.leases back to dhcpd (after starting it), the leases rotated now without error in syslog. So it looks like it's related to that bug you fixed in the past. (testing is time-consuming as dhcpd rewrites the leases file once per hour (hardcoded))16:04
SpamapSzul: is LP having issues right now?16:04
SpamapSzul: can't seem to get to that bug report :p16:05
zulSpamapS:  not that i know of16:05
zulSpamapS:  meh...that patchset seemed to got abandoned16:06
SpamapSzul: yeah will try to revive16:06
SpamapSif launchpad will talk to me. :-P16:07
smoserhallyn, because i'm that lazy...16:07
smoserlxc ppa link ?16:07
hallynsmoser: pad.lv/~ubuntu-lxc/daily16:07
hallynnot that16:07
zulSpamapS:  you have to talk to it nicely :)16:10
koolhead17hi SpamapS16:10
SpamapSzul: and s l o w l y ...16:10
SpamapS[1235993.357149] systemd-hostnamed[17620]: Warning: nss-myhostname is not installed. Changing the local hostname might make it unresolveable. Please install nss-myhostname!16:11
zulstop using fedora ;)16:12
zuljamespage:  before you go https://code.launchpad.net/~zulcss/heat/sqile-fix/+merge/17878616:26
jamespagezul, +116:26
zuljamespage:  thanks16:26
=== alamar is now known as julian
=== julian is now known as alamar
jjohansenjdstrand, geser: so yes file ownership matters. However the link rule unless you stick the owner conditional won't enforce that restriction.  Where you will see the restriction is apparmor's stacking with capabilities. Capabilities may require dac_override to access a file with different ownership16:52
jjohansenyou will see capability messages with an apparmor message, and this will require a capability dac_override, permission in the profile16:52
jdstrandjjohansen: the weird thing is dac_override wasn't logged. maybe kernel logging is getting in the way...16:53
jdstrandgeser: can you do sudo sysctl -w kernel.printk_ratelimit=016:53
SpamapSsmoser: I'm debugging a problem that has cropped up since we started building raring images....16:53
jdstrandgeser: and report back your denials?16:53
SpamapSsmoser: could you see a problem with having this: https://github.com/stackforge/diskimage-builder/blob/master/elements/cloud-init-nocloud/install.d/05-set-cloud-init-sources16:54
jdstrandjjohansen: have I mentioned how annoying kernel logging is? :P (/me knows you've mentioned it to me)16:54
SpamapSsmoser: having trouble debugging because my console is local kvm vga and I can't see what was on it easily. :-/16:55
jjohansenjdstrand, geser: possible, also note the linkat message is not an apparmor message. geser is that file on a different device? you can't hard link across devices16:55
TimRcan anybody tell me why my domain name keeps redirecting to my mail server for?16:56
jjohansenjdstrand: kernel logging is an absolute mess16:56
jdstrandjjohansen: re linkat> oh, duh-- it isn't :) I'm so used to seeing apparmor denials seeing another kernel denial was not even in my headspace :)16:57
jdstrandnote, that is not me blaming apparmor-- that is me doing far too much profiling ;)16:57
sarnoldsomething else in the kernel prints denials?16:58
jjohansenjdstrand: heh, no its an easy mistake to make, I had to double check it16:58
jjohansensarnold: a few things16:58
jdstrandsarnold: 07:11 < geser> and later "kernel: [2773234.120934] type=1702 audit(1375788247.840:13): op=linkat action=denied pid=31888 comm="dhcpd" path="/var/lib/dhcp/dhcpd6.leases" dev="dm-0" ino=275716:58
geserjjohansen: no, all files are in /var/lib/dhcp, dhcpd runs as dhcpd while the dhcpd6.leases is owned as root (I've chowned /var/lib/dhcp back to dhcpd so it can create the temporary file there again)16:59
sarnoldjdstrand: wow, cool, crazy :) never seen a 1702 before.16:59
jdstrandsarnold: yeah, me either :) maybe I can be forgiven for thinking it was apparmor then :)16:59
sarnoldjdstrand: definitely :) I hope I'd wonder why it feels so short...17:00
jdstrandI definitely thought it was a weird looking line17:00
jdstrandI even looked around for it in the apparmor docs, then got distracted by my rememberance of dac_override and the previous bug17:01
TimRanybody know the solution to my problem17:01
sarnoldTimR: sorry, you haven't described it in enough detail to even hazard guesses. can you pastebin commands that work, commands that don't work, and what you think those ocmmands ought to do differently? maybe then someone could help..17:02
geserjdstrand: I can only see a dac-override message if I reproduce the state for the old bug (#1028526)17:03
geserthe next leases should rotate in around 30 min17:03
TimROk I dont see how much more clear I can get with that issue I am having when domain name is redirecting to my mail server17:03
sarnoldTimR: maybe pastebin your host or dig output and your zone files?17:04
TimRdomain name and mail server is pointing at the same address17:05
jjohansengeser, jdstrand: its the kernel link protections17:06
geserjdstrand, jjohansen: when both the directory and the leases file is owned by dhcpd:dhcpd dhcpd doesn't start (old bug), with root:root dhcpd starts but can't rotate the leases file (current bug), when I change the owner back to dhcpd:dhcpd (both dir and file) *after* dhcpd started leases file rotating works (till restart)17:12
geserwould chowning to dhcpd:dhcpd in the upstart job and adding the dac_override cap to the apparmor profile fix both (old and current) bugs?17:13
jjohansengeser: why would you chown the file in an upstart job? That is just papering over the problem17:14
jjohansengeser: what is the bug # of the old problem? I want to make sure I am correct in my understanding of the old issue before I answer?17:15
geserjjohansen: I guess it was done to ensure the right permissions, see http://launchpadlibrarian.net/111078972/isc-dhcp_4.2.4-1ubuntu3_4.2.4-1ubuntu4.diff.gz for the fix for the old bug17:16
geserjjohansen: old bug: bug #1028526 ; current bug: bug #118666217:16
uvirtbotLaunchpad bug 1028526 in isc-dhcp "dhcpd failed to start with apparmor denied: capname="dac_override"" [High,Fix released] https://launchpad.net/bugs/102852617:16
uvirtbotLaunchpad bug 1186662 in isc-dhcp "isc-dhcp-server fails to renew lease file" [Undecided,Confirmed] https://launchpad.net/bugs/118666217:16
jjohansenjdstrand: ^ this is a problem, it runs foul of the kernels link restrictions17:17
jjohansengeser: so adding capability dac_override to the profile will fix any apparmor induced problems. However that is not the problem here17:19
jjohansenwe have a conflict betwen dhcpd privilege sep, and kernel link restrictions17:19
=== andreas__ is now known as ahasenack
geserjjohansen: what about the owner of the leases files: dhcpd recreates it after it drops priv as user dhcpd, but dhcpd can't open if for append at startup as root (it happens before dhcpd drops priv). Who should own that file? root or dhcpd?17:23
sarnoldeww. sounds like dhcpd folks didn't design their privsep correctly?17:25
geserI get slowly that impression too17:26
jjohansensarnold: I agree, they have a broken priv sep design, and the only solutions are17:29
jjohansen1. Fix dhcpd17:29
jjohansen2. turn off kernel link restrictions (which I would consider only a temporary solution)17:29
jjohansengeser: you can temporarily fix this by setting /proc/sys/fs/protected_hardlinks to 0, please not this will globally disable kernel link restrictions17:39
jjohansenunfortunately this is not something we can control on a per profile or task basis17:39
geserjjohansen: my temporary fix for now is to remember to chown the leases file and dir back to dhcpd after I restart dhcpd or the whole server (shouldn't happen too often) till it gets fixed properly17:42
jjohansengeser: yeah, that is a more localized fix17:42
=== ffio is now known as hack
=== hack is now known as nerd
jdstrandjjohansen: you pointed me at backscroll, but I'm not sure what you were pointing to. are you saying what we did for quantal was wrong?18:02
=== nerd is now known as baba
jjohansenjdstrand: I am saying that the dhcpd priv sep patch is in conflict with the kernel link restrictions18:08
jdstrandjjohansen: yes, that seems clear now. at the time (quantal), Ubuntu dropped our privsep patch that had worked18:09
jdstrandand the new one behaved differently-- and I didn't want to grant dac_override18:09
jdstrandfunny how this is only coming up now18:10
jjohansenjdstrand: its likely because of changes made to the kernel link restrictions during upstreaming vs what was in yama18:11
jjohansenanyways that is a guess18:11
jjohansenjdstrand: just a guess as to why its surfacing now instead of before18:11
jjohansenjdstrand: there where some changes but I'd have to go back to the ml to figure out what they where18:12
geserdo you both have an idea how to fix it?18:12
jdstrandam I interpreting that the current workaround is to chown root:root /var/lib/dhcp/dhcpd6.leases~ if it exists, start dhcpd, then chown dhcpd:dhcpd /var/lib/dhcp/dhcpd6.leases~ after it starts?18:12
jdstrand(that would be insane)18:12
jjohansenjdstrand: yes or disable kernel link restrictions18:13
geserjdstrand: yes, that works for me18:13
jdstrandplease file an upstream bug :P18:13
geserjdstrand: I did 'sudo chown dhcpd /var/lib/dhcp{,/dhcpd6.leases}'18:13
jjohansenjdstrand: that is because if the process has an open file handle to the file, the restrictions are applied differently18:13
jdstrandinteresting. I have not looked at the code at all, but it seems fairly obvious that if the lease files are going to be handled as the dhcpd user, oh, I don't know, open them as the dhcpd user18:15
jjohansenhrmmm, actually no this one is just doing an ownership test on startup, and then ignoring that in the future18:15
=== james_ is now known as Guest7355
jjohansenjdstrand: yeah18:15
geserjdstrand: dhcpd opens the leases file, drops priv to dhcpd and does later the leases file rotation as part of normal operation18:16
* jdstrand nods18:17
jjohansengeser: it makes the very broken assumption that it can hard link a file it doesn't own as part of the rotation18:18
geserI guess upstream assumes that dhcpd can write that file and create files in that dir18:19
geserbut this conflicts with trying to open the leases file for append as root during the startup phase18:20
jdstrandwell, it can-- but this is linking files and hardlink restrictions are now part of the linux kernel, so it needs to handle it correctly18:20
jdstrandthey either need to open as root and rotate as root, or open as dhcpd and rotate as dhcpd, aiui18:21
jdstrandagain, I've not looked at the code18:21
geserhttp://bazaar.launchpad.net/~ubuntu-branches/ubuntu/saucy/isc-dhcp/saucy/view/head:/server/db.c#L1083 new_lease_file() does the rotation18:22
jdstrandgeser: I think reporting the bug upstream and employing a short term workaround until they fix it is reasonable18:24
jdstrand(we could then cherrypick the fix)18:24
sarnoldno O_EXCL in that open(2) call? hrm.18:24
=== Ursinha is now known as Ursinha-afk
geserhttp://bazaar.launchpad.net/~ubuntu-branches/ubuntu/saucy/isc-dhcp/saucy/view/head:/server/dhcpd.c#L703 is where the leases file is read during startup while the privs get dropped later in line 77518:27
=== chuck__ is now known as zul
jkewHi folks; I'm looking for information on some of the modifications required for running ubuntu on azure; we are trying to isolate the source of some intermittent io issues and our next step is looking at the distribution provided by microsoft.18:49
jkewIt looks like Ben Howard is the goto person for this; but I have no idea what channels or methods I would use to ask technical questions on this matter.18:49
geserjdstrand: do you know if doing the chown to dhcpd call for those files in a post-startup stance in the upstart job would work? (as a ugly workaround till it can get properly fixed)19:00
=== Ursinha-afk is now known as Ursinha
rbasakjkew: Ben Howard is utlemming on here, or try the ubuntu-cloud mailing list19:07
jkewrbasak: thanks19:08
jdstrandgeser: I don't-- I only mentioned that based on your comments19:19
jdstrandgeser: it seems plausible and worth testing19:19
geserjdstrand: I tested it, but it doesn't work (unless I made a mistake with the upstart job)19:20
geserwill file the problem upstream to get it fixed properly19:21
geserstgraber: do you know if it's possible to use a post-start stance in an upstart job where the daemon get started in the foreground?19:26
mac_nibbletHmm, since when does ubuntu-server name it's network devices as nm ?19:29
mac_nibbletthis is a vanila install and i have not added network-manager19:30
stgrabergeser: I think so, though if it really starts in the foreground and doesn't send a signal back to upstart or fork at some point, I'd then expect post-start to happens immediately after the command is started, so quite possibly before the daemon is actually operational19:30
sarnoldmac_nibblet: /etc/udev/rules.d/70-persistent-net.rules is usually responsible for NIC names..19:30
mac_nibbletsarnold: but is this really expected behavior ?19:31
guntbertmac_nibblet: I've never seen that (and it has nothing to do with Network manager as someone  said)19:31
sarnoldmac_nibblet: maybe? :) what specifically are you seeing?19:32
mac_nibbletfile does not exist19:32
geserstgraber: I tried to chown some files (the isc-dhcp-server issue) as a workround after dhcpd get started but they stayed root:root19:32
mac_nibbletsarnold, guntbert: im wondering if i should try and just reinstall the server before i get more weird things19:34
mac_nibblettakes like 8 minutes to reinstall so ..19:34
stgrabergeser: so I really think the right way is to switch the ownership of the directory and figure out exactly what's going on with apparmor that prevents isc-dhcp from doing the whole create/rename/destroy thing19:35
guntbertmac_nibblet: I would not expect to see different results19:35
stgrabergeser: sarnold or jjohansen should be able to help you there19:35
jdstrandstgraber: we know what the problem is (it was discussed in backscroll)19:35
jdstrandstgraber: dhcpd is doing their priv separation wrong19:36
geserstgraber: it's a combination of AppArmor, dhcpd's priv sep and Kernel Link Protection19:36
geserduring startup the file needs to belong root and during operation dhcpd :(19:37
stgraberjdstrand: just read the backlog now, that'd indeed explain it...19:37
jdstrandit really isn't apparmor, it is the kernel link protection19:38
stgrabernote that ISC isn't terribly good at fixing bugs or giving any feedback outside of security issues19:38
stgrabergetting an upstream priv dropping code took over 2-3 years so having it changed/fixed may take just as long19:38
stgraberso we should either look into fixing that and carrying a patch until they eventually merge it or find a robust workaround (I don't want that bit to be racy in 14.04)19:39
geseromg, perhaps we should add some chown calls before dhcpd calls setuid()19:39
geserjdstrand: without AppArmor we could let dhcpd own those dir and files (AppArmor was the reason they got changed to root:root)19:41
jdstrandI hope you aren't suggesting dropping apparmor :)19:42
jdstrandit needs to either open and rotate the files as dhcpd, or open and rotate the files as root19:43
jdstrandopening as root and rotating as dhcpd is the problem19:43
geserjdstrand: certainly not suggesting it19:43
geserjdstrand: what about changing the owner of those files before calling the setuid()/setgid()?19:44
jdstrandthey are probably opening as root and handing off the fd as a security protection19:44
jdstrandgeser: that should work-- sarnold ^19:45
stgraberIn an ideal world dhcpd should own /var/lib/dhcpd and not open any fd until after it's done dropping privileges19:45
sarnoldyeah, I like stgraber's ideal world :)19:46
stgraberthe main/only reason why it even needs to run as root is to open a raw network socket, so it really should do that and then drop privs like any proper daemon should19:46
jdstrandthat's what I was suggesting with opening as dhcpd19:46
stgraberand I think that's the patch we should apply to Ubuntu, get into Debian and forward to ISC so maybe one day it'll be done properly upstream19:47
=== Ursinha is now known as Ursinha-afk
stgrabergeser: if you want to have a try at doing this, feel free, if not, please comment in the bug report (I think you mentioned one earlier right?) and assign to me so I have it on my todo19:48
geserjdstrand: dhcpd has an option to check the validity of the leases file and it does it before it does its deamonizing19:48
stgraber(I'm on vacation until Thursday and at Debconf next week but I may find some quiet time to do that anyway or will look at it once I'm back home on the 20th)19:49
geserstgraber: will add my findings to the bug tomorrow and see if I can code a workaround19:49
sarnoldstgraber: oh nice, enjoy your vacation and debconf :)19:50
jdstrandseems like that check should definitely be done as non-root19:51
geserI agree, so dhcpd can complain if it has access issues to the leases file when run as non-root19:52
=== james_ is now known as Guest20713
=== Ursinha-afk is now known as Ursinha
SpamapSsmoser: need your opinion about a problem we're seeing...21:44
SpamapSsmoser: so the problem from earlier is that we don't have a serial console defined on some of the vms we boot..21:44
SpamapSsmoser: when that happens, anything that uses 'console output' fails because /dev/console is inoperable21:44
SpamapSsmoser: it is inoperable because cloud images specify console=tty1 console=ttyS0...21:45
SpamapSsmoser: is it reasonable to expect the cloud images to boot w/o a serial device? (I think.. yes)21:45
utlemmingSpamapS: this seems more like a bug for Ubuntu21:47
utlemmingbut I would agree that that shouldn't trigger a failure21:47
uvirtbotLaunchpad bug 1123220 in cloud-initramfs-tools "cloud-image VM causes kernel panic if image is resized" [Low,Triaged]21:48
smoserthats the bug21:48
smoserand you can read the email thread on it.21:48
smoserSpamapS, ^ you see that ?21:53
smoserthe simplist solution is "well attach a serial device for petes sake!"21:54
* smoser has to go to bed, but there is a very complete email thread attached to that bug.21:55
smoserit is very much less than trivial to accomplish what we want.21:55
smoserand i'm open to any ideas.21:55
SpamapSsmoser: in the past we had a different kernel bug that required us to not have a serial device.. ;)22:02
SpamapSsmoser: thanks for the bug link. That is in fact the bug I was looking for.22:11
hadifarnoudI followed this guide http://pleasefeedthegeek.wordpress.com/2012/04/21/l2tp-ubuntu-server-setup-for-ios-clients/ for L2TP vpn. it connects but I cannot access any website. I think step 3 is wrong. can anyone help?23:50

Generated by irclog2html.py 2.7 by Marius Gedminas - find it at mg.pov.lt!