=== skeezix-hf is now known as Ofir === Into_the_Pit is now known as Frickelpit === demon_spork is now known as demonspork [06:54] rbasak: thanks, upload worked [07:33] I need help. My release upgrade went a bit wrong. Now when I do apt-get -f install it wants to install a ppp and upgrade the x11-common packages. However while "Preparing to unpack .../x11-common-1%3a7.7+13ubuntu3_all.deb ..." it seems to want to stop/start the x11-common via /etc/init.d/x11-common start/stop which does not work. [07:35] I can't remove it either; [07:35] dpkg: error processing package x11-common (--remove): [07:35] package is in a very bad inconsistent state; you should [07:35] reinstall it before attempting a removal [07:36] why doesn't the x11-common start / stop work? [07:36] no idea [07:38] wild guess, maybe apt-get install --reinstall x11-common ? [07:40] panicstr: what is the exact output? can you put it to pastebin? [07:41] http://pastebin.com/Tz9UT6Yg [07:41] preparing to unpack, and then? [07:41] You missed the error there [07:42] it's stuck trying to do the /etc/init.d/x11-common stop, which doesn't work [07:42] And it's not possible to put that to pastebin? [07:43] WARNING: The following packages cannot be authenticated! --- that's odd [07:43] why not? [07:43] It's possible that your apt sources are not up to date [07:44] Can you put the *whole* output of `apt-get update; apt-get --yes dist-upgrade; apt-get install -f` to pastebin? [07:44] no problem, hold on. [07:46] first, this is what it returns after i manually kill the x11-common stop request [07:46] http://pastebin.com/4Ek8D1sP [07:48] apt-get update: http://pastebin.com/3TH4pTaF [07:48] woaaaah [07:49] apt-get --yes dist-upgrade: http://pastebin.com/pWVM5Ewg [07:49] okay two ideas come to mind: take a look at dmesg output, maybe there's some hardware or filesystem errors -- or check df and df -u output, maybe your filesystems are full [07:50] apt-get install -f: http://pastebin.com/ri4qN5EN [07:50] and it's now trying to stop the x11-common again... [07:54] panicstr: if even apt-get update has issues, something's more wrong than just x11-common [07:54] Try with the main server instead [07:54] I still think he's got a trashed filesystem [07:55] what does this mean [07:55] [77193.869996] cron[11310]: segfault at 504 ip 00481bcf sp bfbd4d10 error 4 in libpthread-2.23.so[47d000+19000] [07:55] drive errors or ram errors [07:56] it could also be a bug in the software [07:56] You can test with a live cd [07:56] If you see segfaults there, suspect hardware issues [07:56] If you don't, suspect corrupted installation [07:56] but with the symptoms here it sure feels like hardware .. [07:56] There's also `debsums -s` [07:56] memtest86 also a good idea [07:56] ...but you might not be able to install it, hm... [07:56] Yeah memtest too, from the boot manager [07:58] I did a memtest couple days ago, there were no errors. HDs seem ok too, did plenty of relocation of big vm files recently... [07:58] a corrupted file system doesn't necessarily imply hardware errors [07:59] Boot from a live cd first [07:59] You'll be able to fix things from there using "chroot", if the hardware is ok [07:59] Well that's a bit of a problem as this server is about 200km away [08:00] does the ipmi interface perhaps allow mounting a local iso as a boot media? it might be the worlds slowest boot.. [08:00] or maybe pxe boot off a machine in the same rack? [08:01] this one wouldn't even be fun to debug if the machine were in the same room... 200km away, ugh. [08:02] It's possible to write an .iso to the local disk and boot from it using the existing grub, but it's kinda hard to instruct someone to do it over irc [08:02] and on compromised or potentially compromised filesystem, even less fun [08:06] Let's just assume it's not a hardware problem for now. [08:09] alright, in that case the debsums idea is pretty good; apt-get download debsums, use 'ar x' to extract the tarball, 'tar x' to unpack the tarball, and try running debsums that way [08:09] most packages include md5sums of their files, and debsums can report mismatches [08:12] dpkg -i might also work, even if apt fails [08:12] I would start by trying the main server though, instead of the si one [08:13] I've seen such issues with the .gr servers some times, and also recently with the change to the hashed repositories [08:13] if dpkg -i works it'd certainly be easier :) [08:33] can't use apt because of the ppp package not installed, dpkg -i however works ok [08:33] how do i change .si to main? [08:33] I bet apt-get download works fine [08:33] right [08:37] panicstr: you either run software-properties-gtk, if you have gui, or manually edit /etc/apt/sources.list [08:39] I have some weird issue, when I provision a server the install goes well, after the install it keeps rebooting when it wants to boot from disk === iberezovskiy|off is now known as iberezovskiy [08:45] please have a look at this http://pastebin.com/pCAfi0JV [08:45] How could i get this x11 in order by hand first [08:46] Couldn't create tempfiles for splitting up /var/lib/apt/lists/partial/main.archive.ubuntu.com_ubuntu_dists_xenial-updates_InRelease [08:46] If it can't create temp files to do its tasks... it's a serious issue [08:46] You need to be able to access the file system properly before doing package management jobs [08:46] Now, you have file system issues [08:50] panicstr: boot from a live cd before you start losing existing data... [08:56] panicstr: Is the ram full? [09:02] i don't think so [09:02] panicstr: Can you check? [09:03] How do you check that? [09:04] free -m ? [09:06] panicstr: For example. [09:06] it's not full [09:06] 4258 available [09:07] panicstr: Can you create files using 'touch'? [09:09] i can [09:10] What's the output of `df -h` ? [09:11] Also in /var/lib/apt/lists/partial/? [09:12] lordievader yes [09:12] alkisg http://pastebin.com/cCiVpndU [09:14] No space issue, then [09:14] panicstr: What is the output of 'sudo apt-key finger'? [09:15] http://pastebin.com/D79C7W5T [09:16] Hmm, so apt-key does work... [09:17] I take it that gnupg is installed too? [09:19] How do you check that [09:19] dpkg -l|grep gnupg [09:20] http://pastebin.com/f9QdE6Ss [09:21] is this ok? [09:21] Yes, it makes the output of the 'apt-get update' just a bit more strange. [09:22] What happens if you make it 'sudo apt-get update'? [09:23] https://paste.ee/p/7rH0S === chmurifree is now known as chmuri [09:25] panicstr: What are the permissions for /tmp? [09:26] drwxr-xr-x 50 root root 8192 Feb 1 10:25 tmp [09:27] panicstr: As I figured ;), 'sudo chmod 0777 /tmp && sudo apt-get upgrade' [09:29] cool, that solved the Couldn't create tempfiles problem [09:29] however, there's another problem we discussed earlier; [09:30] Which is? [09:30] Why had /tmp 755 anyways? [09:31] don't know, could be i changed it although i don't remember [09:31] have a look at this [09:31] apt-get update: https://paste.ee/p/vk4vi [09:32] apt-get upgrade: https://paste.ee/p/6JMfj [09:32] Sounds like gnome... [09:32] Never seen that error, to be honest. [09:32] apt-get -f install: https://paste.ee/p/xSh0d [09:33] ps aux: https://paste.ee/p/8TTW1 [09:34] root@vmhost:~# /etc/init.d/x11-common start/stop/status does nothing [09:35] i can't remove x11-common either [09:36] https://paste.ee/p/Be2Gl [09:36] what if i just rm the file from /etc/init.d/ ? [09:37] Is that a service??? [09:37] panicstr: What happens when you let apt purge it for you? [09:38] Unmet dependencies [09:38] https://paste.ee/p/uG9fu [09:41] Did the paste of 'apt-get install -f' continue after unpacking? [09:41] no, it just tries to stop the x11-common [09:41] which does nothing [09:42] doesn't even return the prompt [09:42] Let it run for a bit. [09:43] How long should i leave it? [09:44] Sometime... guess if the promt doesn't return after, say, 15 minutes we can establish it is not doing much. [09:59] nothing so far... [10:03] panicstr: you can put an "exit 0" at the beginning of the x11-common init script, if you want to bypass it during installation [10:03] panicstr: how are you connecting to the server, via ssh from a linux box? [10:04] via putty from w7 [10:05] sudo nano /etc/init.d/x11-common, and put "exit 0" before "set -e" [10:05] Remember where you put it, so that you remove it after apt finishes [10:06] He wants to remove x11-common, that should take that script with it ;) [10:06] No, he wants to remove it only because he can't properly upgrade it [10:06] So if he puts "exit 0" there and ugprade finishes, he will no longer want to remove it, I imagine [10:07] Ah, that I didn't know. [10:07] Although I'm not sure if that's the only issue he's still facing ... I wonder if he just copied the whole installation with `cp` and the permissions are bad [10:08] https://paste.ee/p/mMBkT [10:09] That's the main error: Error: GDBus.Error:org.freedesktop.DBus.Error.Spawn.ExecFailed: Cannot launch daemon, file not found or permissions invalid [10:09] panicstr: I think this will take hours that way, do you want help via teamviewer? [10:10] I'd force purge that package at this point... [10:10] The problem is that dbus isn't running [10:10] maybe due to permission issues? [10:11] The package installation should succeed once the other system issues are fixed [10:11] Or because x11-common is installed it tries to do X stuff... [10:13] teamviewer 332 582 407 / 4633 [10:13] Eh wait I need to update teamviewer 11 to 12 === disposable3 is now known as disposable2 [10:35] ...it turns out the dbus package is not configured properly, and thus breaks a whole lot of postinsts [10:41] Ouch [11:27] Hello all [11:28] What do you think about mounting a clustering system in cloud environement ? [11:30] I want to mount a galera cluster system with 3 VM hosted at a remote hoster [11:31] I don't how stable it could be, and what do I need to cope with clustering requirements ? === JanC_ is now known as JanC [14:30] nacc: when you have time (no rush), could you review https://code.launchpad.net/~racb/usd-importer/+git/usd-importer/+merge/315120 please? I want to slowly filter my patchsets into master, to eventually get queue support landed in there. [14:31] jamespage, beisner: hi when you get a chance can you promote neutron 2:9.1.1-0ubuntu1~cloud0 from neutron-staging -> neutron-proposed and python-oslo.messaging 4.6.1-2ubuntu2~cloud0 from mitaka-staging -> mitaka-proposed? [14:32] s/neutron-/newton-/ [14:33] jamespage, beisner: actually might as well just promote everything in mitaka-staging -> proposed [14:33] coreycb, on it [14:39] smoser: is the cloud-init SRU exception documented anywhere? [14:41] rbasak, it does not exist. [14:41] we do intend to get one.. [14:41] powersj is looking at doing that, first for curtin, and then for cloud-init [14:46] Who handled the SRUs in the past? [14:46] Have full upstream releases been backported before? [14:46] probably [14:46] its not really a "full upstream release" [14:46] if you look at the diff, its fairly small other than the license change. [14:47] i do agree that we need to get the exception into place [14:47] and better integration test also [14:47] as we intend to keep pulling back new things [14:51] smoser, we should try to knock out the SRU documentation for both curtin/cloud-init next week's sprint [14:51] smoser: powersj has started it for curtin here -> https://wiki.ubuntu.com/CurtinUpdates [14:52] but really shouldn't take long to knock both of those out if ya'll feed him the important bits to emphasize [14:54] rbasak, http://paste.ubuntu.com/23905579/ [14:54] filters that pretty well [15:06] smoser: what are you expecting to do for SRU verification on this? [15:08] smoser: you don't currently have any sort of TB exception for cloud-init, correct? [15:10] rbasak, well, i'll verify each of the bugs as the bug sru template describes. for entries in the changelog that do not have a bug, i'd not do anything. [15:10] all bug changes other than 1647910 1582323 1655934 1379080 [15:10] have been previously successfully SRU'd to yakkety [15:13] beisner, thanks [15:14] rbasak, changelog entries are taken from upstream git commits. you're welcome to walk through those for yourself, but the entries i consider even remotely interesting for an sru that have not already been sru'd to yakkety are [15:14] http://paste.ubuntu.com/23905655/ [15:15] there are admittedly three changes there that do not have bugs with them. [15:15] (i'm ignorning cloudinit/config/cc_rh_subscription.py for ubuntu) [15:16] rbasak, i agree on the need for changes to get exception in place. [15:16] how can i help you? [15:26] smoser: I'm still trying to figure out how to approach this. === admcleod_ is now known as admcleod [15:30] rbasak, thats fair. [15:30] i think if you look at it from commit by commit on the ubuntu/xenial branch, it doens't look that bad. [15:30] smoser: I think I need to review against https://wiki.ubuntu.com/StableReleaseUpdates#When and https://wiki.ubuntu.com/StableReleaseUpdates#New_upstream_microreleases, so need to identify which bits of diff correspond to which criteria. [15:30] git log ubuntu/0.7.8-49-g9e904bb-0ubuntu1_16.04.4..ubuntu/0.7.9-0ubuntu1.16.04.1 [15:31] smoser: have you pushed your xenial branch? In https://code.launchpad.net/~cloud-init-dev/cloud-init/+git/cloud-init/+ref/ubuntu/xenial I see only up to 16.04.1. [15:31] and realize that dropping the cpicks in debian/patches represent some of those commits. [15:31] i'll push [15:31] wait... it should be there. [15:32] thats right [15:32] that has everything [15:32] what did you miss ? [15:32] smoser: https://git.launchpad.net/cloud-init/log/?h=ubuntu/xenial [15:32] 0.7.9-0ubuntu1~16.04.1 [15:32] What about 2, 3 and 4? [15:32] Oh, sorry. [15:32] My mistake. It's all there. [15:33] Upstream version bump :) [15:36] hey folks so given the recent gitlab issues with their outage it got me thinking at work on how to force a bash prompt on prod servers... is there a way to do this and override any users bash prompt no matter what [15:45] i'm having problems with Predictable Network Interface Names in trusty [15:45] virtio drivers are named eth0 [15:45] but should be named ens3 for instance [15:45] is ubuntu affected by something similar? [15:45] https://bugzilla.redhat.com/show_bug.cgi?id=1259015 [15:45] bugzilla.redhat.com bug 1259015 in systemd "persistent interface names for virtio interfaces" [Urgent,Closed: errata] [15:46] on physical machines I get predictable names like em1 [15:46] but not on my virtual machines (virtio) [15:47] i'm having a hard time handling this fact and are starting to break down in tears [15:49] Predictable? isnt eth0 predictable? [15:49] Raboo: kvm renumbers pci buses when adding stuff, like another disk. iirc. [15:54] I cant use virtio nic drivers because of the problems. I use e1000, and see em0 or igb0 or eth0, depending on the distro [15:58] maswan compdoc in xenial i can decide if I want eth0 or ens3 [15:58] but in trusty whatever I do the interface is called eth0 when using virtio [15:59] compdoc it isn't predictable when I discover a interface name with the name ens3 and I install OS and it is called eth0 [15:59] and I want that feature because of the physical machines. [15:59] and turing it off leads to have all machines name their interfaces eth* [16:00] but when having multiple NICs it's useful, leads to less guesswork [16:01] Raboo: Have you tried setting in kernel load line: biosdevname=1 net.ifnames=1 [16:02] that disables naming and goes back to eth0, etc [16:04] genii i have tried net.ifnames=1 [16:05] genii biosdevname should default 1 if package biosdevname is installed. [16:06] genii those stuff works for interfaces that aren't virtio [16:06] but not for virtio interfaces. [16:08] oops, biosdevname=0 turns off Consistent Network Device Naming. nm [16:32] yea, net.ifnames=0 and biosdevname=0 to disable, although the installation kernel may not be booted with those params, so ymmv.. you can make the install kernel revert to old school ethN though [16:33] and it will work for virtio devices, they're no different to standard nics in the naming sense [16:35] joelio i want to enable the new naming policy, except it doesn't work in trusty for virtio drives, do not want to disable it. [16:35] it works for other drivers like intel. [16:35] how do I ensure a service doesn't get started automatically? [16:35] or check that it is set to do so? [16:35] DammitJim depens on what init system you use [16:36] systemv, systemd, upstart? [16:36] Ubuntu 14.04 defauot [16:36] 14.04 got a mix of systemv and upstart [16:36] DammitJim you can use the update-rc.d tool [16:36] it's defined /etc/init.d [16:36] yeah, update-rc.d probably [16:36] thanks [16:36] it's tomcat [16:37] and whenever I update it, it resets that setting for some reason [16:37] Raboo: Trusty only had biosdevname on it's kernel, not net.ifnames [16:38] they're different beasts, just to make matters more interesting :) [16:38] also, it's linked to kernel - so perhaps you need to get the linux-lts-generic-xenial kernel on [16:39] if you have already created interfaces too, then they'll be locked in udev [16:39] and it won't rename them unless you decouple that [16:40] ooi, any reason you're not using Xenial (16.04)? [16:41] Raboo: do you mean interfaces created on the host or in guests too? virto wise? [16:42] joelio ok so that is why physical machines get em1/em2 and virtual get eth0 in trusty [16:42] joelio guests [16:43] because physical != virtual.. the naming is based upon bus [16:43] well joelio i'm testing both xenial and trusty and get different results. [16:43] the reason i wanted ens3 instead of eth0 is that on xenial it was named ens3 [16:44] yea, as xenial comes with net.ifnames [16:44] em == biosdevname [16:44] I wanted it consistent so i didn't have to build exceptions in the install script. [16:44] yea, unfortunately you'll always get these edge cases [16:44] but i did a exception now, trying the install now. [16:45] https://major.io/2015/08/21/understanding-systemds-predictable-network-device-names/ [16:46] ^^ that's systemd (net.ifnames) cdn [16:46] https://access.redhat.com/documentation/en-US/Red_Hat_Enterprise_Linux/7/html/Networking_Guide/sec-Consistent_Network_Device_Naming_Using_biosdevname.html [16:46] is biosdevname [16:46] just to make things more fun, let's make 2 standards of naming scheme [16:46] well, 3 if you include the original [16:47] ok, is there a way to enable net.ifnames on the original trusty kernel? [16:47] some relies on systemd [16:47] you could try on an lts xenial kernel [16:47] but ymmv [16:48] true, we do run xenial kernel on some nodes, but that is just cause the application would benefit from the newer kernel [16:49] you can also rename devices in udev, if you wanted to patch it in a hacky way ;) [16:49] but still edge case considerations, have to do some work to get consistent [16:49] whether it's managing kernels, grub lines or udev etc [16:49] sucks, but afiak no simple way around it [16:49] i ended up using eth0 names in virtual nodes [16:50] yea, path of least resistance :D [16:50] was only two lines [16:50] I disabled it on all our rollouts for a while [16:50] # check if virtio [16:50] [[ -e /sys/class/net/${i}/device ]] && ls -l /sys/class/net/${i}/device | grep -q virtio [16:50] # also if it is systemd, then disable predictable network interface names. [16:50] but the realised was probably like Canute, trying to hold back the ever impending waves of 'progress' :) [16:50] [[ $? -eq 0 ]] && [[ -d /etc/systemd/network ]] && ln -s /dev/null /etc/systemd/network/99-default.link [16:51] so this way our physical machines gets em1 or p1p2 [16:51] and virtual eth0 [16:51] these stuff feels a bit retarded.. [16:52] @everyone I got a ticket to cfgmgmtcamp.eu [16:52] tomorrow I will speak with the boss about sponsoring my flight and hotel if possible. [16:52] Raboo: nice [16:52] where should one stay? [16:52] * joelio may be going to kubecon later this year too fwiw (maybe see some of you?) [16:53] joelio you know the rule, when you sit in seminars you have to be hung over. [16:54] Raboo: always :D [17:16] smoser: are you planning on only pulling new upstream releases from here on in, rather than cherry-picking? Or a combination of both? [17:16] probably a combination based on urgency of fixes and such. [17:17] OK [17:22] rbasak: ack [17:22] rbasak: i'll probably push a commit on top that adds messages, as per smoser's review [17:25] nacc: sure, thanks. [17:48] zul, hey just starting some b3 testing here [17:48] zul, looking at neutron-openvswitch-agent not starting [17:48] coreycb: ryu [17:48] zul, ah. ok [17:49] coreycb: fixed is in zesty-proposed [17:50] zul, alright i'll backport that to the uca [17:50] coreycb: python-tinyrpc needs to backport if it isnt already backported [17:50] zul, ok [18:04] <_MoBeats_> Afternoon. I'd like to know what are the hardware requirements for MAAS and Autopilot servers. Had a good look on ubuntu.com but can't see the info anywhere. Can someone point me in the right direction please? [18:08] rbasak: done, fyi [18:12] Thanks! [18:40] coreycb: MIR filed #1661060 (python-tinyrpc) #1660163 (python-os-xenapi) [18:40] zul, ok thanks [19:02] coreycb: got magnum [19:03] zul, sorry i started on that and ironic. hold up if you haven't started. [19:03] coreycb: oh...sorry...4.0.0 for magnum right? [19:04] zul, yeah [19:05] coreycb: ok cool [19:13] coreycb: ill take a look at aodh [19:13] zul, ok. did they release anything yet for ocata? [19:13] coreycb: of course not [19:14] zul, ok. i guess we could do another git snapshot. [19:14] coreycb: yeah...the packaging is in a bad shape atm [19:14] zul, oh? [19:15] zul, the last snapshot we released should be ok [19:16] coreycb: this is the commit that broke things https://github.com/openstack/aodh/commit/7b0435a706095bfb6c5bac28bd25d1bdd91e1fe1 [19:16] coreycb: i was looking at it this morning [19:18] zul, do you have to run aodh-config-generator now? [19:20] zul, oh that may be a part of moving paste defaults to the code base [19:20] coreycb: yeah it is [19:21] coreycb: right now its ftbfs because it cant find stuff in /usr/etc ? [19:21] zul, we'll have to do something similar to what tox.ini does to run aodh-config-generator and generate aodh.conf [19:22] coreycb: yeah i got that already :) [19:22] zul, ok === nacc_ is now known as nacc === Amgine_ is now known as Amgine [22:34] anyone know, given a /dev/ path to a disk, how to verify/determine it's connected via iscsi? `lsscsi` does not seem to indicate the transport correclty (at least in xenial) === JanC is now known as Guest9668 === JanC_ is now known as JanC [22:37] I think iscsiadm has something for this [22:47] genii: yeah, the problem is that iscsiadm doesn't know (afaict) about the local disks (so you can't ask iscsiadm what, if any, iscsi disk /dev/sdb corresponds to) [23:01] jamespage, I thought I'd do a merge for python-boto if you have no objections (checking with previous you as last touched, or any openstack concerns). I figure I owe you more than a few from last cycle yet. :)