| manukapua | cool that seemed to work ok cheers sarnold for the assistance | 00:09 |
|---|---|---|
| sarnold | manukapua: don't forget to put back the -meta kernel package if you removed it earlier, to keep getting kernel updates | 00:09 |
| manukapua | its still there : ) | 00:10 |
| lxle | Anyone know if when 'unattended-upgrades' is set to download the packagelist, does it download an entire packagelist or just the 'allowed' repository parameters you have set, like 'security' | 00:22 |
| === Tsunami is now known as Guest18089 | ||
| lordievader | Good morning | 06:20 |
| zioproto | coreycb, rbasak at SWITCH testing our new Xenial/Newton setup we figured out that we are hitting this bug https://bugs.launchpad.net/ubuntu/+source/qemu/+bug/1647389 This must be fixed before we can move on from Trusty/Mitaka. We need to bug fixed to be able to live-migrate VMs safely, because we evacuate the nova-compute nodes to reprovision them with | 11:50 |
| zioproto | Xenial/Newton. Do you know about this bug ? | 11:50 |
| ubottu | Launchpad bug 1647389 in qemu (Ubuntu Xenial) "Regression: Live migrations can still crash after CVE-2016-5403 fix" [High,Confirmed] | 11:50 |
| rbasak | I don't know about it, sorry. | 11:52 |
| rbasak | I'd ask cpaezler but he's out | 11:52 |
| rbasak | jgrimm: ^ | 11:52 |
| zioproto | rbasak: but you commented on the bug :O ! | 11:53 |
| zioproto | are you Robie Basak (racb) ? | 11:54 |
| rbasak | zioproto: not with any knowledge, just referring more relevant people. | 11:54 |
| rbasak | Yes. | 11:54 |
| zioproto | ah ok :) | 11:54 |
| zioproto | I all come back here when is daytime in the US :) | 11:54 |
| coreycb | zioproto, did you see the workaround mentioned? | 11:57 |
| coreycb | zioproto, i'm just scrolling through the bug coming up to speed | 11:57 |
| zioproto | coreycb: if you mean 'virsh dommemstat --live --period 0 <VM instance name>' we tried this and did not work | 11:59 |
| coreycb | zioproto, it looks like turning live_migration_tunnelled off is a work around too | 12:00 |
| zioproto | coreycb: ok, I found something else to check. We had the default value for live_migration_tunnelled | 12:03 |
| mdeslaur | zioproto: FYI, there are _untested_ pre-release packages to work around that bug here: https://launchpad.net/~ubuntu-security-proposed/+archive/ubuntu/ppa/+packages | 12:04 |
| zioproto | now it looks like Mitaka has live_migration_tunnelled default to True and Newton has live_migration_tunnelled default to False | 12:04 |
| coreycb | thanks mdeslaur | 12:04 |
| mdeslaur | they haven't had any sort of QA yet, but you're welcome to try them out in a test environment | 12:05 |
| zioproto | mdeslaur: thanks ! I will try them ! | 12:05 |
| zioproto | mdeslaur: the source server is Trusty, the destination server is Xenial. It is enough to install the qemu package on the Xenial server to test the live migration fix ? | 12:09 |
| mdeslaur | zioproto: I think so, but I'm not sure, sorry. | 12:10 |
| cpaelzer | rbasak: here chiluk is the expert on this bug | 12:11 |
| * cpaelzer is not here | 12:11 | |
| cpaelzer | and I see mdeslaur is also here, perfect | 12:13 |
| cpaelzer | so I can be away then | 12:13 |
| * cpaelzer silences his speaker to not get back to the office on a highlight | 12:13 | |
| zioproto | I have a question on live_migration_tunnelled in nova.conf | 12:16 |
| zioproto | so this is an important setting to be explicit in nova.conf on the controller or on the compute nodes ? | 12:16 |
| zioproto | mdeslaur: so we installed qemu-system-x86_2.5+dfsg-5ubuntu10.11_amd64.deb and qemu-block-extra_2.5+dfsg-5ubuntu10.11_amd64.deb only. These packages we installed them only on the Xenial target compute-node. We did not touch at all the source Trusty compute node. I can confirm this packages fixed the bug #1647389 for me | 12:30 |
| ubottu | bug 1647389 in qemu (Ubuntu Xenial) "Regression: Live migrations can still crash after CVE-2016-5403 fix" [High,Confirmed] https://launchpad.net/bugs/1647389 | 12:30 |
| mdeslaur | zioproto: oh, great, thanks for testing it | 12:31 |
| zioproto | my team here is thanking you, because these package repo is not linked on the bug report | 12:31 |
| zioproto | I guess if the link was there more people had tested the packages, because 12 people are watching the bug | 12:32 |
| mdeslaur | zioproto: they will be released as security updates once they have been through QA...probably in a couple of weeks | 12:32 |
| mdeslaur | I'll add a comment to the bug | 12:32 |
| zioproto | thank you | 12:32 |
| nacc | cpaelzer: re: LP: #1673714, what is dquilt? | 15:10 |
| ubottu | Launchpad bug 1673714 in usd-importer "provide a subcommand to convert a git sha to ubuntu patch" [Undecided,New] https://launchpad.net/bugs/1673714 | 15:10 |
| === JanC is now known as Guest75772 | ||
| === JanC_ is now known as JanC | ||
| === Guest18089 is now known as PacketSCan | ||
| roaksoax | /w/win 8 | 16:00 |
| === Guest96888 is now known as IdleOne | ||
| === nchambers is now known as Guest99703 | ||
| === Guest99703 is now known as dtscode | ||
| erick3k | quick question | 16:40 |
| erick3k | the command route add is permanent | 16:40 |
| nacc | erick3k: that's not a question, but no | 16:44 |
| nacc | erick3k: if you reboot, the routes will be gone | 16:45 |
| erick3k | hehe | 16:53 |
| erick3k | nacc whats the best way to add them permanent that don't involve puting them in /etc/network/interfaces? | 16:55 |
| erick3k | and apart from puting them in /etc/rc.local | 16:55 |
| erick3k | or rc.local best way? | 16:55 |
| nacc | erick3k: any reason for not using /e/n/i ? | 16:56 |
| erick3k | Yes, cloud-init will remove the post up / post down commands | 16:57 |
| erick3k | in interfaces | 16:57 |
| erick3k | so it has to be somewhere else | 16:57 |
| nacc | erick3k: that seems like soething to ask cloud-init about, i assume there is either a way to not do that or some config option | 16:58 |
| erick3k | yes am sure cloud-init has an option, but my current system won't allow me | 16:59 |
| erick3k | so forget about cloud-init | 16:59 |
| erick3k | back to first question hehe | 16:59 |
| nacc | erick3k: will c-i remove /e/n/i.d entries too? | 16:59 |
| erick3k | well am wrong, and you are kind of right | 17:00 |
| erick3k | thats where ci puts the config | 17:01 |
| erick3k | so i guess it doesn't touch interfaces | 17:01 |
| nacc | there you go :) | 17:01 |
| erick3k | thats the best place anyway to put them permanent routes? | 17:01 |
| nacc | erick3k: aiui, but i'm not necessarily an expert on it :) | 17:01 |
| erick3k | kool ty | 17:02 |
| nacc | erick3k: np | 17:02 |
| erick3k | ummm | 17:13 |
| erick3k | any ideas what can be wrong? looks good to me https://i.imgur.com/dyJhjhD.png | 17:13 |
| nacc | erick3k: do you have the `route` command? or what is happenig? | 17:15 |
| erick3k | not sure, i am following this guide http://docs.ovh.ca/en/guides-network-bridging.html | 17:15 |
| erick3k | but yea route command is there | 17:16 |
| erick3k | i can use it manually to add the routes and works | 17:16 |
| nacc | erick3k: oh sorry misread the bottom f it | 17:16 |
| nacc | erick3k: which line is 14? | 17:16 |
| erick3k | first post-up command | 17:17 |
| nacc | erick3k: oh of course | 17:18 |
| nacc | erick3k: it's an iface option | 17:19 |
| nacc | erick3k: what are you post-up'ing in that file? :) | 17:19 |
| erick3k | umm the routes | 17:19 |
| nacc | erick3k: as in, post "what"'s up state woudl that command run? | 17:19 |
| nacc | erick3k: routes aren't up | 17:19 |
| erick3k | it should run route add when the interface goes up | 17:19 |
| nacc | erick3k: right, *which* interface | 17:19 |
| nacc | erick3k: syntactically your file makes no sense | 17:19 |
| erick3k | is on /.d | 17:19 |
| nacc | erick3k: post-up only makes sense in the context of a interface definition | 17:20 |
| erick3k | ok so can't put them on .d cuz is gonna be modified by ci | 17:20 |
| erick3k | so am stuck to rc.local then? | 17:20 |
| nacc | erick3k: so you want to add a route to a iface that c-i configures? | 17:22 |
| erick3k | correct | 17:22 |
| erick3k | ci configures on /e/n/i/.d | 17:22 |
| erick3k | without having to tell ci to do it because i can't | 17:23 |
| nacc | i don't understand fully why you can't but ok | 17:24 |
| nacc | erick3k: you don't control your cloud-config? | 17:24 |
| nacc | erick3k: it seems like you can define routes vai systemd.network as well | 17:24 |
| erick3k | yes and no, is limited. That would be the obvious solution but perhaps why am here looking for another solution | 17:25 |
| erick3k | rc.local is gonna work, ive used before but i don't think is the best way | 17:25 |
| erick3k | nacc i would have to somehow modify rhev completly, templates etc to be able to add routes on cloud-init launch | 17:26 |
| erick3k | that is beyond me | 17:26 |
| erick3k | thats why | 17:26 |
| nacc | erick3k: huh? | 17:26 |
| erick3k | rhev / orvirt | 17:26 |
| nacc | erick3k: you aren't able to pass cloud-config to your instances? | 17:26 |
| erick3k | yes | 17:26 |
| erick3k | but limited config | 17:26 |
| erick3k | so ovirt which is the system i use | 17:27 |
| nacc | erick3k: hrm, dunno | 17:27 |
| erick3k | doesn't have an option to pass routes throught cloud-init | 17:27 |
| nacc | erick3k: but ok | 17:27 |
| erick3k | so i simple can't use cloud init to pass routes | 17:27 |
| erick3k | very simple | 17:27 |
| erick3k | so i want permanent routes on the O.S | 17:28 |
| erick3k | that has nothing to do with cloud-init | 17:28 |
| nacc | erick3k: but you can write arbitrary files? i'm super confused :) | 17:29 |
| erick3k | nacc sorry got dced | 17:34 |
| erick3k | but nvm i'll just stick to adding routes with rc.local | 17:35 |
| nacc | erick3k: yeah, i think that's easiest for wht you've described | 17:35 |
| teward | has anyone had an issue where an upgrade on Xenial of mysql-server-5.7 would totally lag out and just consume all the memory and swap? | 18:27 |
| teward | because that happened on my mail server :/ | 18:27 |
| erick3k | hi | 18:44 |
| erick3k | i see in ubuntu 16.04 /etc/init/rc-sysinit.conf was removed | 18:44 |
| erick3k | what is it now? | 18:44 |
| sarnold | I can't figure out what that thing's supposed to do.. | 18:46 |
| erick3k | what thing | 18:47 |
| erick3k | ? | 18:47 |
| sarnold | erick3k: systemd aims to run multi-user.target, and runs whatever is supposed to be run to make that happen | 18:47 |
| sarnold | erick3k: if you want to change the systemd target, this might help https://access.redhat.com/documentation/en-US/Red_Hat_Enterprise_Linux/7/html/System_Administrators_Guide/sect-Managing_Services_with_systemd-Targets.html#sect-Managing_Services_with_systemd-Targets-Change_Default | 18:47 |
| erick3k | ummm | 18:48 |
| erick3k | i just wanted to change start on (filesystem and static-network-up) or failsafe-boot | 18:48 |
| erick3k | but is no longer in ubuntu 16 | 18:48 |
| erick3k | 14 the file is there | 18:49 |
| sarnold | what's the task that you're actually trying to solve? | 18:49 |
| erick3k | my problems is network timeouts on boot | 18:51 |
| erick3k | so i want to disable network to start on boot | 18:52 |
| erick3k | or bypass all timeouts or waits for network | 18:52 |
| erick3k | in centos i put ONBOOT=no | 18:52 |
| erick3k | ubuntu 14 change start on (filesystem) | 18:52 |
| erick3k | now on 16 i don't know | 18:53 |
| erick3k | doesn't have same options as 14 | 18:53 |
| erick3k | the root of the problem is that i don't use dhcp for network, instead i use static but is injected by cloud-init | 18:54 |
| erick3k | and is done after the network is started | 18:54 |
| erick3k | so it will timeout | 18:54 |
| erick3k | sarnold this worked perfectly on ubuntu 14 | 18:56 |
| erick3k | http://askubuntu.com/questions/185515/disable-network-configuration-services-during-boot-time | 18:56 |
| erick3k | i want the same for ubuntu 16 | 18:56 |
| erick3k | simple | 18:56 |
| Logos01 | ... I'm assuming you mean 14.04 and 16.04 respectively | 18:57 |
| erick3k | correct | 18:57 |
| Logos01 | ( Y.ear Y.ear . M.onth M.onth ) -- sorry, that's a pet peeve of mine | 18:57 |
| erick3k | but those files are not present on ubuntu 16.04 only in 14.04 | 18:57 |
| Logos01 | Right because it operates differently | 18:58 |
| Logos01 | You're seeing the timeout/delay? | 18:58 |
| erick3k | yes | 18:58 |
| erick3k | very long | 18:58 |
| Logos01 | Oh wait ... "instead I use static but is injected by cloud-init" | 18:58 |
| Logos01 | This is in what cloud provider? | 18:58 |
| erick3k | yes | 18:58 |
| rharper | erick3k: please join #cloud-init and we can help debug from there | 18:59 |
| erick3k | oh trust me | 18:59 |
| erick3k | already tried | 18:59 |
| erick3k | only one that can help is smores and he busy all the time | 18:59 |
| rharper | I work on cloud-init as well, have you filed a bug with the details ? | 19:00 |
| erick3k | i know cloud-init has a timeout on cloud-init-nonet.conf that i already removed | 19:00 |
| erick3k | am not looking for cloud-init timeouts but the looking for network in ubuntu 16 while booting | 19:00 |
| Logos01 | erick3k: Have you looked at /etc/network/interfaces yet? | 19:00 |
| erick3k | yep, no timeout | 19:01 |
| Logos01 | No there wouldn't be there; that's not even a concept. | 19:01 |
| rharper | in general, the 'networking' service in 16.04 will look at the eni and for each iface thats marked 'auto'; it will wait for all of those to become up | 19:01 |
| Logos01 | Care to share your /etc/network/interfaces ? | 19:01 |
| rharper | if one or more of those marked auto does not come up, then the networking service itself blocks; | 19:02 |
| Logos01 | Yeah -- I was going to suggest changing the device from "auto" to "manual" in the config and then let cloud-init handle initializing it | 19:02 |
| Logos01 | It's not so much that it *blocks* | 19:02 |
| Logos01 | But that you never pass the network target and so never reach the multi-user target | 19:02 |
| rharper | but the service itself does block | 19:02 |
| rharper | that's by design | 19:02 |
| erick3k | lord i just want the corresponding files /etc/init/failsafe.conf and /etc/init/rc-sysinit.conf but for ubuntu 16.04 that are no longer present | 19:02 |
| Logos01 | rharper: I'm saying this in systemd terminology | 19:02 |
| rharper | you told it, bring all of these interfaces up | 19:02 |
| rharper | I mean, it's a oneshot service, calling ifup and friends | 19:02 |
| rharper | so, is it systemd is it the unit script? | 19:03 |
| Logos01 | erick3k: The concept doesn't exist anymore; there is no corresponding files. | 19:03 |
| erick3k | right | 19:03 |
| rharper | doesn't really matter; the effect is to block until all interfaces in eni marked auto are up | 19:03 |
| erick3k | so i want to know where are the waiting for dhcp timeouts on boot | 19:03 |
| Logos01 | erick3k: The thing that those files were doing, that you needed to tweak -- it no longer exists. | 19:03 |
| Logos01 | You have a different problem. | 19:03 |
| erick3k | logos01 i understand they no longer exist perhabs why am asking for a 16.04 solution | 19:04 |
| Logos01 | rharper: Well if you speak it in systemd nomenclature it helps you to troubleshoot/diagnose what/where is going on. | 19:04 |
| rharper | it does an ifquery --allow-auto to list the interfaces from eni which are marked auto; until ifup on each of those interfaces returns and they're up it blocks (for some fixed time) | 19:04 |
| Logos01 | erick3k: Right. Are the interfaces marked auto or manual in /etc/network/interfaces ? | 19:04 |
| erick3k | logos01 they will be auto after cloud-init injects it | 19:05 |
| erick3k | now say i seal the vm before using cloud-init | 19:06 |
| Logos01 | erick3k: Right but they're *NOT* auto when the VM is booting. | 19:06 |
| erick3k | what should be on interfaces to avoid any timeouts | 19:06 |
| erick3k | correct | 19:06 |
| erick3k | so what should i put there | 19:06 |
| erick3k | to avoid them timeouts | 19:06 |
| Logos01 | erick3k: Can we see your /etc/network/interfaces file? | 19:06 |
| erick3k | auto lo | 19:07 |
| erick3k | iface lo inet loopback | 19:07 |
| erick3k | source /etc/network/interfaces.d/*.cfg | 19:07 |
| erick3k | thats it | 19:07 |
| Logos01 | Okay what's in the /etc/network/interfaces.d/ directory? | 19:07 |
| rharper | so cloud-init will write a 'dhcp on "first" interface' on instances which do not provide a network configuration; if do not want cloud-init to attempt to configure networking via fallback, you can tell cloud-init to disable network configuration | 19:07 |
| erick3k | https://i.imgur.com/EJDmklx.png | 19:07 |
| rharper | there's a 50-cloud-init eni file which will attemtp to dhcp on one of the interfaces | 19:07 |
| erick3k | yes i can delete that 50-cloud-init.cfg before sealing the vm | 19:08 |
| erick3k | thats not a problem | 19:08 |
| Logos01 | erick3k: lpaste the output of: `grep -H '' /etc/network/interfaces.d/*.cfg` | 19:08 |
| erick3k | the only file that is on inderfaces.d will be deleted | 19:08 |
| erick3k | before sealing the vm logos01 | 19:09 |
| erick3k | so makes no sense showing you that file | 19:09 |
| Logos01 | Well this is where your problem is ... because the machine is attempting to dhcp up -- it has to, because that's how cloud-init *works* | 19:09 |
| erick3k | right | 19:09 |
| Logos01 | Even if you get a static lease on whatever the following interface/address is, that's different. | 19:09 |
| rharper | no, you need to tell cloud-init via config that you don't want it to attempt ot configure networking; you can write out "network: {config: disabled}" into a /etc/cloud/cloud.cfg.d/network-disable.cfg | 19:10 |
| Logos01 | You're getting blocked because you're not *getting* dhcp addressing the machine is looking for. | 19:10 |
| erick3k | so on ubuntu 14 i fixed by using those commands or mods i showed you on ask ubuntu | 19:10 |
| rharper | that prevents cloud-init from attempt to configure networking for you | 19:10 |
| Logos01 | erick3k: That didn't actually fix the problem in question. 14.04 used a *completely different* networking model than 16.04 | 19:10 |
| Logos01 | All it did was make the problem non-blocking to boot. | 19:10 |
| Logos01 | You cannot do that in 16.04. It cannot be done. | 19:10 |
| Logos01 | You have to either resolve the problem or have a successful startup of the networking service that doesn't depend on the problem's being solved. | 19:11 |
| erick3k | ok so i will get 15 minutes time out while booting no matter what i do? | 19:11 |
| Logos01 | Did I just say that? | 19:11 |
| Logos01 | (No.) | 19:11 |
| erick3k | well the solution would be not to get network up until cloud-init injects the static ip | 19:11 |
| erick3k | but that is not how is happening | 19:12 |
| rharper | how are you injecting networking in cloud-init ? | 19:12 |
| erick3k | config drive / nocloud | 19:12 |
| rharper | then you have control over what gets rendered in the 50-cloud-init.cfg file | 19:12 |
| erick3k | and it works | 19:13 |
| rharper | in the network config in the nocloud seed, make sure you don't attempt to dhcp on non-existent interfaces | 19:13 |
| erick3k | https://i.imgur.com/PXBt5N2.png | 19:13 |
| erick3k | perfectly | 19:13 |
| erick3k | only after 15 minutes of timeouts | 19:13 |
| rharper | so I suspect that your first nic isn't getting the name 'eth0' | 19:14 |
| rharper | 16.04 uses persistent nic naming, so it'll be like ens3 or ens1p1 | 19:14 |
| erick3k | yea i did the grub biosdevicename thingy | 19:14 |
| erick3k | na, it must be eth0 | 19:15 |
| rharper | that looks fine, if eth0 is present, then ifup on eth0 should be just fine; | 19:15 |
| erick3k | right but when i seal the vm | 19:15 |
| erick3k | eth0 will not exist until cloud-init injects it | 19:15 |
| erick3k | so ubuntu will timeout trying to get it with dhcp | 19:15 |
| rharper | so you must not be embedded the datasource into /var/lib/cloud/seed/nocloud-net/ | 19:16 |
| erick3k | ummm | 19:16 |
| erick3k | let me try | 19:16 |
| rharper | when you boot it soemwhere else, it'll have a new instance id, in which case the previous config may not apply | 19:16 |
| rharper | in there you'll also set a meta-data with an instance-id: <unique string> | 19:16 |
| erick3k | yes that i delete before sealing | 19:17 |
| erick3k | correct? | 19:17 |
| erick3k | /var/lib/cloud/instances | 19:17 |
| rharper | if this is a template then, yes; | 19:18 |
| erick3k | okay no timeout | 19:22 |
| erick3k | that worked | 19:23 |
| erick3k | i think i wasn't clearing /var/lib/cloud/instances and caused the problem | 19:23 |
| rharper | you can also write a network-config file for the nocloud-net seed like so | 19:23 |
| rharper | http://paste.ubuntu.com/24335893/ | 19:24 |
| rharper | format is documented here: http://curtin.readthedocs.io/en/latest/topics/networking.html ; I've an PR to get that published under the cloud-init docs | 19:25 |
| ThiagoCMC | Guys, I'm trying to launch an Instance on Ocata / Ubuntu 16.04, with DPDK and hugepages, the following error is appearing on nova-compute.log: | 20:18 |
| ThiagoCMC | can't open backing store /dev/hugepages-1048576/libvirt/qemu for guest RAM: Permission denied | 20:18 |
| ThiagoCMC | Any tips? | 20:18 |
| ThiagoCMC | Might be related to https://bugs.launchpad.net/ubuntu/+source/libvirt/+bug/1524737 ? | 20:18 |
| ubottu | Launchpad bug 1524737 in libvirt (Ubuntu Wily) "systemd presents hugetblfs at /dev/hugepages" [Undecided,New] | 20:18 |
| ThiagoCMC | I'm also seeing "apparmor="DENIED"" at my syslog... =/ | 20:21 |
| sarnold | ThiagoCMC: ubuntu-bug libvirt, please, and make sure to include those DENIED lines | 20:25 |
| ThiagoCMC | Damn... ok! | 20:25 |
| ThiagoCMC | =P | 20:25 |
| ThiagoCMC | "ubuntu-bug libvirt0", right? | 20:27 |
| ThiagoCMC | never mind, sending | 20:27 |
| rharper | https://help.ubuntu.com/lts/serverguide/DPDK.html | 20:28 |
| rharper | may be of help as well | 20:28 |
| sarnold | rharper: nice, thanks | 20:29 |
| rharper | all thanks go to cpaelzer | 20:29 |
| sarnold | cpaelzer, thanks for the nice https://help.ubuntu.com/lts/serverguide/DPDK.html guide it looks like good reading :) | 20:30 |
| ThiagoCMC | Reported: https://bugs.launchpad.net/ubuntu/+source/libvirt/+bug/1524737 | 20:33 |
| ubottu | Launchpad bug 1524737 in libvirt (Ubuntu Wily) "systemd presents hugetblfs at /dev/hugepages" [Undecided,Confirmed] | 20:33 |
| ThiagoCMC | Oops | 20:33 |
| ThiagoCMC | Damn clipboard are... | 20:33 |
| ThiagoCMC | https://bugs.launchpad.net/cloud-archive/+bug/1680956 | 20:33 |
| ubottu | Launchpad bug 1680956 in Ubuntu Cloud Archive "Fail to launch an OpenStack Instance with hugepages on top of OVS+DPDK" [Undecided,New] | 20:33 |
| ThiagoCMC | I can easily use Ubuntu with DPDK for my VMs... | 20:33 |
| ThiagoCMC | Problem is when with OpenStack... Plain KVM is fine. | 20:34 |
| nacc | ThiagoCMC: do you know ifthe libvirt in openstack has the fix to LP: #1524737 in it? | 20:35 |
| ubottu | Launchpad bug 1524737 in libvirt (Ubuntu Wily) "systemd presents hugetblfs at /dev/hugepages" [Undecided,Confirmed] https://launchpad.net/bugs/1524737 | 20:35 |
| ThiagoCMC | It is: libvirt-bin 2.5.0-3ubuntu5~cloud0 | 20:36 |
| nacc | ThiagoCMC: do you know ifthe libvirt in openstack has the fix to LP: #1524737 in it? | 20:36 |
| ubottu | Launchpad bug 1524737 in libvirt (Ubuntu Wily) "systemd presents hugetblfs at /dev/hugepages" [Undecided,Confirmed] https://launchpad.net/bugs/1524737 | 20:36 |
| nacc | bah | 20:36 |
| nacc | ThiagoCMC: do you know ifthe libvirt in openstack has the fix to LP: #1524737 in it? | 20:36 |
| nacc | i'm also a bit confused why that bug was closed by 1.2.21-2ubuntu3 | 20:36 |
| ThiagoCMC | Hmmm.. I don't... | 20:36 |
| ThiagoCMC | If it was fixed a long time ago... | 20:37 |
| nacc | ThiagoCMC: sorry bad keystroke earlier | 20:37 |
| ThiagoCMC | np | 20:37 |
| nacc | ah it was fixed in 1.2.21-2ubuntu2 and lp is being weird | 20:37 |
| ThiagoCMC | maybe re-introduced? | 20:37 |
| nacc | ThiagoCMC: about to eat lunch but i can check on it | 20:38 |
| ThiagoCMC | Enjoy it! :-D | 20:38 |
| erick3k | omg help | 21:15 |
| erick3k | rharper you still on? | 21:16 |
| tsimonq2 | !help | erick3k | 21:16 |
| ubottu | erick3k: Please don't ask to ask a question, simply ask the question (all on ONE line and in the channel, so that others can read and follow it easily). If anyone knows the answer they will most likely reply. :-) See also !patience | 21:16 |
| erick3k | the machine now gets stuck here https://i.imgur.com/DVpf0RX.png | 21:16 |
| erick3k | why is raid loading am using no raid thing | 21:16 |
| nacc | erick3k: that's mdadm reloading | 21:29 |
| nacc | erick3k: *loading | 21:29 |
| nacc | erick3k: it has to know what raid levels are allowed so that you can use raid if you want or not | 21:29 |
| nacc | erick3k: i'm 99% sure that's not what' | 21:29 |
| nacc | *that's not what's hanging | 21:29 |
| nacc | ThiagoCMC: back, sorry for the delay | 21:31 |
| erick3k | nacc ok checking | 21:32 |
| ThiagoCMC | nacc, oh, that's totally ok! ^_^ | 21:33 |
| nacc | ThiagoCMC: ok, so 2.5.0-3ubuntu5, in the source, does have | 21:34 |
| nacc | # for access to hugepages | 21:34 |
| nacc | owner "/run/hugepages/kvm/libvirt/qemu/**" rw, | 21:34 |
| nacc | owner "/dev/hugepages/libvirt/qemu/**" rw, | 21:34 |
| ThiagoCMC | Hmm... | 21:35 |
| ThiagoCMC | =/ | 21:35 |
| nacc | ThiagoCMC: should be abel to confirm by looking at /etc/apparmor.d/abstractions/libvirt-qemu on the host | 21:36 |
| ThiagoCMC | Checking... | 21:36 |
| nacc | ThiagoCMC: oh! | 21:37 |
| nacc | ThiagoCMC: the path changed! | 21:37 |
| nacc | ThiagoCMC: why is it /dev/hugepages-1048576 ? | 21:37 |
| nacc | ThiagoCMC: can you pastebin `ls -ahl /dev/hugepages*`? | 21:37 |
| ThiagoCMC | Yes, /etc/apparmor.d/abstractions/libvirt-qemu have those two lines. | 21:38 |
| ThiagoCMC | ls -ahl output: https://paste.ubuntu.com/24336518/ | 21:40 |
| nacc | ThiagoCMC: is this the system where you were using 1g hugepages? | 21:40 |
| ThiagoCMC | yep | 21:41 |
| ThiagoCMC | I can launch an Instance on top of OVS+DPKD (OVN topology), but, when I try the hugepages on OpenStack's flavor, it doesn't boot because of this apparmor stuff | 21:42 |
| nacc | sarnold: being completely dumb about apparmor, can you look at LP: #1689056 ? I am unable to put my comment in lp, due to timeouts righ tnow, but: http://paste.ubuntu.com/24336524/ | 21:42 |
| ubottu | Error: Launchpad bug 1689056 could not be found | 21:42 |
| nacc | bah | 21:42 |
| nacc | LP: #1680956 | 21:42 |
| ubottu | Launchpad bug 1680956 in Ubuntu Cloud Archive "Fail to launch an OpenStack Instance with hugepages on top of OVS+DPDK" [Undecided,New] https://launchpad.net/bugs/1680956 | 21:42 |
| nacc | sarnold: also, why does `man apparmor` on 17.04 refer to upstart? :) | 21:43 |
| nacc | "Ubuntu systems use upstart(8) instead of a traditional SysV init system." | 21:43 |
| nacc | ugh | 21:43 |
| ThiagoCMC | Nevertheless, something else is going on here... I mean, this is for sure a problem (LP 1680956). And, when I try it without hugepages, the VM boots up but, it can't ping OVN L3 Router... But, this is a different problem entirely. | 21:45 |
| ubottu | Launchpad bug 1680956 in Ubuntu Cloud Archive "Fail to launch an OpenStack Instance with hugepages on top of OVS+DPDK" [Undecided,New] https://launchpad.net/bugs/1680956 | 21:45 |
| ThiagoCMC | Oh this ubottu... lol | 21:45 |
| ThiagoCMC | =P | 21:45 |
| nacc | sarnold: and one last thing for the security time, the manpage refers to : http://wiki.apparmor.net/index.php/Distro_ubuntu | 21:45 |
| nacc | which, um, is as currnet as 11.04 :) | 21:46 |
| erick3k | nacc what could it be? | 21:54 |
| erick3k | i can't find out why is getting stuck | 21:54 |
| erick3k | nacc now booted, where can i look for that ugly timeout | 21:55 |
| erick3k | ? | 21:55 |
| nacc | erick3k: i'd look in syslog, messages, dmesg, etc | 21:55 |
| erick3k | k | 21:55 |
| erick3k | nacc is the stupid network again | 21:57 |
| erick3k | i can reproduce by using cloud-init network as dhcp | 21:57 |
| nacc | erick3k: ah | 21:57 |
| erick3k | i use static btw | 21:57 |
| nacc | erick3k: do you not have network on first boot or something? or using dhcp but not really? | 21:57 |
| erick3k | this is the options on the template hold on | 21:58 |
| sarnold | nacc: wow, nice bugs all around :) | 21:58 |
| erick3k | https://i.imgur.com/IG9URrT.png | 21:58 |
| nacc | sarnold: my guess on profile seems ok, at least :) but will probably need a fix and sru, cpaelzer | 21:58 |
| erick3k | should i remove the start on boot? | 21:58 |
| nacc | erick3k: i have no idea how ovirt works so i don't know | 21:59 |
| erick3k | or untick network | 21:59 |
| erick3k | nice back to square 1 again :( | 21:59 |
| erick3k | thats why | 21:59 |
| erick3k | before | 21:59 |
| nacc | erick3k: i would guess you could try those options, but i wonder if it will just fail to boot at all | 21:59 |
| erick3k | i wanted to attack the stupid timeouts | 21:59 |
| erick3k | not cloudinit or ovirt who cares | 22:00 |
| erick3k | is stupid ubuntu timing out looking for dhcp | 22:00 |
| erick3k | all other O.S work great, including 14.04 | 22:00 |
| erick3k | 5 days trying to fix the stupid timeout, bout to give up on ubuntu | 22:01 |
| erick3k | my customers can stick to 14.04 lol | 22:01 |
| nacc | this seems relevant: https://review.openstack.org/#/c/416664/ | 22:04 |
| nacc | https://bugs.launchpad.net/tripleo/+bug/1653812 | 22:04 |
| ubottu | Launchpad bug 1653812 in tripleo "Five minute delay DHCP'ing isolated nics" [High,Fix released] | 22:04 |
| nacc | looks like ONBOOT is the issue, but i'm not sure if you actually need it for your case | 22:04 |
| erick3k | nacc on centos 7 yes i put ONBOOT=no and thats it issue fixed | 22:05 |
| erick3k | is there an option like that in ubuntu? | 22:05 |
| nacc | erick3k: where do you put that? | 22:05 |
| erick3k | centos /etc/sysconfig/network-scripts/ifcfg-eth0 | 22:06 |
| erick3k | is there a way to completly block the network from starting on boot and then just rc.local a command to start it? | 22:07 |
| erick3k | am desperate | 22:07 |
| erick3k | anything works i dont care | 22:07 |
| nacc | erick3k: ip=off maybe | 22:08 |
| nacc | erick3k: on the kernel commandline | 22:08 |
| nacc | but i don't understand, if you're cloud based, you depend on network in order to boot, i'd assume | 22:08 |
| erick3k | not really since am not using dhcp | 22:08 |
| erick3k | am using static network | 22:08 |
| nacc | that doesn't matter | 22:09 |
| erick3k | i put the ip's | 22:09 |
| nacc | you still depend on network | 22:09 |
| nacc | it sounds like you've got a configuration issue | 22:09 |
| nacc | somewhere, id on't understand your deployment | 22:09 |
| nacc | but it should not be doing dhcp, i think is your root point | 22:09 |
| erick3k | right | 22:09 |
| nacc | but you aren't telling it (afaict) that it's got static networking | 22:09 |
| nacc | erick3k: you can of course pass networking information to guests on the kernel command-line, but i don't understand when you're having this issue | 22:09 |
| erick3k | so i got 5 oses working but ubuntu 16.04 am sure the system / deployment is fine | 22:10 |
| nacc | first boot with cloud-init? | 22:10 |
| erick3k | i just want ubuntu to stop waiting for dhcp | 22:10 |
| erick3k | where is the dhcp wait time set in ubuntu 16? | 22:10 |
| nacc | probably /etc/dhcp/dhclient.conf | 22:10 |
| nacc | yeah, it's set to 300 there | 22:10 |
| nacc | but again, it only would do that if it is configured to dhcp at all | 22:11 |
| nacc | so don't configure your images to dhcp | 22:11 |
| erick3k | ok gonna try that | 22:11 |
| nacc | i don't understand why you think you don't have a configuration issue | 22:11 |
| erick3k | yes am not sure why is even picking dhcp | 22:11 |
| nacc | i don't carea bout your other images | 22:11 |
| erick3k | i don't have no dhcp nowhere | 22:11 |
| nacc | yes, i understand -- so how are your other instances getting network configuration? | 22:11 |
| nacc | erick3k: you boot an instance and configure it? | 22:11 |
| erick3k | all get them throught cloud-init | 22:11 |
| nacc | erick3k: how do you do that? | 22:11 |
| nacc | *how* | 22:12 |
| nacc | they dont' have network at boot | 22:12 |
| nacc | since you're not using dhcp | 22:12 |
| nacc | i think you or I am missing something obvious | 22:12 |
| erick3k | no, you clean the vm, sealed, boot it with cloud-init (ovirt plugs a floppy with config drive / nocloud) and boots while picking up the info injected throught the floppy | 22:12 |
| nacc | ah you are using a config drive | 22:12 |
| nacc | ok | 22:12 |
| erick3k | oyes | 22:13 |
| nacc | i don't know what 'sealed' is | 22:13 |
| nacc | yeah, so if you don't need network to boot, don't set it to onboot (that gui you showed earlier) | 22:13 |
| nacc | or send ip=off, i guess | 22:13 |
| erick3k | just get it ready for customer, delete /var/lib/cloud/instances, history, interfaces etc | 22:13 |
| erick3k | yes, i tried all options on that gui, nothing works | 22:14 |
| erick3k | where do i add that ip=off | 22:14 |
| erick3k | GRUB_CMDLINE_LINUX="net.ifnames=0 biosdevname=0" | 22:15 |
| erick3k | there nacc? | 22:15 |
| nacc | yeah, i think so | 22:15 |
| nacc | erick3k: but that will disable networking period, possibly, that's what i'm not sure about | 22:15 |
| nacc | erick3k: hence, test it :) | 22:15 |
| erick3k | by disable you mean you can't even start it with manual command after it boots? | 22:16 |
| nacc | erick3k: i'm not sure, i can't remember if that twiddles anythin gmore than just not doing any network configuration at boot | 22:17 |
| erick3k | that would work | 22:17 |
| erick3k | trying | 22:17 |
| sarnold | kind of strange that the only mention of 'network-interface' in the docs is on the nocloud datasource :( http://cloudinit.readthedocs.io/en/latest/topics/datasources/nocloud.html?highlight=network-interfaces | 22:18 |
| nacc | sarnold: right, but earlier erick3k said they didn't have control over cloud-init | 22:19 |
| nacc | sarnold: but agreed, it's strange | 22:19 |
| sarnold | nacc: I'm surprised no one's needed to integrate cloud-init with an IPAM service before to make this easier.. | 22:20 |
| jgrimm | sarnold, nacc: more docs on the way fwiw -> https://code.launchpad.net/~raharper/cloud-init/+git/cloud-init/+ref/network-config-doc | 22:20 |
| erick3k | ok with ip=off booted now let me add an ip see what happens | 22:21 |
| nacc | jgrimm: thanks | 22:21 |
| * jgrimm EOWs | 22:21 | |
| === jgrimm is now known as jgrimm-away | ||
| sarnold | jgrimm-away: thanks, have a good weekend | 22:23 |
| * jgrimm-away waves | 22:30 | |
| erick3k | nacc ip=off just completly remove network | 22:31 |
| erick3k | cloud-init doesn't even add an interface | 22:31 |
| erick3k | nacc, is there a default that after the 300 seconds timeout on dhcp, ubuntu will retry? | 22:34 |
| erick3k | i see on /etc/dhcp/dhclient.conf is commented out #retry 60; so by default ubuntu does not retry? | 22:34 |
| nacc | erick3k: did you rebuild your initrd after changing that file? | 22:38 |
| erick3k | if you mean update-grub yes | 22:39 |
| nacc | no | 22:39 |
| nacc | that updates grub | 22:39 |
| nacc | has nothing to do with your initrd | 22:39 |
| nacc | i'm fairly sure you need to update the initrd on the image in order for that file to affect boot | 22:39 |
| erick3k | yes i know :) | 22:40 |
| erick3k | eitherway ip=off is a nono | 22:40 |
| erick3k | is like removing the nic card physically | 22:40 |
| erick3k | nacc sorry you mean dhclient? | 22:45 |
| erick3k | file | 22:45 |
| nacc | erick3k: yeah /etc/dhcp/dhclient.conf edited without an initrd update doesn't do anything afaict | 22:45 |
| erick3k | ummm | 22:45 |
| erick3k | k doing it | 22:49 |
| erick3k | nacc update-initramfs -? | 22:51 |
| erick3k | -u? | 22:51 |
| nacc | erick3k: yeah, maybe with -k <version> or -k all | 22:52 |
| sarnold | if you're editing the image and you just don't want networking, could you use systemctl disaable networking or something similar? | 22:54 |
| nacc | sarnold: afaict, they do want networking, just not hte dhcp on boot | 22:55 |
| nacc | sarnold: and i still don't understand how this works on other OSes or if this is a systemd change or what | 22:56 |
| nacc | as i would think most would be attempting to dhcp on all interfaces by default, if not told otherwise | 22:56 |
| erick3k | at this point | 22:56 |
| erick3k | am not even sure is a dhcp problem | 22:56 |
| erick3k | i mean a machine its been stuck for 1 hour | 22:57 |
| erick3k | tried that nothing | 22:57 |
| erick3k | same | 22:57 |
| erick3k | stuck https://i.imgur.com/UDKYAHZ.png | 22:58 |
| erick3k | why is raid showing any idea? am not using any raid device | 22:58 |
| nacc | erick3k: again, i already explained that to you | 22:58 |
| sarnold | you could probably blacklist the modules if you don't want their startup routines firing | 22:58 |
| nacc | erick3k: mdadm loads and determines what raid levels are supported | 22:58 |
| nacc | iirc, at that poitn, it's trying to mount your root fs | 22:59 |
| nacc | (also based upon my own dmesg) | 22:59 |
| erick3k | the only difference | 23:00 |
| erick3k | between me firing the vm manually and with my script (they get stuck) is that the disk size is changed | 23:00 |
| erick3k | could that make ubuntu not boot? | 23:00 |
| erick3k | or get stuck there | 23:01 |
| erick3k | i am sorry guys, maybe has nothing to do with the network | 23:02 |
| nacc | you're changing the disk size under a fs? | 23:02 |
| erick3k | i increased the disk size before turning on the machine and reproduced the error :( | 23:03 |
| erick3k | yes, virtio | 23:03 |
| nacc | well, at least you disparaged ubuntu in the process! :) | 23:03 |
| * nacc goes back to other work | 23:03 | |
| erick3k | lol | 23:03 |
| erick3k | why, isnt it suppose to run growpart and just expand? | 23:04 |
| erick3k | like the other cloud images i have working? | 23:04 |
| erick3k | nvm me | 23:05 |
| erick3k | it booted and did resize the disk https://i.imgur.com/TINxMPD.png | 23:05 |
| erick3k | i don't know guys i give up for now, maybe one will boot tomorrow and will collect some more info. Thank you nacc and all for your help and time. | 23:11 |
| nacc | erick3k: gl! | 23:13 |
Generated by irclog2html.py 2.7 by Marius Gedminas - find it at mg.pov.lt!
