| echosystm | i've added a service to systemd | 00:31 |
|---|---|---|
| echosystm | i can manually start it no problems | 00:31 |
| echosystm | however, it never starts on boot | 00:31 |
| echosystm | systemctl enable myservice does nothing | 00:31 |
| echosystm | any ideas? | 00:31 |
| ChmEarl | echosystm, any `WantedBy=`? | 00:45 |
| echosystm | nope | 00:46 |
| ChmEarl | [Install] -> WantedBy=multi-user.target | 00:47 |
| echosystm | okie doke | 00:47 |
| ChmEarl | echosystm, if you know how it needs to sequence, there are other keys to use | 00:48 |
| echosystm | ChmEarl: that fixed it | 00:53 |
| echosystm | thanks | 00:53 |
| drab | anybody around familiar with hp switches way of doing trunking/lacp? | 02:29 |
| drab | I had lacp on 4 ports and bonding with lacp on the host | 02:30 |
| drab | and it worked | 02:30 |
| drab | but you can't have dhcpsnooping on a dyn trunk which is what enabling lacp will give you | 02:30 |
| drab | so I have to set up a static trunk, but then I'm not sure what on the server side | 02:31 |
| lordievader | Good morning. | 06:14 |
| patsToms | morning | 06:19 |
| patsToms | could I make local repositories mirror on my raspberry? | 06:19 |
| hateball | provided it has ample storage, yes | 06:24 |
| hateball | !aptmirror | 06:24 |
| hateball | boo | 06:24 |
| zioproto | hello Openstack folks. We are facing a new Neutron bug ad SWITCH. I found that it is a known bug. https://bugs.launchpad.net/neutron/+bug/1632540 | 08:11 |
| ubottu | Launchpad bug 1632540 in neutron "l3-agent print the ERROR log in l3 log file continuously ,finally fill file space,leading to crash the l3-agent service" [Undecided,In progress] | 08:11 |
| zioproto | we see this bug in Mitaka but it looks like it is still present in master | 08:12 |
| === schmidtm_ is now known as schmidtm | ||
| zioproto | jamespage: are you online ? | 08:58 |
| zioproto | jamespage, coreycb we are facing some neutron sync issue between neutron-servers and agents because of the time change from CET to CEST in the night of 26th of March. Let me know if you faced any similar issues. We had a small 3 minutes time window when the time changed where neutron thought all its agents where offline since 1 hour. This caused a lot of | 09:00 |
| zioproto | load. Neutron tried to recreate all routers and reapply all iptables rules. I am still reading logs ... | 09:00 |
| ivoks | zioproto: i'm not jamespage or coreycb, but are you sure those are related to time? | 09:34 |
| ivoks | zioproto: time tracking in unix is done with unix clock, and date presented to user is almost just translation into human readable format (+ some mangling like DST) | 09:35 |
| ivoks | leap seconds are more likely to cause issues like this | 09:36 |
| zioproto | there is a race condition well this happened exactly when the time changed from CET to CEST, yes it could be a leap second issue | 09:41 |
| zioproto | I just figured out I have heartbeat_timeout_threshold=0 in my neutron.conf, this could be the source of trouble | 09:41 |
| ivoks | leap seconds in 2017 are scheduled for june and december, iirc | 09:42 |
| zioproto | so, it is not a leap second :) | 09:43 |
| ivoks | it's not | 09:43 |
| ivoks | we've always had incrementing leap seconds so far | 09:43 |
| ivoks | which are fairly simple and uneventful | 09:44 |
| ivoks | problem will be when we will have to do a negative one; which means that earth rotation has slowed down | 09:44 |
| ivoks | but in any case, don't get date to get in your way | 09:46 |
| ivoks | that couldn't have caused that issue | 09:46 |
| ivoks | time tracking is based on time pased since unix epoch, and that is not impacted by DST | 09:46 |
| zioproto | ok, I am still debugging the issue, I will report here when I am sure that happened | 09:50 |
| === Isla_de_Muerte is now known as NwS | ||
| fnordahl | xnox: ping? | 11:07 |
| xnox | fnordahl, hello | 11:08 |
| fnordahl | xnox: hi there. | 11:08 |
| fnordahl | xnox: re LP: #1642966 | 11:08 |
| ubottu | Launchpad bug 1642966 in cups (Ubuntu Yakkety) "package cups-daemon 2.1.3-4 failed to install/upgrade: subprocess new pre-removal script returned error exit status 1" [High,Fix committed] https://launchpad.net/bugs/1642966 | 11:08 |
| fnordahl | xnox: the SRU'ed package caused just that error here now apport-info in LP: #1676380 | 11:09 |
| ubottu | Launchpad bug 1676380 in cups (Ubuntu) "package cups-daemon 2.1.3-4 failed to install/upgrade: subprocess new pre-removal script returned error exit status 1" [Undecided,New] https://launchpad.net/bugs/1676380 | 11:09 |
| fnordahl | xnox: ftr, the update has now been pulled. | 13:45 |
| zul | coreycb; ping do we really need this? https://git.launchpad.net/~ubuntu-server-dev/ubuntu/+source/keystone/tree/debian/patches/add-version-info.patch, keystone is the only package that does this and my devops side of me you really dont want server banners to be identified in production | 13:58 |
| freakynl | Hi, anyone know why a minimal server install has open-iscsi, lxcfs and snapd running by default? | 13:58 |
| zul | coreycb: i know why we added it but its something we started and dropping it makes keystone so much more easier to maintain | 13:59 |
| coreycb | zul, i'm not really sure if we can drop that or not. you may want to run it by james. | 14:00 |
| zioproto | hey there | 14:00 |
| zul | coreycb: ugh | 14:01 |
| zioproto | hey guys the ubuntu packaging for neutron still includes a cron job /etc/cron.d/neutron-l3-agent-netns-cleanup | 14:02 |
| zioproto | I found another race condition when this cronjob deletes namespaces and other parts of neutron want to apply iptables rules to it forever | 14:03 |
| zioproto | I will open soon a bug | 14:03 |
| zioproto | just wondering, why this cronjob ? other distributions dont ship it AFAIK | 14:03 |
| coreycb | zioproto, i think we may have dropped the cron jobs in a recent release | 14:04 |
| coreycb | zioproto, checking | 14:04 |
| zioproto | that would be great, for sure is still there in Mitaka :) | 14:04 |
| tom___ | Hi, i use ubuntu server 16.04.2 but it takes unusually long to start the server. See the full dmesg (http://paste.ubuntu.com/24260947/) and the lines which seem to hang somehow (http://paste.ubuntu.com/24260952/). | 14:06 |
| zioproto | tom___: check if you have weird udev rules | 14:09 |
| tom___ | zioproto: are udev rules capable to delay the boot process that much? | 14:11 |
| zioproto | yes we had a problem like that when we upgraded Trusty to Xenial | 14:12 |
| zioproto | tom___: we had a line that we added it looked like | 14:13 |
| zioproto | tom___: KERNEL!="sr*", IMPORT{builtin}="blkid" | 14:13 |
| zioproto | this was in our custom file in /etc/udev/rules.d/ | 14:13 |
| zioproto | in trusty it was no problem | 14:13 |
| zioproto | in xenial there was a delay at boot for 120 seconds | 14:14 |
| tom___ | Do you remember which udev-rule was the problem? | 14:15 |
| tom___ | zioproto: It is a clean install with no custom udev rules. Should I add one myself? | 14:15 |
| zioproto | this problem was very special for our setup | 14:16 |
| zioproto | because we create symlinks for disks in /devv | 14:16 |
| zioproto | so no matter if the the disk is /dev/sda or /dev/whatever | 14:16 |
| zioproto | we create symlinks to /dev/sra | 14:16 |
| zioproto | so | 14:17 |
| zioproto | you most probably dont have our same problem | 14:17 |
| zioproto | tom___: just make sure you dont have custom stuff in udev because that could be a root cause for slow boot | 14:17 |
| tom___ | zioproto: no there is nothing like that. I found somewhere that the raid controller could be the problem as i have one but dont use it. But adding raid=nodetect as boot parameter didn't help anything. | 14:18 |
| coreycb | zioproto, https://bugs.launchpad.net/cloud-archive/+bug/1623664 | 14:22 |
| ubottu | Launchpad bug 1623664 in Ubuntu Cloud Archive "Race between L3 agent and neutron-ns-cleanup" [Undecided,New] | 14:22 |
| coreycb | zioproto, they haven't been dropped yet | 14:22 |
| zioproto | oh, so not even in master, right ? | 14:24 |
| coreycb | zioproto, correct | 14:25 |
| zioproto | coreycb: but that bug is fixed with the patch I merged in Barcelona | 14:25 |
| zioproto | I mean router deleting is not a problem anymore | 14:25 |
| zioproto | now I triggered a similar condition | 14:25 |
| zioproto | if you have a glitch in rabbitmq | 14:25 |
| zioproto | the neutron server asks all l3agents to reapply security groups | 14:26 |
| zioproto | and you fail apply the iptables rules in a namespace that does not exist because the cronjob deleted it | 14:26 |
| zioproto | I am writing a neutron patch to make the code more robust again doing stuff on namespaces that are gone for some reason | 14:26 |
| zioproto | but the real bug here is that cronjob | 14:26 |
| zioproto | :D | 14:26 |
| drab | moin | 14:27 |
| zioproto | coreycb: I cant make a patch that creates again the namespace, to have the cron again coming and deleting the namespace, we have to find a common design between the neutron devs and the ubuntu packaging | 14:28 |
| coreycb | zioproto, it looks to me like namespaces are cleaned up my neutron and neutron-lbaas upstream code these days so i think we can drop these cron jobs | 14:35 |
| zioproto | ok, but we need a fix at least for Mitaka ? what about Newton ? | 14:38 |
| coreycb | zioproto, we'd have to start at pike and work our way back with SRUs | 14:43 |
| zioproto | coreycb: this patch needs a lot of love, but this is the key idea https://review.openstack.org/450271 | 14:44 |
| zul | zioproto: can you open up a bug about dropping the netns cron stuff please? | 15:30 |
| coreycb | zul, we can use this bug: https://bugs.launchpad.net/cloud-archive/+bug/1623664 | 15:59 |
| ubottu | Launchpad bug 1623664 in Ubuntu Cloud Archive "Race between L3 agent and neutron-ns-cleanup" [Undecided,New] | 15:59 |
| === JanC_ is now known as JanC | ||
| === dpb1_ is now known as dpb1 | ||
| === MapspaM is now known as SpamapS | ||
| zul | coreycb: okie dokie | 16:29 |
| zul | coreycb: http://paste.ubuntu.com/24261969/ | 16:51 |
| coreycb | zul, looks good. i'd just update the changelog message to say that upstream now cleans up netns. | 16:55 |
| coreycb | zul, neutron-lbaas needs an update too | 16:55 |
| zul | coreycb: ack | 16:56 |
| mike-zal | what is host server key? can't find anything on that :( . it sounds like the key of the host computer where my VPS is, but what it has to do with my VPS? | 17:49 |
| nacc | mike-zal: in what context? ssh key? | 17:49 |
| mike-zal | well, when I connect to my server with ssh through filezilla, I got a meesage: | 17:50 |
| mike-zal | host server key is unknown. you have no guarantee that server is the one you want. | 17:51 |
| mike-zal | then I have some details about the host: name of the server, hostkey algorith and some fingerprints | 17:52 |
| mike-zal | hmm... it seems like it's about my VPS key, because as host I see name of my server | 17:53 |
| mike-zal | but strange thing is: I don't have any keys yet | 17:53 |
| mike-zal | I can connect to the server when I agree to trust it anyway | 17:54 |
| mike-zal | still, I am confused by the message | 17:54 |
| nacc | mike-zal: can you be clearer where the error is? you say 'host' and 'VPS' | 17:54 |
| nacc | mike-zal: if you ssh from your machine to the VPS, do you see the same error? | 17:54 |
| mike-zal | nacc: in filezilla, it shows when I try to connect with my VPS | 17:54 |
| mike-zal | in terminal it's all good | 17:55 |
| mike-zal | it also works with filezilla, the difference is that I get this warinig message | 17:55 |
| nacc | mike-zal: sounds like something to ask filezilla about? | 17:55 |
| mike-zal | nacc: I hoped that some commone server knwolege would explain it | 17:56 |
| nacc | mike-zal: if ssh doesn't say it can't find the host key, then i don't know why filezilla does | 17:56 |
| mike-zal | nacc: also, I was wondering if that is not the reason why krusader won't connect to my server. it always klaims about some changed keys | 17:56 |
| nacc | mike-zal: it's not good for your keys to be changing | 17:57 |
| mike-zal | nacc: the thing is, I don't have keys | 17:57 |
| mike-zal | yet | 17:57 |
| nacc | mike-zal: what do you mean? sshd won't run without there being a host key (afaik) | 17:58 |
| mike-zal | when I was connecting through root on krusader, all was well. but when I blocked root and try to log with a user, it won't let me | 17:58 |
| blackflow | mike-zal: that's the host's signature key. accept it the first time you connect | 17:58 |
| nacc | mike-zal: i think you're confusing your VPS' host key and your local ssh keys | 17:58 |
| mike-zal | yes, you are right | 17:59 |
| mike-zal | I knew that something was missing for me ;) | 17:59 |
| mike-zal | ok, must look what is this host key then | 17:59 |
| blackflow | it's the host side of key exchange algorightm | 17:59 |
| blackflow | *algorithm | 18:00 |
| mike-zal | now I know that's not about ssh keys, I know what to look for, thanks :) | 18:01 |
| mike-zal | found this: https://www.vandyke.com/solutions/host_keys/host_keys.pdf | 18:01 |
| blackflow | well, it _is_ about ssh keys, just on the server side :) | 18:02 |
| blackflow | it's not about public key authentication, specifically. | 18:03 |
| mike-zal | then can you refer to me some good source on this? I never knew about tis host key and no article mentioned it before, or at least not in a clear manner | 18:04 |
| mike-zal | I never creates that key, never messed with anything related to it, or at least not awarely. | 18:05 |
| mike-zal | created* | 18:05 |
| blackflow | mike-zal: man ssh then /HOST KEYS | 18:05 |
| sarnold | the sshd startup script normally creates the host keys on first boot | 18:05 |
| nacc | mike-zal: iirc, sshd creates keys on start if they are not present | 18:05 |
| sarnold | some people generate the keys in their host automation script and ditribute the keys to hosts tht way | 18:05 |
| mike-zal | ok. then maybe this host key is the "key" why krusader won't connect to the VPS that way, although strangely it had no issues with root user | 18:07 |
| mike-zal | it's not a big issue, there are plenty of ways connecting to server then krusader but I just don't like not to know what is it ;)? | 18:08 |
| mike-zal | ok, I start to unserstand it slowly. I did changed some things during server setup i cryptographic keys, just as articles suggested. so I guess that's the change that causesed krusader to complain. | 18:13 |
| mike-zal | and filezilla just checks the key and I must remeber it during first connection and if it doesn't change "by iteslf" in future, it's all good | 18:14 |
| mike-zal | ups, I meant: filezilla must remeber it | 18:14 |
| sarnold | this is often called "TOFU", "trust on first use" | 18:15 |
| nacc | mike-zal: right, the host key is like a fingerprint of the remote server. you locally (at some point) said 'remember this server is saying to trust its identity as being this key' and then you hanged the key | 18:17 |
| mike-zal | ok | 18:18 |
| mike-zal | so this is merely first connection info, on filezilla part, hence the message, "host server key is unknown" and it gives me possibility to remeber it | 18:21 |
| nacc | mike-zal: presumably | 18:21 |
| nacc | mike-zal: yes, host key checking assumes you know when the keys change, i guess | 18:21 |
| sarnold | and if it's unknown because it's the first time you've connected to it, then that makes sense. if it's unknown and you've connected to it before, then perhaps someone is running a man-in-the-middle attack on you. | 18:22 |
| mike-zal | no, it's first connection | 18:22 |
| mike-zal | but it is possible that before I had the chanse to secure server, someone got in and I didn't notice it. | 18:22 |
| mike-zal | let's assume to worst case scenario: where to look for traces of that? | 18:23 |
| mike-zal | is there any way to check date of last change on file that holds that host key? where is it located? | 18:26 |
| mike-zal | sarnold: during my ssl setup, I installed some cryptographic packages as suggested on article. could they have changed it? | 18:28 |
| sarnold | mike-zal: it shouldn't -change- those files, the host key should be generated very nearly at machine creation time and then never again. of course there's a chance that e.g. new sshd packages support curve 25519 keys and older ones didn't, so that key gets created at a reboot.. | 18:29 |
| patdk-wk | the issue is, if someone got in there, and was able to change those files | 18:34 |
| patdk-wk | they got root access | 18:34 |
| patdk-wk | and they could change the timestamps on those files also | 18:34 |
| zul | coreycb: we should be good now | 18:53 |
| === codedmart_ is now known as codedmart | ||
| ThiagoCMC | Hey guys! I'm facing a weird problem here with Ubuntu 16.04 HWE, I'm trying to enable 2 x 1G Hugepages, like this: "default_hugepagesz=1GB hugepagesz=1G hugepages=2" | 22:46 |
| ThiagoCMC | However, after "update-grup ; reboot", the /proc/meminfo shows "HugePages_Total: 121"! WTF... | 22:47 |
| ThiagoCMC | Server have 128G, what is preallocating those extra 119 x 1G hugepages? | 22:48 |
| ThiagoCMC | I just want 2, not 121! | 22:48 |
| keithzg | Hmm I swear I followed https://help.ubuntu.com/lts/serverguide/mail-filtering.html but even with "$sa_tag_level_deflt = -999;" I'm not seeing any spam info headers in emails being sent to and then delivered by the server in question, and nothing is showing up in the mail log to indicate Amavis is actually checking anything. | 22:53 |
| nacc | ThiagoCMC: `cat /proc/cmdline` and `cat /proc/meminfo` and `hugeadm --pool-list` in a pastebin? | 22:56 |
| sarnold | I've never seen hugeadm before; thanks nacc | 22:58 |
| nacc | sarnold: np, helped write it way back when :) | 22:59 |
| sarnold | nacc: ha! :D | 22:59 |
| nacc | there's also hugectl for manipulating programs with hugepages | 22:59 |
| nacc | a la numactl | 22:59 |
| * keithzg seems to have discovered that the problem was just that running `mail` from the mail server itself was bypassing things; via SMTP things seem fine, which is fair enough. Time to de-verbose the loglevel settings! | 23:01 | |
| sarnold | keithzg: woo :) | 23:02 |
| keithzg | sarnold: This is the thing I kindof love about the Linux side of my daily job; most of the time the solution is simple and I just need to stop and think what *I* am doing wrong ;) | 23:13 |
| sarnold | keithzg: hehe, that's not a bad place to be ;) | 23:14 |
| ThiagoCMC | nacc, here: https://paste.ubuntu.com/24263955/ | 23:17 |
| sarnold | 32 TB VmallocTotal -- oy :) | 23:19 |
| nacc | ThiagoCMC: hrm, `dmesg | grep Huge` ? | 23:19 |
| nacc | sarnold: i think that's the kernel default | 23:19 |
| nacc | sarnold: it's true on my lappy too | 23:19 |
| sarnold | awww. mine too. | 23:20 |
| sarnold | now I'm dissapointed again. | 23:20 |
| nacc | ThiagoCMC: i tentatively think it's this line: DirectMap1G: 131072000 kB | 23:20 |
| nacc | that's 125 1G pages (oddly not 121 :)) | 23:21 |
| ThiagoCMC | That's creepy! dmesg output: | 23:23 |
| ThiagoCMC | HugeTLB registered 1 GB page size, pre-allocated 2 pages | 23:23 |
| ThiagoCMC | :-( | 23:23 |
| nacc | ThiagoCMC: i think it's because you changed the default hugepages size | 23:24 |
| ThiagoCMC | Hmmm... How I did that? lol | 23:24 |
| ThiagoCMC | =P | 23:24 |
| nacc | hugepagesz= | 23:24 |
| nacc | err, default_hugepagesz= | 23:24 |
| ThiagoCMC | Hmm.. | 23:24 |
| nacc | that's probably not recommended | 23:25 |
| ThiagoCMC | Weird because I've used this before... just like this... | 23:25 |
| nacc | as it will also mean THP uses 1g pages by default | 23:25 |
| ThiagoCMC | I see... | 23:25 |
| ThiagoCMC | I'll try to remove that line | 23:25 |
| ThiagoCMC | *option | 23:25 |
| nacc | ThiagoCMC: on the same machine and kernel? | 23:25 |
| ThiagoCMC | same machine, previous kernel (4.4)... I also tried 4.4 couple hours ago, same result... | 23:27 |
| ThiagoCMC | I'm seeing that people set default_hugepagesz | 23:27 |
| ThiagoCMC | RedHat docs, DPDK docs... | 23:27 |
| nacc | ThiagoCMC: i mean, changing the default_hugepagesz to 1G is intended to be very intentional | 23:28 |
| ThiagoCMC | Ok... | 23:29 |
| nacc | rhel's kernel is also ancient, i assume | 23:30 |
| nacc | :) | 23:30 |
| nacc | and behavior changes | 23:30 |
| ThiagoCMC | I know... =) | 23:30 |
| ThiagoCMC | By removing that "default_huge...", it is different now! | 23:30 |
| nacc | ThiagoCMC: 2 ? or some other number? | 23:30 |
| ThiagoCMC | Look: https://paste.ubuntu.com/24264024/ | 23:31 |
| ThiagoCMC | Weird that "grep on meminfo" doesn't show the 1G ones... | 23:32 |
| ThiagoCMC | But I think I'm okay with it... | 23:32 |
| nacc | ThiagoCMC: right meminfo ony shows the default huge page size | 23:34 |
| nacc | ThiagoCMC: and then the directmap values | 23:34 |
| ThiagoCMC | Hmm... | 23:37 |
| ThiagoCMC | Thank you! | 23:37 |
| nacc | ThiagoCMC: yw! | 23:38 |
| drab | hi, anybody around that uses something like rundeck or stackstorm? | 23:38 |
| drab | I'm trying to figure out something that can allow me to "package" a set of commands and workflows to end over to operators | 23:39 |
| drab | basically a web frontend to ansible + a bunch of scripts | 23:39 |
| drab | stackstorm seems promising as it could do that and then more, but I'm wary of possible complication, it seems overall fairly new | 23:40 |
Generated by irclog2html.py 2.7 by Marius Gedminas - find it at mg.pov.lt!