/srv/irclogs.ubuntu.com/2018/02/01/#ubuntu-server.txt

pfrielHas anybody seen issues with sssd causing your AWS c5 instance to freeze using either the 20180122 or 20180126 Ubuntu cloud images?  I have narrowed it down to sssd.. if I disable that the box works fine but once I start it the box hard locks and CPU goes to 100% on the instance.  No helpful logs in /var/log/kern.log, syslog, etc00:00
naccOdd_Bloke: ahasenack --^00:01
rbasaknacc: I think the first failure may be correct.00:01
rbasakI'm not sure why it worked before, but for non-native, a version string of '1-1' may be required instead of '1'.00:01
naccrbasak: i was reading the wrong, tbh, every Source() will fail because dpkg-buildpackage will fail00:01
naccreading the output wronng00:01
naccrbasak: i'm not 100% confident our CI was testing it, tbh00:02
rbasakI'm pretty confident I had it passing locally before submitting00:02
naccsure00:02
nacci'm lookoing onw00:02
naccalso it seems like we are usig some unknnown options in tox00:03
nacci'll need to sync with powersj on that00:03
naccah yes, he's having to do it manually too00:03
naccrbasak: see jenkins, i think powerjs is passing all the files by hand,b ecause the directory discovery doesn't work00:03
nacci thik it looks for fiels with test_*.py *_test.py possibly00:04
powersjI believe I currently use something like gitubuntu/*00:04
rbasakI believe so. It's tuneable00:04
naccrbasak: right, but we haven't pullled most tests to their own file yet00:06
naccrbasak: and actually, we'll need tox outside i'm realizing00:08
naccbecause the scripts aren't in the snap00:08
naccwelll, they are, but they won't run and aren't exposed00:08
pfrielnacc is there anything specific I could provide you all to help troubleshoot this sssd causing crashes on AWS c5 problem?00:09
rbasakPerhaps the scripts should be in the snap00:09
naccpfriel: i'd wait to see what they say00:09
rbasak(and exposed)00:09
naccrbasak: i mean they can be, but they don't need to be00:09
pfrielnacc: k, thx00:09
naccrbasak: and that's sort of orthogonal to this task (i can do it next)00:09
rbasakUnderstood00:10
naccpowersj: ah you're using a more recent pytest than is in ubuntu? (hence the --cov option)00:10
ahasenackpfriel: sorry, haven't seen that00:10
rbasaknacc: --cov comes from a plugin00:10
powersj^^ but I also think I am using a version installed via pip and not in-distro00:10
rbasakpython3-pytest-cov package00:10
nacci see00:11
ahasenackpfriel: suggest you file a bug about it00:11
naccrbasak: powersj: ok, let me see if i can make that happen in the snap00:11
rbasakI'd forgotten about that, sorry.00:11
ahasenackpfriel: I'm subscribed to sssd bugs, I'll get it when you file it. Please include all possible logs you have00:11
* ahasenack -> bed00:11
rbasakI'd like to have a ratchet on test coverage and on pylint warnings in the end00:11
rbasakBut we're perhaps not ready for that yet00:12
nacci have to think of how to get that plugin00:13
naccwe eed to build it in our case00:13
naccto match the python we use at runtime00:13
pfrielahasenack: ok, will do that.. thx00:13
naccpowersj: yeah, it's funny (not a bug), we instlal pytest 3.1.3 in our snap and you run with 3.2.1 because of pypi, i think00:15
powersjyeah00:15
naccpowersj: any idea? https://paste.ubuntu.com/26497978/00:22
powersjSo I believe tox installs things in a virtual env and you are calling /usr/bin/python which would be outside that env?00:26
powersjCan you paste your tox.ini00:26
naccllet me tryu making that relative00:26
naccok yeah that fixed that error00:26
naccand gave different ones :)00:26
=== strive is now known as xenobite
=== xenobite is now known as xenobit
=== xenobit is now known as strive
naccrbasak: ok, i'm building it now, but i think have the tests running inside the snap and llinting outside of it00:45
naccrbasak: the unittests in the snap for the scripts/ directory nneed some thinking still00:45
naccrbasak: i would appreciate some time o the above failures, as we should resolvle them before landing the changes00:46
naccoh i see why the scripts aren't workingn (tests). It's what you were saying powersj, you're acutaly using shell expandsion to get the files, not tox00:48
naccpowersj: and that's why we weren't using tox for the unit tests :) because we couldn't get it to find the tsts correctly00:49
rbasaknacc: in tox.ini, try "[pytest]\npython_files = *.py"00:55
rbasakOr py.test-3 -o 'python_files=*.py'00:57
naccrbasak: ah i'll try that01:50
=== led_ir23 is now known as led_ir22
naccrbasak: woo, got the tests passing with the self-test app (excluding the scripts, which is really a separate thing)04:29
nacci'll update the cleanups branch04:30
naccfor it to pass CI, we'll need to adjust the CI runner, though04:32
naccrbasak: and we need to fix those 3 tests, of course04:33
naccpowersj: --^ fyi as well https://code.launchpad.net/~nacc/usd-importer/+git/usd-importer/+merge/33687704:36
naccpowersj: so i think what will change in the pipeline is we'll add a stage that calls git-ubuntu.self-test04:37
naccand drop the tox stage for now04:37
naccrbasak: i'm trying to decide if we want to snap the scripts as their own subcommands? or just expose them directly as applications. The former means i can update setup.py to pull in our deps but they will all need rewriting. The latter is easier, but I need to think about how to ensure their deps get pulled in (since they ahve their own)04:39
naccMabye the latter for now, with a long term goal of the former?04:39
DirtyCajunAnyone know why the iscsi-target package has been broken for ages?04:53
naccDirtyCajun: it's not needed since yakkety04:53
naccDirtyCajun: so it depends on what you mean04:53
nacc(the driver has been mainlined)04:54
DirtyCajuntimeout04:54
DirtyCajunive never read an article about creating a target without that package04:54
naccDirtyCajun: ... you could try and be a bit more verbose?04:54
DirtyCajunbestow your wisdom04:54
DirtyCajunXD04:54
naccDirtyCajun: what version of ubuntu?04:55
DirtyCajun16.04.304:55
DirtyCajuneven the ubuntu page still says it04:55
DirtyCajunhttps://help.ubuntu.com/lts/serverguide/iscsi-initiator.html04:55
naccthat page does not mention the iscsi-target package at all?04:56
DirtyCajuniscsitarget is a dependancy04:56
naccDirtyCajun: of what?04:56
naccthere are no reverse-dependencies on src:iscistarget in ubuntu xenial, afaict04:58
DirtyCajuni linked the initiator package not the target package thats my fault04:58
nacc*iscsitarget04:58
naccand there is no package iscsi-target in ubuntu04:58
DirtyCajunhttps://www.server-world.info/en/note?os=Ubuntu_16.04&p=iscsi&f=204:59
DirtyCajunthats the newest help guide i can find04:59
DirtyCajunand still requires iscsitarget04:59
naccDirtyCajun: if you are on 16.04.2 or later, you don't need any package05:00
naccDirtyCajun: the iscsi target driver is in the kernel05:00
naccso i mean the hwe stack thereof05:00
naccif you are on the 4.4 kernel, then yes, you need the package05:00
DirtyCajunwhere is the config files located. same place?05:00
naccDirtyCajun: what config file?05:00
naccDirtyCajun: honestly, it seems easier to use tgt to server out iscsi disks05:01
naccDirtyCajun: but yes, i think the userspace components of iscsitarget (note what i was referring to earlier was the iscsitarget-dkms stuff) is the same05:02
lordievaderGood morning07:02
rbasaknacc: sounds reasonable09:27
cpaelzerjamespage: FYI while proposed migration might need a bit all you need as base for the new virt stack is in 18.04 proposed09:30
cpaelzersince monday actually09:31
cpaelzersince we said end of january I hope that qualifies :-)09:31
jamespagecpaelzer: awesome09:33
cpaelzerjamespage: it might be hard for you to pick it up now, do you want another ping once it all migrated?09:36
jamespagecpaelzer: I need to poke at the auto-backporter on the next two days anyway (as it looks like I need to rev some deps)09:36
cpaelzeryeah you coud try eliminate all the backport hickups from the sources already09:37
cpaelzerand know them all to make is a fast action once migration is complete09:37
=== trekkie1701c_ is now known as trekkie1701c
=== galeido_ is now known as galeido
=== diddledan_ is now known as diddledan
jamespagecoreycb: ok queens-staging is now good for debhelper compat 11 - sorting out the trunk testing PPA11:55
jamespagecoreycb: we'll need to stage the various pkgs into proposed as its very order dependent11:55
coreycbjamespage: ok great12:50
coreycbjamespage: everything should be uploaded now for b3. heat-dashboard is in the new queue. let me know if you have any thoughts on how to enable tests for heat-dashboard (see comment in d/rules).12:53
MJCDofficehey y'all13:14
MJCDofficeim looking for something that will cache dns requests?13:15
MJCDofficelike, if it has no internal entry, it will check with google dns13:16
MJCDofficealso I have some questions about ubuntu cloud but the channel is invite only13:16
lordievaderMJCDoffice: Something like dnsmasq?13:20
cpaelzerMJCDoffice: https://www.g-loaded.eu/2010/09/18/caching-nameserver-using-dnsmasq/ or actually systemd-resolved is "implements a caching and validating DNS/DNSSEC stub resolver"13:20
lordievaderLocal caching resolver.13:20
MJCDofficewait whats this about systemd-resolved13:21
MJCDofficelike im going to make this device into the networks main gateway13:21
MJCDofficeI just want to keep as much as I can internal traffic13:21
MJCDofficeeven give it a 1TB drive haha13:21
cpaelzerso not client only13:22
cpaelzeryes then dnsmasq might be a good solution to try13:22
MJCDofficeawesome13:22
MJCDofficeand then there's obviously squid13:22
MJCDofficefor http13:22
MJCDofficepostfix for mail yeah?13:23
MJCDofficecan squid cache https stuff?13:25
cpaelzerMJCDoffice: depends on your POV on breaking https by it being a man-in-the middle https://wiki.squid-cache.org/Features/HTTPS13:35
MJCDofficecpaelzer, yeah I know about that but you're saying its already there in that13:37
MJCDofficeso that's perfect13:37
MJCDofficenot that im trying to spy13:37
MJCDofficeits more for like QoS13:37
MJCDofficethb id like to figure out why I cant ping rpi13:39
MJCDofficebut can ping rpi's ip13:39
MJCDofficethe hostname is in caps but I dont think its case sensitive13:39
MJCDofficebut I tried RPI anyway13:39
cpaelzercan dig resolve the IP13:39
cpaelzeryou can triy different nameservers with @nameserver13:40
MJCDofficeim on windows atm13:40
MJCDofficeim heading home but that's where the pi is anyway I think I need to enable ssl access13:41
MJCDofficeback shortly13:41
coreycbjamespage: i'm taking a look at the python-coverage backport13:58
rbasakahasenack: in the libzstd Xenial MP, what's the purpose of:14:49
rbasak-                               --devunversioned \14:49
rbasakplease?14:49
rbasakahasenack: also I don't see the dh_auto_clean rule that you're adding in the SRUs present in Bionic. Is it required there?14:50
jamespagecoreycb: https://code.launchpad.net/~ubuntu-server-dev/ubuntu/+source/percona-xtradb-cluster-5.7/+git/percona-xtradb-cluster-5.7/+merge/33700015:15
zuljamespage: you guys packaged magnum? interesting15:46
jamespagezul: we we've tended magnum - I'd not say we did the original packaging?15:55
* jamespage looks15:55
jamespagezul: yah it came via debian originally15:55
zuljamespage: yeah thought so17:14
=== mundus2018 is now known as mundus
hackeronHi there, I created a file /etc/systemd/system/openvpn@.service.d/override.conf and I created a [Service] section with Restart=always and RestartSec=30 -- this is working well and makes sure OpenVPN client restarts even if there is a temporary issue with the server. However, every time I update openvpn with apt-get upgrade openvpn, it starts 2 openVPN processes and both keep restarting endlessly.18:06
hackeronAny ideas why this happens and how to resolve?18:06
masonhackeron: Are you running it foreground or background? I'd suspect you'd want foreground if you're having lennartd managing it.18:16
masonOh, you've modified the existing service. Looking.18:16
masonhackeron: So, someone might have a better idea, but I'd assume it's having trouble tracking the forking daemon. I'd recommend replacing the entire unit file with a full override (systemctl edit --full) and making it a foreground service.18:22
hackeronmason: but won't my edits be overwritten every upgrade? - my issue is this happens every time the openvpn package is updated18:24
masonhackeron: That's the point of the complete replacement vs the drop-in.18:25
masonYour drop-ins should persist past updates18:25
hackeronmason: ah, hmm, so if I wanted to do this automatically on 100 servers, does creating a /etc/systemd/system/openvpn@.service file override the openvpn@.service provided by the package?18:28
masonhackeron: It should do what you want as I understand it.18:29
hackeronmason: ok, thank you for that, let me give that a try :) -- I think maybe changing from daemon to foreground is what I need18:30
masonhackeron: So, yeah, after a quick test, that file is a complete replacement, vs a partial drop-in, which would show up as /etc/systemd/system/openvpn@.service.d/override.conf18:31
hackeronbut it's strange because /run/openvpn/timeline.is.pid is correct and matchs what is in top - so why isn't it being killed on upgrade. Do you see anything here that can cause it not to stop the process on upgrade online? < https://pastebin.com/5i1SwEni18:32
hackeronmaybe KillMode=mixed18:32
masonCould I bother you to use bpaste.net instead? pastebin does funny things with javascript18:32
hackeronmason: ah that's a nicer one, thank you :) < https://bpaste.net/show/d210e9020c1818:32
masonHrm. I see this: "If you really want to delegate the shutdown from your main process, set KillMode=mixed. SIGTERM will be sent to the main process only."18:35
masonOpenVPN responds well to being run in the foreground - I regularly run it that way, so that's the way I'd go in any event. Simpler.18:36
hackeronmason: makes sense, I'll try that :) - do you run it through systemd or some other way?18:37
masonhackeron: I script it and fire it up by hand as needed.18:37
masonFor my use, I want it to explicitly run only when I want it up.18:37
masonMy scripting fires it up, then sets up a couple related environmental things.18:38
hackeronah, ok18:38
masonFor quite a long time I ran it foreground in a screen session, and it was pretty reasonable.18:38
hackeronmason: hmm, when I do that, systemctl edit --full openvpn@ shows my service, but when I reboot, systemd is still using the default openvpn@ service and still runs as daemon - any ideas?19:45
masonhackeron: ... Um. I'd file a bug then. It shouldn't be ignoring drop-ins or override configs. :/19:46
sdezielhackeron: can you share "systemctl cat openvpn@" via pastebin ?19:47
hackeronmason: yeh, strange, so when I do systemctl edit --full openvpn@ it shows my file - when I do systemctl status openvpn@ it shows Loaded: loaded (/lib/systemd/system/openvpn@.service; disabled; vendor preset: enabled) -- grr19:47
masonhackeron: Did you restart or say daemon-reload ?19:48
hackeronmason: yep, even tried reboot19:48
hackeronsdeziel: https://bpaste.net/show/aab5a207a75e19:48
masonhrm19:48
masonhackeron: I don't think you need to specify/interact with a PIDfile if you're running it in the foreground.19:49
masonBut I'm not systemd expert.19:49
masons/not/no/19:49
sdezielhackeron: "systemctl cat openvpn@xancloud.com" please19:49
hackeronsdeziel: https://bpaste.net/show/5ccef9cf843219:50
hackeronmason: sure, I can remove that - but the issue currently is systemctl is not using my service file at all :(19:50
sdezielhackeron: for some reason, the @xancloud.com instance uses the /lib/... unit19:51
hackeronsdeziel: yeh, but why, the service file is /lib/systemd/system/openvpn@.service and I'm editing openvpn@ - any ideas why that is happening?19:51
masonsudo systemctl edit --full openvpn@foo is a thing19:51
masonmight be worth trying, even if it shouldn't be necessary19:52
masonAnd I'd think about trimming out the PIDfile interactions.19:52
sdezielhackeron: I don't know, editing the generic one (openvpn@) should be enough to apply to all the specific instances19:52
masonEditing openvpn@foo creates an override for that instance.19:52
masonWorth trying.19:53
sdezielhackeron: that said, I don't like "edit --full" ;)19:53
masonsdeziel: He can't remove bits and is prone to have other stuff bleed in if he just does a drop-in.19:53
sdezielhackeron: the original problem you are trying to solve is apt upgrades leading to multiple instances of openvpn@xancloud.com, right?19:54
sdezielmason: changing the unit type feels like voiding the warranty to me19:54
masonThank the gods for systemd, though - everything was so boringly transparent and obvious before. systemd lends a pleasing aura of magic to what used to be mundane, every-day tasks. :)19:55
sarnold:)19:55
masonsdeziel: Yeah, I think that's a foregone conclusion.19:55
hackeronsdeziel: sort of, my problem is when I make openVPN auto restart, by creating /etc/systemd/system/openvpn@.service.d/override.conf and adding Restart=always - then every upgrade, systemd files to kill the openvpn process. So I need to create my own service file probably19:55
masonhackeron: For real, try edit --full openvpn@yourthing19:56
sdezielI for one welcome systemd's magic tough, the power of seccomp and namespaces is now available easily ;)19:56
masonI want to know if it works.19:56
hackeronmason: I have 5 mythings and they can change, so I'd like to avoid doing that if at all possible19:56
sdezielhackeron: don't use Restart=always19:56
masonhackeron: Five mythings is still one automation config. :P19:56
sdezielhackeron: this mode will start it back even when you manually asked to stop it (checking man page)19:56
hackeronsdeziel: I have 300+ servers out there and sometimes the server has a glitch or there's some other problem and they must absolutely always, no matter what, try to get back on the VPN, so I can connect to them remotely -- so what else is there except a Restart=always?19:57
masonAs for that, Restart=on-failure ?19:57
sdezielhackeron: Restart=on-abnormal19:57
sdezielor on-failure19:57
sdezielhackeron: man 5 systemd.service19:57
masonhackeron: You're going to make me test the instanced override myself, aren't you? I can feel it.19:57
sdezielhackeron: there is a nice table explaining the various Restart= modes19:57
hackeronmason: sdeziel: that's what I had before and it would fail to start up sometimes, if say a hard drive was read only or something, or there was some condition error and the network was being restarted or something which stopped the vpn -- I want it to try to restart always, no matter what19:58
sdezielhackeron: Restart=always is what played you trick during apt upgrades, pretty sure of that19:58
masonlikely19:58
hackeronsdeziel: yeh, for sure, but how else do I ensure it restarts no matter what?19:58
masonhackeron: Restart=on-failure19:59
masonand monitoring19:59
sarnoldsdeziel: what's the deal with Restart=always?19:59
zulcoreycb:  ping19:59
zulcoreycb: can you please merge and upload http://paste.ubuntu.com/26502243/ please20:00
hackeronmason: what do you mean by and monitoring?20:00
masonhackeron: I mean, you've got monitoring of some sort already. Check for missing/down VPN, and if Restart=on-failure isn't enough, page someone.20:00
sdezielhackeron: Restart=on-failure seems like the best you can do without inflicting you problems on apt upgrades20:00
hackeronmason: my goal is I want to make sure nothing on the system can accidentally stop the vpn, no matter what20:00
sdezielsarnold: dunno, that's a weird mode if you ask me ;)20:01
sdezielsarnold: you manually stop a service and systemd starts it behind your back20:01
sarnoldsdeziel: alright :) hehe, the table can describe the intentions, but I figured if you'd fought it over something, that kind of knowledge doesn't show up in tables. hehe.20:01
sarnoldsdeziel: aha. yeah.20:01
hackeronmason: the system can be at the top of a mountain or the bottom of the ocean, there is no page someone, it needs to never ever ever go down and if something or someone takes it down, it needs to always restart :) -- I think my intent is solved by editing openvpn@.service -- but question is, why is my edit ignored?20:02
hackeronmason: because something like networking or multiuser target or a bunch of other things can stop the service, I don't want that to ever happen20:03
masonhackeron: It clearly doesn't match the instanced service.20:03
sdezielsarnold: I never had to fight with it20:03
TJ-hackeron: with your Restart=always could you add RestartSec=X to impose a delay to cover the package-upgrade cycle ?20:03
hackeronTJ-: I have RestartSec=30 already20:03
sdezielhackeron: FYI, I run my openvpn instances with Restart=on-failure and it works well here20:03
masonhackeron: Try the edit --full foo@bar and see if that matches, and there's your answer20:03
hackeronmason: What do you mean? systemctl status openvpn@xancloud.com shows /lib/systemd/system/openvpn@.service -- so it's openvpn@.service, is it not?20:04
masonhackeron: Spock says, “An ancestor of mine maintained that if you eliminate the impossible, whatever remains, however improbable, must be the solution.”20:05
sarnold*snort*20:05
TJ-hackeron: I'm wondering if the issue here is not systemd so much as the openvpn pre/postinst scripts.20:06
coreycbZul: heya, sure20:08
zulcoreycb: merci buckets20:09
zulcoreycb: there is probably going to be more btw20:09
hackeronTJ-: no, just checked - changing openVPN from daemon to foreground fixes it -- now the question is why isn't editing openvpn@ working, hmm20:10
hackeronmason: yes, editing openvpn@xancloud.com explicitly works - now need to figure out how to make it apply to all vpn configs, hmmm20:10
masonhackeron: If you're using Ansible or Puppet or something, you should be able to base it on a template and make it fairly safe.20:13
=== Guest74281 is now known as karstensrage
=== karstensrage is now known as Guest68163
hackeronmason: I guess - but I wonder what's wrong with systemd that it's ignoring edit of openvpn@20:15
=== Guest68163 is now known as karstensrage
masonhackeron: I can help with that: http://without-systemd.org/20:15
masonTake your pick. :P20:15
hackeronI wish you could do a rolling eyes smiley in text :)20:16
TJ-@ @20:16
TJ-\_/20:16
TJ-hackeron: I wonder, maybe you have to make use of the template@'s %i instance variable in some way? Match= or somethig?20:17
pfrielnacc: I believe you were the one I reached out to yesterday about sssd causing lockups on AWS c5/m5 instances. You pinged someone else who requested I create a bug report.. I have now done that: https://bugs.launchpad.net/ubuntu/+source/sssd/+bug/1746806.  I don't remember who it was you pinged though so if you could bump that person it would be much appreciated.  thanks20:58
ubottuLaunchpad bug 1746806 in sssd (Ubuntu) "sssd appears to crash AWS c5 and m5 instances, cause 100% CPU" [Undecided,New]20:58
naccahasenack: --^20:58
nacc(i think)20:58
pfrielnacc: ty sir20:58
naccpfriel: yw, thanks for filing the bug report!20:58
pfrielno prob.. wish I could have provided more details. I have sunk my last few days trying to figure this thing out with no luck.  Hopefully you all can provide some tips on getting more useful debug info20:59
nacccpaelzer: let's sync on kopanocore, i would like to get it updated correctly to match debian (which has a php-mapi), i thikn you're right onn the breaks/replaces (which will successfully upgrade artful and existing bionic) and then we can drop that once 18.10 opens22:21
nacccpaelzer: https://code.launchpad.net/~nacc/ubuntu/+source/kopanocore/+git/kopanocore/+merge/33702122:33
nacccpaelzer: need to push some changes on top, fyi, so it builds (libical migrated underneath)23:13
nacccpaelzer: i assume you're already o nit, but qemu-system-x86 is uninstallable i bionic-proposed?23:17
nacccpaelzer: that blocks libguestfs which in turn blocks php7.2 :)23:17

Generated by irclog2html.py 2.7 by Marius Gedminas - find it at mg.pov.lt!