[00:00] Has anybody seen issues with sssd causing your AWS c5 instance to freeze using either the 20180122 or 20180126 Ubuntu cloud images? I have narrowed it down to sssd.. if I disable that the box works fine but once I start it the box hard locks and CPU goes to 100% on the instance. No helpful logs in /var/log/kern.log, syslog, etc [00:01] Odd_Bloke: ahasenack --^ [00:01] nacc: I think the first failure may be correct. [00:01] I'm not sure why it worked before, but for non-native, a version string of '1-1' may be required instead of '1'. [00:01] rbasak: i was reading the wrong, tbh, every Source() will fail because dpkg-buildpackage will fail [00:01] reading the output wronng [00:02] rbasak: i'm not 100% confident our CI was testing it, tbh [00:02] I'm pretty confident I had it passing locally before submitting [00:02] sure [00:02] i'm lookoing onw [00:03] also it seems like we are usig some unknnown options in tox [00:03] i'll need to sync with powersj on that [00:03] ah yes, he's having to do it manually too [00:03] rbasak: see jenkins, i think powerjs is passing all the files by hand,b ecause the directory discovery doesn't work [00:04] i thik it looks for fiels with test_*.py *_test.py possibly [00:04] I believe I currently use something like gitubuntu/* [00:04] I believe so. It's tuneable [00:06] rbasak: right, but we haven't pullled most tests to their own file yet [00:08] rbasak: and actually, we'll need tox outside i'm realizing [00:08] because the scripts aren't in the snap [00:08] welll, they are, but they won't run and aren't exposed [00:09] nacc is there anything specific I could provide you all to help troubleshoot this sssd causing crashes on AWS c5 problem? [00:09] Perhaps the scripts should be in the snap [00:09] pfriel: i'd wait to see what they say [00:09] (and exposed) [00:09] rbasak: i mean they can be, but they don't need to be [00:09] nacc: k, thx [00:09] rbasak: and that's sort of orthogonal to this task (i can do it next) [00:10] Understood [00:10] powersj: ah you're using a more recent pytest than is in ubuntu? (hence the --cov option) [00:10] pfriel: sorry, haven't seen that [00:10] nacc: --cov comes from a plugin [00:10] ^^ but I also think I am using a version installed via pip and not in-distro [00:10] python3-pytest-cov package [00:11] i see [00:11] pfriel: suggest you file a bug about it [00:11] rbasak: powersj: ok, let me see if i can make that happen in the snap [00:11] I'd forgotten about that, sorry. [00:11] pfriel: I'm subscribed to sssd bugs, I'll get it when you file it. Please include all possible logs you have [00:11] * ahasenack -> bed [00:11] I'd like to have a ratchet on test coverage and on pylint warnings in the end [00:12] But we're perhaps not ready for that yet [00:13] i have to think of how to get that plugin [00:13] we eed to build it in our case [00:13] to match the python we use at runtime [00:13] ahasenack: ok, will do that.. thx [00:15] powersj: yeah, it's funny (not a bug), we instlal pytest 3.1.3 in our snap and you run with 3.2.1 because of pypi, i think [00:15] yeah [00:22] powersj: any idea? https://paste.ubuntu.com/26497978/ [00:26] So I believe tox installs things in a virtual env and you are calling /usr/bin/python which would be outside that env? [00:26] Can you paste your tox.ini [00:26] llet me tryu making that relative [00:26] ok yeah that fixed that error [00:26] and gave different ones :) === strive is now known as xenobite === xenobite is now known as xenobit === xenobit is now known as strive [00:45] rbasak: ok, i'm building it now, but i think have the tests running inside the snap and llinting outside of it [00:45] rbasak: the unittests in the snap for the scripts/ directory nneed some thinking still [00:46] rbasak: i would appreciate some time o the above failures, as we should resolvle them before landing the changes [00:48] oh i see why the scripts aren't workingn (tests). It's what you were saying powersj, you're acutaly using shell expandsion to get the files, not tox [00:49] powersj: and that's why we weren't using tox for the unit tests :) because we couldn't get it to find the tsts correctly [00:55] nacc: in tox.ini, try "[pytest]\npython_files = *.py" [00:57] Or py.test-3 -o 'python_files=*.py' [01:50] rbasak: ah i'll try that === led_ir23 is now known as led_ir22 [04:29] rbasak: woo, got the tests passing with the self-test app (excluding the scripts, which is really a separate thing) [04:30] i'll update the cleanups branch [04:32] for it to pass CI, we'll need to adjust the CI runner, though [04:33] rbasak: and we need to fix those 3 tests, of course [04:36] powersj: --^ fyi as well https://code.launchpad.net/~nacc/usd-importer/+git/usd-importer/+merge/336877 [04:37] powersj: so i think what will change in the pipeline is we'll add a stage that calls git-ubuntu.self-test [04:37] and drop the tox stage for now [04:39] rbasak: i'm trying to decide if we want to snap the scripts as their own subcommands? or just expose them directly as applications. The former means i can update setup.py to pull in our deps but they will all need rewriting. The latter is easier, but I need to think about how to ensure their deps get pulled in (since they ahve their own) [04:39] Mabye the latter for now, with a long term goal of the former? [04:53] Anyone know why the iscsi-target package has been broken for ages? [04:53] DirtyCajun: it's not needed since yakkety [04:53] DirtyCajun: so it depends on what you mean [04:54] (the driver has been mainlined) [04:54] timeout [04:54] ive never read an article about creating a target without that package [04:54] DirtyCajun: ... you could try and be a bit more verbose? [04:54] bestow your wisdom [04:54] XD [04:55] DirtyCajun: what version of ubuntu? [04:55] 16.04.3 [04:55] even the ubuntu page still says it [04:55] https://help.ubuntu.com/lts/serverguide/iscsi-initiator.html [04:56] that page does not mention the iscsi-target package at all? [04:56] iscsitarget is a dependancy [04:56] DirtyCajun: of what? [04:58] there are no reverse-dependencies on src:iscistarget in ubuntu xenial, afaict [04:58] i linked the initiator package not the target package thats my fault [04:58] *iscsitarget [04:58] and there is no package iscsi-target in ubuntu [04:59] https://www.server-world.info/en/note?os=Ubuntu_16.04&p=iscsi&f=2 [04:59] thats the newest help guide i can find [04:59] and still requires iscsitarget [05:00] DirtyCajun: if you are on 16.04.2 or later, you don't need any package [05:00] DirtyCajun: the iscsi target driver is in the kernel [05:00] so i mean the hwe stack thereof [05:00] if you are on the 4.4 kernel, then yes, you need the package [05:00] where is the config files located. same place? [05:00] DirtyCajun: what config file? [05:01] DirtyCajun: honestly, it seems easier to use tgt to server out iscsi disks [05:02] DirtyCajun: but yes, i think the userspace components of iscsitarget (note what i was referring to earlier was the iscsitarget-dkms stuff) is the same [07:02] Good morning [09:27] nacc: sounds reasonable [09:30] jamespage: FYI while proposed migration might need a bit all you need as base for the new virt stack is in 18.04 proposed [09:31] since monday actually [09:31] since we said end of january I hope that qualifies :-) [09:33] cpaelzer: awesome [09:36] jamespage: it might be hard for you to pick it up now, do you want another ping once it all migrated? [09:36] cpaelzer: I need to poke at the auto-backporter on the next two days anyway (as it looks like I need to rev some deps) [09:37] yeah you coud try eliminate all the backport hickups from the sources already [09:37] and know them all to make is a fast action once migration is complete === trekkie1701c_ is now known as trekkie1701c === galeido_ is now known as galeido === diddledan_ is now known as diddledan [11:55] coreycb: ok queens-staging is now good for debhelper compat 11 - sorting out the trunk testing PPA [11:55] coreycb: we'll need to stage the various pkgs into proposed as its very order dependent [12:50] jamespage: ok great [12:53] jamespage: everything should be uploaded now for b3. heat-dashboard is in the new queue. let me know if you have any thoughts on how to enable tests for heat-dashboard (see comment in d/rules). [13:14] hey y'all [13:15] im looking for something that will cache dns requests? [13:16] like, if it has no internal entry, it will check with google dns [13:16] also I have some questions about ubuntu cloud but the channel is invite only [13:20] MJCDoffice: Something like dnsmasq? [13:20] MJCDoffice: https://www.g-loaded.eu/2010/09/18/caching-nameserver-using-dnsmasq/ or actually systemd-resolved is "implements a caching and validating DNS/DNSSEC stub resolver" [13:20] Local caching resolver. [13:21] wait whats this about systemd-resolved [13:21] like im going to make this device into the networks main gateway [13:21] I just want to keep as much as I can internal traffic [13:21] even give it a 1TB drive haha [13:22] so not client only [13:22] yes then dnsmasq might be a good solution to try [13:22] awesome [13:22] and then there's obviously squid [13:22] for http [13:23] postfix for mail yeah? [13:25] can squid cache https stuff? [13:35] MJCDoffice: depends on your POV on breaking https by it being a man-in-the middle https://wiki.squid-cache.org/Features/HTTPS [13:37] cpaelzer, yeah I know about that but you're saying its already there in that [13:37] so that's perfect [13:37] not that im trying to spy [13:37] its more for like QoS [13:39] thb id like to figure out why I cant ping rpi [13:39] but can ping rpi's ip [13:39] the hostname is in caps but I dont think its case sensitive [13:39] but I tried RPI anyway [13:39] can dig resolve the IP [13:40] you can triy different nameservers with @nameserver [13:40] im on windows atm [13:41] im heading home but that's where the pi is anyway I think I need to enable ssl access [13:41] back shortly [13:58] jamespage: i'm taking a look at the python-coverage backport [14:49] ahasenack: in the libzstd Xenial MP, what's the purpose of: [14:49] - --devunversioned \ [14:49] please? [14:50] ahasenack: also I don't see the dh_auto_clean rule that you're adding in the SRUs present in Bionic. Is it required there? [15:15] coreycb: https://code.launchpad.net/~ubuntu-server-dev/ubuntu/+source/percona-xtradb-cluster-5.7/+git/percona-xtradb-cluster-5.7/+merge/337000 [15:46] jamespage: you guys packaged magnum? interesting [15:55] zul: we we've tended magnum - I'd not say we did the original packaging? [15:55] * jamespage looks [15:55] zul: yah it came via debian originally [17:14] jamespage: yeah thought so === mundus2018 is now known as mundus [18:06] Hi there, I created a file /etc/systemd/system/openvpn@.service.d/override.conf and I created a [Service] section with Restart=always and RestartSec=30 -- this is working well and makes sure OpenVPN client restarts even if there is a temporary issue with the server. However, every time I update openvpn with apt-get upgrade openvpn, it starts 2 openVPN processes and both keep restarting endlessly. [18:06] Any ideas why this happens and how to resolve? [18:16] hackeron: Are you running it foreground or background? I'd suspect you'd want foreground if you're having lennartd managing it. [18:16] Oh, you've modified the existing service. Looking. [18:22] hackeron: So, someone might have a better idea, but I'd assume it's having trouble tracking the forking daemon. I'd recommend replacing the entire unit file with a full override (systemctl edit --full) and making it a foreground service. [18:24] mason: but won't my edits be overwritten every upgrade? - my issue is this happens every time the openvpn package is updated [18:25] hackeron: That's the point of the complete replacement vs the drop-in. [18:25] Your drop-ins should persist past updates [18:28] mason: ah, hmm, so if I wanted to do this automatically on 100 servers, does creating a /etc/systemd/system/openvpn@.service file override the openvpn@.service provided by the package? [18:29] hackeron: It should do what you want as I understand it. [18:30] mason: ok, thank you for that, let me give that a try :) -- I think maybe changing from daemon to foreground is what I need [18:31] hackeron: So, yeah, after a quick test, that file is a complete replacement, vs a partial drop-in, which would show up as /etc/systemd/system/openvpn@.service.d/override.conf [18:32] but it's strange because /run/openvpn/timeline.is.pid is correct and matchs what is in top - so why isn't it being killed on upgrade. Do you see anything here that can cause it not to stop the process on upgrade online? < https://pastebin.com/5i1SwEni [18:32] maybe KillMode=mixed [18:32] Could I bother you to use bpaste.net instead? pastebin does funny things with javascript [18:32] mason: ah that's a nicer one, thank you :) < https://bpaste.net/show/d210e9020c18 [18:35] Hrm. I see this: "If you really want to delegate the shutdown from your main process, set KillMode=mixed. SIGTERM will be sent to the main process only." [18:36] OpenVPN responds well to being run in the foreground - I regularly run it that way, so that's the way I'd go in any event. Simpler. [18:37] mason: makes sense, I'll try that :) - do you run it through systemd or some other way? [18:37] hackeron: I script it and fire it up by hand as needed. [18:37] For my use, I want it to explicitly run only when I want it up. [18:38] My scripting fires it up, then sets up a couple related environmental things. [18:38] ah, ok [18:38] For quite a long time I ran it foreground in a screen session, and it was pretty reasonable. [19:45] mason: hmm, when I do that, systemctl edit --full openvpn@ shows my service, but when I reboot, systemd is still using the default openvpn@ service and still runs as daemon - any ideas? [19:46] hackeron: ... Um. I'd file a bug then. It shouldn't be ignoring drop-ins or override configs. :/ [19:47] hackeron: can you share "systemctl cat openvpn@" via pastebin ? [19:47] mason: yeh, strange, so when I do systemctl edit --full openvpn@ it shows my file - when I do systemctl status openvpn@ it shows Loaded: loaded (/lib/systemd/system/openvpn@.service; disabled; vendor preset: enabled) -- grr [19:48] hackeron: Did you restart or say daemon-reload ? [19:48] mason: yep, even tried reboot [19:48] sdeziel: https://bpaste.net/show/aab5a207a75e [19:48] hrm [19:49] hackeron: I don't think you need to specify/interact with a PIDfile if you're running it in the foreground. [19:49] But I'm not systemd expert. [19:49] s/not/no/ [19:49] hackeron: "systemctl cat openvpn@xancloud.com" please [19:50] sdeziel: https://bpaste.net/show/5ccef9cf8432 [19:50] mason: sure, I can remove that - but the issue currently is systemctl is not using my service file at all :( [19:51] hackeron: for some reason, the @xancloud.com instance uses the /lib/... unit [19:51] sdeziel: yeh, but why, the service file is /lib/systemd/system/openvpn@.service and I'm editing openvpn@ - any ideas why that is happening? [19:51] sudo systemctl edit --full openvpn@foo is a thing [19:52] might be worth trying, even if it shouldn't be necessary [19:52] And I'd think about trimming out the PIDfile interactions. [19:52] hackeron: I don't know, editing the generic one (openvpn@) should be enough to apply to all the specific instances [19:52] Editing openvpn@foo creates an override for that instance. [19:53] Worth trying. [19:53] hackeron: that said, I don't like "edit --full" ;) [19:53] sdeziel: He can't remove bits and is prone to have other stuff bleed in if he just does a drop-in. [19:54] hackeron: the original problem you are trying to solve is apt upgrades leading to multiple instances of openvpn@xancloud.com, right? [19:54] mason: changing the unit type feels like voiding the warranty to me [19:55] Thank the gods for systemd, though - everything was so boringly transparent and obvious before. systemd lends a pleasing aura of magic to what used to be mundane, every-day tasks. :) [19:55] :) [19:55] sdeziel: Yeah, I think that's a foregone conclusion. [19:55] sdeziel: sort of, my problem is when I make openVPN auto restart, by creating /etc/systemd/system/openvpn@.service.d/override.conf and adding Restart=always - then every upgrade, systemd files to kill the openvpn process. So I need to create my own service file probably [19:56] hackeron: For real, try edit --full openvpn@yourthing [19:56] I for one welcome systemd's magic tough, the power of seccomp and namespaces is now available easily ;) [19:56] I want to know if it works. [19:56] mason: I have 5 mythings and they can change, so I'd like to avoid doing that if at all possible [19:56] hackeron: don't use Restart=always [19:56] hackeron: Five mythings is still one automation config. :P [19:56] hackeron: this mode will start it back even when you manually asked to stop it (checking man page) [19:57] sdeziel: I have 300+ servers out there and sometimes the server has a glitch or there's some other problem and they must absolutely always, no matter what, try to get back on the VPN, so I can connect to them remotely -- so what else is there except a Restart=always? [19:57] As for that, Restart=on-failure ? [19:57] hackeron: Restart=on-abnormal [19:57] or on-failure [19:57] hackeron: man 5 systemd.service [19:57] hackeron: You're going to make me test the instanced override myself, aren't you? I can feel it. [19:57] hackeron: there is a nice table explaining the various Restart= modes [19:58] mason: sdeziel: that's what I had before and it would fail to start up sometimes, if say a hard drive was read only or something, or there was some condition error and the network was being restarted or something which stopped the vpn -- I want it to try to restart always, no matter what [19:58] hackeron: Restart=always is what played you trick during apt upgrades, pretty sure of that [19:58] likely [19:58] sdeziel: yeh, for sure, but how else do I ensure it restarts no matter what? [19:59] hackeron: Restart=on-failure [19:59] and monitoring [19:59] sdeziel: what's the deal with Restart=always? [19:59] coreycb: ping [20:00] coreycb: can you please merge and upload http://paste.ubuntu.com/26502243/ please [20:00] mason: what do you mean by and monitoring? [20:00] hackeron: I mean, you've got monitoring of some sort already. Check for missing/down VPN, and if Restart=on-failure isn't enough, page someone. [20:00] hackeron: Restart=on-failure seems like the best you can do without inflicting you problems on apt upgrades [20:00] mason: my goal is I want to make sure nothing on the system can accidentally stop the vpn, no matter what [20:01] sarnold: dunno, that's a weird mode if you ask me ;) [20:01] sarnold: you manually stop a service and systemd starts it behind your back [20:01] sdeziel: alright :) hehe, the table can describe the intentions, but I figured if you'd fought it over something, that kind of knowledge doesn't show up in tables. hehe. [20:01] sdeziel: aha. yeah. [20:02] mason: the system can be at the top of a mountain or the bottom of the ocean, there is no page someone, it needs to never ever ever go down and if something or someone takes it down, it needs to always restart :) -- I think my intent is solved by editing openvpn@.service -- but question is, why is my edit ignored? [20:03] mason: because something like networking or multiuser target or a bunch of other things can stop the service, I don't want that to ever happen [20:03] hackeron: It clearly doesn't match the instanced service. [20:03] sarnold: I never had to fight with it [20:03] hackeron: with your Restart=always could you add RestartSec=X to impose a delay to cover the package-upgrade cycle ? [20:03] TJ-: I have RestartSec=30 already [20:03] hackeron: FYI, I run my openvpn instances with Restart=on-failure and it works well here [20:03] hackeron: Try the edit --full foo@bar and see if that matches, and there's your answer [20:04] mason: What do you mean? systemctl status openvpn@xancloud.com shows /lib/systemd/system/openvpn@.service -- so it's openvpn@.service, is it not? [20:05] hackeron: Spock says, “An ancestor of mine maintained that if you eliminate the impossible, whatever remains, however improbable, must be the solution.” [20:05] *snort* [20:06] hackeron: I'm wondering if the issue here is not systemd so much as the openvpn pre/postinst scripts. [20:08] Zul: heya, sure [20:09] coreycb: merci buckets [20:09] coreycb: there is probably going to be more btw [20:10] TJ-: no, just checked - changing openVPN from daemon to foreground fixes it -- now the question is why isn't editing openvpn@ working, hmm [20:10] mason: yes, editing openvpn@xancloud.com explicitly works - now need to figure out how to make it apply to all vpn configs, hmmm [20:13] hackeron: If you're using Ansible or Puppet or something, you should be able to base it on a template and make it fairly safe. === Guest74281 is now known as karstensrage === karstensrage is now known as Guest68163 [20:15] mason: I guess - but I wonder what's wrong with systemd that it's ignoring edit of openvpn@ === Guest68163 is now known as karstensrage [20:15] hackeron: I can help with that: http://without-systemd.org/ [20:15] Take your pick. :P [20:16] I wish you could do a rolling eyes smiley in text :) [20:16] @ @ [20:16] \_/ [20:17] hackeron: I wonder, maybe you have to make use of the template@'s %i instance variable in some way? Match= or somethig? [20:58] nacc: I believe you were the one I reached out to yesterday about sssd causing lockups on AWS c5/m5 instances. You pinged someone else who requested I create a bug report.. I have now done that: https://bugs.launchpad.net/ubuntu/+source/sssd/+bug/1746806. I don't remember who it was you pinged though so if you could bump that person it would be much appreciated. thanks [20:58] Launchpad bug 1746806 in sssd (Ubuntu) "sssd appears to crash AWS c5 and m5 instances, cause 100% CPU" [Undecided,New] [20:58] ahasenack: --^ [20:58] (i think) [20:58] nacc: ty sir [20:58] pfriel: yw, thanks for filing the bug report! [20:59] no prob.. wish I could have provided more details. I have sunk my last few days trying to figure this thing out with no luck. Hopefully you all can provide some tips on getting more useful debug info [22:21] cpaelzer: let's sync on kopanocore, i would like to get it updated correctly to match debian (which has a php-mapi), i thikn you're right onn the breaks/replaces (which will successfully upgrade artful and existing bionic) and then we can drop that once 18.10 opens [22:33] cpaelzer: https://code.launchpad.net/~nacc/ubuntu/+source/kopanocore/+git/kopanocore/+merge/337021 [23:13] cpaelzer: need to push some changes on top, fyi, so it builds (libical migrated underneath) [23:17] cpaelzer: i assume you're already o nit, but qemu-system-x86 is uninstallable i bionic-proposed? [23:17] cpaelzer: that blocks libguestfs which in turn blocks php7.2 :)