[00:00] <blackflow> but it's not installed, even on the server that does create /run/motd.dynamic. that'st he key here, what creates that file
[00:01] <rbasak> pam_motd creates it
[00:01] <blackflow> 50-motd-news is apparently creating /var/cache/motd-news, but not /run/motd.dynamic. both are straight files, not a symlink or something
[00:01] <rbasak> From /etc/pam.d/login and /etc/pam.d/sshd
[00:02] <blackflow> no it doesn't _create_ it
[00:02] <rbasak> See pam_motd(8)
[00:02] <blackflow> it _displays_ existing file, doesnt' trigger its creation
[00:03] <rbasak> No. Read the man page.
[00:03] <rbasak> noupdate: Don't run the scripts in /etc/update-motd.d to refresh the motd file.
[00:03] <rbasak> IOW, don't specify that, and it will update.
[00:04] <blackflow> the server that has /run/motd.dynamic has that too
[00:04] <blackflow> so it's not it
[00:06] <blackflow> do I need anything special after changing a pam.d/ config? restart something?
[00:07] <sarnold> I think those changes are picked up by services without restarts
[00:10] <blackflow> welp that didn't change anything. I removed "noupdate" both from login and sshd  pam.d modules. Still no /run/motd.dynamic and no MOTD on login via ssh.
[00:12] <sarnold> try fatrace or perf trace or something similar? strace might not be fantastic ..
[00:12] <blackflow> trace what?
[00:12] <sarnold> whichever service you're using for testing
[00:13] <blackflow> I'm logging in via ssh
[00:13] <cryptodan_mobile> Wouldnt motd go in /etc
[00:13] <blackflow> on the server installed from server ISO I get motd. On the server I installed from debootstrap, I don't.
[00:14] <rbasak> Are you using -Snone?
[00:14] <blackflow> and I tried forcign a static motd even, not /run/motd.dynamic,  it doesn't show
[00:14] <rbasak> Check "sudo login -f root"
[00:14] <blackflow> that shows the motd yes
[00:15] <blackflow> (twice, even)
[00:15] <rbasak> What about -Snone?
[00:15] <blackflow> what is -Snone ?
[00:15] <rbasak> To ssh
[00:15] <rbasak> To make sure you aren't sharing an existing connection
[00:16] <sarnold> hahahahaha
[00:16] <sarnold> oh man
[00:16] <blackflow> there's no other connection and yeah I tried just with -Snone (on the client side), still no motd.
[00:17] <rbasak> You messed with your PAM configuration, didn't you?
[00:17] <rbasak> For ssh but not for login?
[00:17] <blackflow> I also don't have that local ~/.hushwhatever   file
[00:18] <blackflow> rbasak: just now to force a static /etc/motd since default configured /run/motd.dynamic   doesn't exist
[00:18] <blackflow> I touched nothing else in pam config
[00:18]  * rbasak shrugs
[00:19] <rbasak> If it works for login but not ssh it's either in your PAM configuration or in sshd configuration most likely.
[00:19] <rbasak> Or how you're using ssh.
[00:19] <blackflow> Oh I see. Yes it's in sshd config
[00:20] <blackflow> UsePAM no
[00:21] <blackflow> yay, motd!
[00:22] <blackflow> thanks. herp derps like this one are impossible to solve without external input. I went through entire config tempalte twice, didn't see it until now.
[00:22] <sarnold> nice find with UsePAM no.. I certainlywouldn't have stumbled on that quickly :)
[00:23] <sarnold> .. similarly to the -Snone, I probably wouldn't have thought of that, either.
[00:23] <cryptodan_mobile> blackflow: https://www.tecmint.com/protect-ssh-logins-with-ssh-motd-banner-messages/
[00:25] <rbasak> People trying ssh for unauthorised access use scripts and will never see a threatening motd. As if that would stop them anyway.
[00:26] <blackflow> heh
[00:26] <blackflow> k I got my static motd, now to solve the no dynamic motd.... reboot again, changing pam.d/* doesn't change anything (still uses static motd). restarted ssh.service even.
[00:29] <rbasak> I suggest you start by removing all your customisations and then add them bit by bit.
[00:31] <blackflow> rbasak: there aren't any for pam.d/* or /update-motd.d/*  . Now I can get a static /etc/motd displayed, but if I remove it and switch back pam.d/{login,sshd} to using /run/motd.dynamic (and drop the noupdate flag), I don't get any motd.
[00:32] <blackflow> the part I don't understand is where does /run/motd.dynamic come from. /etc/update-motd.d/50-motd-news is writing to /var/cache/motd-news
[00:34] <blackflow> hmm... /var/cache/motd.news is just _part_ of entire motd which shows disk usage, memory,  updates...... now I'm intrigued, what is creating that?
[00:38] <blackflow> okay, update-motd(5) explains how /run/motd.dynamic is constructed, but it states it's done by update-motd!  that package is NOT installed, and the manapage is part of libpam-modules package. wth is this witchcraft! :)
[00:40] <blackflow> installed update-motd package.  now I have /run/motd (as a straight file) but still no /run/motd.dynamic    LOL?
[00:42] <blackflow> okay update-motd is red herring, it just sources all the scripts in /etc/update-motd.d/   and outputs to /run/motd   (but not /run/motd.dynamic)
[00:42] <cryptodan_mobile> blackflow: did you read my link
[00:44] <blackflow> I skimmed through it
[00:45] <blackflow> there's a serious disconnect here between sanity, consistency, manpages and what's actually going on :)
[01:00] <blackflow> I think something is wrong with systemd/session/sshd/pam integration in this server that was debootstrap'd. libpam-sytemd IS installed tho'. Thing is, unlike on the server that DOES show dynamic motd, this server does not seem to log same entries of User slice creation
[01:01] <blackflow> I think that's it, pam_motd(8) manapge states it's a session module type.
[01:09] <blackflow> hrm....... one forced reinstall of libpam-systemd and one reboot later it appears to be working now.
[01:10] <blackflow> I'm sure dpkg showed libpam-systemd installed, and I'm sure I rebooted.... a ton of times since enabling PAM for sshd....
[01:14] <blackflow> rbasak: sarnold: thanks for suggestions!
[01:16] <sarnold> blackflow: are you all sorted? or just sick of it for the day? :) heh
[01:17] <blackflow> sarnold: sorted, I got it working. Now I'll disable it because it's friggin annoying, but the point was to understand how it works. :)
[01:17] <Goop> cryptodan_mobile, sarnold, blackflow I think I'm going to try Keycloaker. Only thing is that I have never really used docker before.
[01:17] <sarnold> blackflow: hahahaha <3 that's awesome :)
[01:18] <Goop> Keycloak, sorry.
[01:19] <sarnold> Goop: ooh. it looks neat.
[01:19] <blackflow> sarnold: it started with people complaining that it uses bit.ly links and I was like "Wait, I don't have that at all"... so naturally, me hating blackboxes, I had to get into that and figure out why I wasn't getting it. UsePAM no. Part of my sshd_config template for years.
[01:20] <sarnold> blackflow: do you recall why you set usepam no in the first place?
[01:20] <blackflow> (and there was some derp with this particular machine as reinstallation of libpam-systemd fixed it -- I managed to replicate and confirm by removing the package and reinstalling, UsePAM it is)
[01:20] <blackflow> sarnold: not really, it was long time ago
[01:20] <sarnold> I've got memories of setting it once upon a time too, back when it was first introduced..
[01:21] <sarnold> probably under the "pam looks complicated and underdocumented" gripe :)
[01:21] <blackflow> I think it's really jsut about it not being needed on my machines back then and for reasons of simplicitly and less moving parts, I disabled it.
[01:22] <blackflow> Truth be told.... I still don't need it, lol. :) it's jsut that now I understand fully the relationship between ssh, pam and motd.
[01:22] <blackflow> sarnold: yah that'd be the same gripe that got me disabling it :)
[01:22] <sarnold> blackflow: quick, write it down, before you forget it all again, like me :)
[01:23] <blackflow> did, added a comment in my standard sshd_config template :)
[01:24] <sarnold> :D
[01:26] <blackflow> and speaking of bit.ly complaints (like bug #1789850) .... I saw http://ubu.one/  shortened link in today's motd. why not use that?
[01:26] <ubottu> bug 1789850 in update-motd (Ubuntu) "Advertising in the MOTD" [Undecided,Confirmed] https://launchpad.net/bugs/1789850
[01:30] <sarnold> that's just another name for bitly iirc
[01:34] <blackflow> what is?  ubu.one is owned by canonical, bit.ly is..... LYbian Telecom.
[01:35] <mybalzitch> I'm never clicking on another bit.ly link
[01:37] <sarnold> host ubu.one --> 67.199.248.12 ; whois 67.199.248.12 | grep Organization --> Bitly Inc (BITLY)
[01:38] <blackflow> huh, the plot thickens!
[01:39] <blackflow> there's still that lybian telecom in the play and that's why people dislike it. using just ubu.one would aleviate that.
[01:54] <Glorfindel> blackflow: using ubu.one will not alleviate that until canonical stops relying on bitly for their link shortening service :P
[01:56] <blackflow> Glorfindel: which is what I sad. use ubu.one instead of bit.ly
[01:56] <blackflow> *said
[01:59] <mybalzitch> still bitly infrastructure
[02:00] <Glorfindel> blackflow: but using ubu.one is still using bitly, like I just said
[02:00] <sarnold> how on earth libya's telecom came up with such a useful service I'll never understand :)
[02:02] <blackflow> the telco is registrar only here, bitly is a US company
[02:04] <blackflow> I personally don't have an issue with it because it doesn't matter. gubermnts hijacking traffic via faek BGP, TLS CA security based *solely* on those CAs sayin "we swear we won't abuse".    shrug.
[02:04] <blackflow> but I get it why someone would dislike .ly in their MOTD feeds.
[06:17] <cpaelzer> good morning
[09:51] <kstenerud> I've gotten myself stuck with dput. I uploaded a ppa but not to the right place, and I got a rejection email as a result. Now when I try to upload to the right place:
[09:51] <kstenerud> $ dput ppa:kstenerud/xenial-tomcat-resource-names-1606331 tomcat8_8.0.32-1ubuntu1.9_source.changes
[09:51] <kstenerud> Package has already been uploaded to ppa on ppa.launchpad.net
[09:51] <kstenerud> Nothing more to do for tomcat8_8.0.32-1ubuntu1.9_source.changes
[09:59] <DK2> im wanting to apt-get upgrade my mailserver: https://paste.ee/p/6h6P4
[10:00] <DK2> im a bit afraid because it says: following new packages will be installed: dovecot-core {..} etc.
[10:00] <DK2> however they will also be upgraded
[10:00] <DK2> is smth broken here?
[11:13] <ahasenack> good morning
[12:46] <ahasenack> rbasak: cpaelzer what shall I do when there have been other seed changes besides the one I did for server? They are surfacing when I re-generate the ubuntu-meta pacakges: https://pastebin.ubuntu.com/p/7VKt2zzWsh/
[12:46] <ahasenack> (d/changelog automatically updated by the update script)
[12:48] <xnox> ahasenack, that debdiff looks incomplete. first upload to new series must also include updates to update.cfg to switch to the new series.
[12:48] <xnox> ahasenack, cause you want this for disco, right?!
[12:50] <ahasenack> yes, I did update and commit update.cfg in the git branch, I just didn't update d/changelog with it. It's my first run ever of that script and I didn't know all that it would do
[12:50] <xnox> ahasenack, e.g. see the end of http://launchpadlibrarian.net/370085872/ubuntu-meta_1.417_1.418.diff.gz
[12:51] <xnox> ahasenack, well, you should paste complete debdiff then =) and you should mention that you are switching to new series.
[12:51] <ahasenack> sure
[12:51] <xnox> (in debian/changelog that is)
[12:51] <ahasenack> I was specifically asking about the results of the update script, which showed to be there are other seed changes
[12:52] <ahasenack> i.e., others made seed changes and didn't update the ubuntu-meta package (yet?)
[12:52] <ahasenack> I should probably wait for disco to be opened
[12:58] <cpaelzer> ahasenack: it is usually fine and correct to pick those changes up
[12:58] <cpaelzer> ahasenack: the question is more why they ahve not yet been picked up, but for disco the reason is clear if changes are recent
[12:58] <ahasenack> yeah, I didn't check how recent
[12:58] <ahasenack> but since the archive is still closed
[12:58] <ahasenack> it's moot anyway
[12:59] <ahasenack> I'll leave that card in the doing lane together with all the others (or TODO)
[13:05] <rbasak> I think people generally leave ubuntu-meta updates to be batched.
[13:05] <rbasak> (unless they want to see the results immediately)
[13:09] <ahasenack> but there is no way to upload it with just your changes, right
[13:09] <ahasenack> ignoring the other seed changes
[13:09] <ahasenack> it's not how it's supposed to be done, I mena
[13:09] <ahasenack> mean
[13:10] <rbasak> Right
[15:29] <shubjero> # timedatectl
[15:29] <shubjero> Failed to create bus connection: No such file or directory
[15:29] <shubjero> anyone see this before? just trying to use timedatectl on a vanilla ubuntu 16.04 server. no docker containers or anything going on...
[15:30] <setuid> connect(3, {sa_family=AF_UNIX, sun_path="/run/dbus/system_bus_socket"}, 29) = 0
[15:31] <openfire> shubjero: apt install dbus
[15:31] <setuid> ^^ shubjero
[15:32] <shubjero> it appears to be installed already
[15:32] <shubjero> dbus:
[15:32] <shubjero>   Installed: 1.10.6-1ubuntu3.3
[15:32] <shubjero>   Candidate: 1.10.6-1ubuntu3.3
[15:33] <openfire> Is it running?
[15:33] <setuid> strace timedatectl
[15:34] <setuid> See what it's failing on
[15:34] <TJ-> shubjero: check /var/log/syslog - look for the dbus service starting, and any apparmor messages (especially DENYs)
[15:35] <shubjero> https://paste.ubuntu.com/p/BvVBCX38tt/
[15:35] <shubjero> yeah the service wont start and i dont see any apparmor about denying it
[15:35] <TJ-> shubjero: you'd expect something like this https://paste.ubuntu.com/p/G3fCmQc2ns/
[15:35] <openfire> "failed to create bus connection" generally implies that dbus either isn't installed or failed to start.
[15:36] <TJ-> shubjero: does "/var/run/dbus/system_bus_socket" exist?
[15:36] <openfire> You could just check dbus' service status via systemctl.
[15:36] <shubjero> hmm yeah dbus isnt started and wont let me start it
[15:37] <openfire> journalctl -u dbus |& curl -F c=@- https://ptpb.pw
[15:37] <shubjero> https://paste.ubuntu.com/p/QCcK2t3tpX/
[15:37] <openfire> Or that.
[15:38] <openfire> Any error messages in 'journalctl -u dbus'?
[15:38] <shubjero> no entries
[15:39] <shubjero> i have a fleet of compute nodes and two of them are behaving this way.. so its nice to be able to compare working / not working
[15:39] <shubjero> just not sure why these two are misbehaving :)
[15:42] <TJ-> shubjero: maybe there's a dbus system-local.conf ?
[15:55] <shubjero> TJ-: /var/run/dbus/system_bus_socket exists on systems where i dont have an issue, but not on the two systems that dont appear to be working properly
[15:57] <shubjero> infact on the broken systems i dont even have /var/run/dbus
[16:02] <TJ-> shubjero: right; that socket is created by the service to listen on. You need to deep-dive into the dbus service/config
[16:03] <shubjero> yeah, fun
[16:03] <shubjero> lol
[16:04] <TJ-> shubjero: are the hosts supposed to be identical clones?
[16:11] <shubjero> pretty much
[16:14] <TJ-> which means "no" :D
[16:15] <shubjero> haha well you know how things go
[16:15] <shubjero> we try to manage configs with ansible so things should be changed and configured in the same way
[16:19] <TJ-> shubjero: right, so can you do a diff of a good and bad system's /etc/dbus-1/ directories? maybe start with a simple 'md5sum' complare
[16:20] <TJ-> shubjero: also /usr/share/dbus-1/
[16:21] <TJ-> shubjero: as in "find /usr/share/dbus-1 /etc/dbus-1 -type f -exec md5sum {} \; "
[16:23] <shubjero> apt-install --reinstall dbus
[16:23] <shubjero> fixed :)
[16:23] <shubjero> TJ-: thank you for your help & support :), cheers
[16:23] <shubjero> and others :)
[17:28] <smoser> rbasak: how do the 'Approved' branches get landed?
[17:36] <rbasak> smoser: I merge and push manually. Then from master I grab the snap from the Jenkins nightly job and upload it.
[17:37] <rbasak> smoser: I was planning to build a snap from the MPs I approved and do a bit of testing first.
[17:37] <rbasak> (so using "Approved" as a holding place really)
[19:45] <ahasenack> has anybody else seen this pattern in a cosmic server: https://pastebin.ubuntu.com/p/yHMdN8Dgxc/
[19:45] <ahasenack> the stuck "mount" call
[19:46] <ahasenack> it's in this line of code:
[19:46] <ahasenack>     ext_partitions=$(mount | awk '$5 ~ /^ext(2|3|4)$/ { print $1 }')
[19:46] <ahasenack> from, rather
[19:46] <ahasenack> which I can call interactively just fine:
[19:47] <ahasenack> root@duo:~# mount | awk '$5 ~ /^ext(2|3|4)$/ { print $1 }'
[19:47] <ahasenack>  /dev/sdb2
[19:47] <sarnold> ahasenack: try mounting an nfs server and then take away the nfs server
[19:47] <ahasenack> but it's in R state
[19:47] <ahasenack> not D
[19:48] <sarnold> oh. good point.
[19:48] <sarnold> how'd it get stuck there? :)
[19:48] <ahasenack> fd 0 is /dev/null, 1 is pipe, 2 is /dev/null
[19:48] <sarnold> do you have one *currently* stuck?
[19:49] <ahasenack> yes
[19:49] <sarnold> YES
[19:49] <ahasenack> it's that pastebin
[19:49] <ahasenack> using 100% cpu
[19:49] <sarnold> strace it?
[19:49] <sarnold> what syscalls is this stupid thing doing? :)
[19:49] <ahasenack> shows nothing
[19:49] <ahasenack> root@duo:~# strace -f -p 6395
[19:49] <ahasenack> strace: Process 6395 attached
[19:49] <ahasenack> (stuck)
[19:49] <sarnold> neat. not at all what I expected.
[19:49] <sarnold> ltrace?
[19:50] <ahasenack> and ctrl-c doesn't kill strace
[19:50] <ahasenack> I have to ctrl-z and do some kill %1
[19:50] <sarnold> hah. that sure smells like hung NFS..
[19:50] <sarnold> but again, R. you've got a crazy problem there :)
[19:50] <ahasenack> ltrace is also silent
[19:50] <sarnold> I bet that strace is still there?
[19:51] <ahasenack> I killed it with -9
[19:51] <ahasenack> and it died
[19:51] <sarnold> okay, that feels like a good sign
[19:51] <ahasenack> I can probably kill that mount in the same way
[19:51] <sarnold> don't :)
[19:51] <sarnold> perf top?
[19:52] <ahasenack> I've seen it a few times already, I don't think it will be hard to reproduce
[19:52] <ahasenack> what's perf top?
[19:52] <ahasenack> from linux-tools-common?
[19:54] <sarnold> ahasenack: yes
[19:54] <sarnold> ahasenack: here's my favourite perf guide http://www.brendangregg.com/perf.html#OneLiners
[19:54] <sarnold> perf top is just the easiest thing to recall off the top of my head :)
[19:55] <ahasenack> sarnold: what it is showing: https://pastebin.ubuntu.com/p/HBsbSDHRCy/
[19:56] <sarnold> ahasenack: hrm. I expected it to be spining in userspace.. since strace didn't show an *entry* into a syscall, I assumed it wasn't *in* the kernel. but that perf top output sure looks like something in the kernel is spinning madly
[19:56] <ahasenack> https://pastebin.ubuntu.com/p/tbH4t2P2rF/ that's the mount process
[19:57] <ahasenack> wonder what branches is
[19:58] <sarnold> ifeq, ifneq, etc
[20:01] <ahasenack> machine load is 1