| blackflow | but it's not installed, even on the server that does create /run/motd.dynamic. that'st he key here, what creates that file | 00:00 |
|---|---|---|
| rbasak | pam_motd creates it | 00:01 |
| blackflow | 50-motd-news is apparently creating /var/cache/motd-news, but not /run/motd.dynamic. both are straight files, not a symlink or something | 00:01 |
| rbasak | From /etc/pam.d/login and /etc/pam.d/sshd | 00:01 |
| blackflow | no it doesn't _create_ it | 00:02 |
| rbasak | See pam_motd(8) | 00:02 |
| blackflow | it _displays_ existing file, doesnt' trigger its creation | 00:02 |
| rbasak | No. Read the man page. | 00:03 |
| rbasak | noupdate: Don't run the scripts in /etc/update-motd.d to refresh the motd file. | 00:03 |
| rbasak | IOW, don't specify that, and it will update. | 00:03 |
| blackflow | the server that has /run/motd.dynamic has that too | 00:04 |
| blackflow | so it's not it | 00:04 |
| blackflow | do I need anything special after changing a pam.d/ config? restart something? | 00:06 |
| sarnold | I think those changes are picked up by services without restarts | 00:07 |
| blackflow | welp that didn't change anything. I removed "noupdate" both from login and sshd pam.d modules. Still no /run/motd.dynamic and no MOTD on login via ssh. | 00:10 |
| sarnold | try fatrace or perf trace or something similar? strace might not be fantastic .. | 00:12 |
| blackflow | trace what? | 00:12 |
| sarnold | whichever service you're using for testing | 00:12 |
| blackflow | I'm logging in via ssh | 00:13 |
| cryptodan_mobile | Wouldnt motd go in /etc | 00:13 |
| blackflow | on the server installed from server ISO I get motd. On the server I installed from debootstrap, I don't. | 00:13 |
| rbasak | Are you using -Snone? | 00:14 |
| blackflow | and I tried forcign a static motd even, not /run/motd.dynamic, it doesn't show | 00:14 |
| rbasak | Check "sudo login -f root" | 00:14 |
| blackflow | that shows the motd yes | 00:14 |
| blackflow | (twice, even) | 00:15 |
| rbasak | What about -Snone? | 00:15 |
| blackflow | what is -Snone ? | 00:15 |
| rbasak | To ssh | 00:15 |
| rbasak | To make sure you aren't sharing an existing connection | 00:15 |
| sarnold | hahahahaha | 00:16 |
| sarnold | oh man | 00:16 |
| blackflow | there's no other connection and yeah I tried just with -Snone (on the client side), still no motd. | 00:16 |
| rbasak | You messed with your PAM configuration, didn't you? | 00:17 |
| rbasak | For ssh but not for login? | 00:17 |
| blackflow | I also don't have that local ~/.hushwhatever file | 00:17 |
| blackflow | rbasak: just now to force a static /etc/motd since default configured /run/motd.dynamic doesn't exist | 00:18 |
| blackflow | I touched nothing else in pam config | 00:18 |
| * rbasak shrugs | 00:18 | |
| rbasak | If it works for login but not ssh it's either in your PAM configuration or in sshd configuration most likely. | 00:19 |
| rbasak | Or how you're using ssh. | 00:19 |
| blackflow | Oh I see. Yes it's in sshd config | 00:19 |
| blackflow | UsePAM no | 00:20 |
| blackflow | yay, motd! | 00:21 |
| blackflow | thanks. herp derps like this one are impossible to solve without external input. I went through entire config tempalte twice, didn't see it until now. | 00:22 |
| sarnold | nice find with UsePAM no.. I certainlywouldn't have stumbled on that quickly :) | 00:22 |
| sarnold | .. similarly to the -Snone, I probably wouldn't have thought of that, either. | 00:23 |
| cryptodan_mobile | blackflow: https://www.tecmint.com/protect-ssh-logins-with-ssh-motd-banner-messages/ | 00:23 |
| rbasak | People trying ssh for unauthorised access use scripts and will never see a threatening motd. As if that would stop them anyway. | 00:25 |
| blackflow | heh | 00:26 |
| blackflow | k I got my static motd, now to solve the no dynamic motd.... reboot again, changing pam.d/* doesn't change anything (still uses static motd). restarted ssh.service even. | 00:26 |
| rbasak | I suggest you start by removing all your customisations and then add them bit by bit. | 00:29 |
| blackflow | rbasak: there aren't any for pam.d/* or /update-motd.d/* . Now I can get a static /etc/motd displayed, but if I remove it and switch back pam.d/{login,sshd} to using /run/motd.dynamic (and drop the noupdate flag), I don't get any motd. | 00:31 |
| blackflow | the part I don't understand is where does /run/motd.dynamic come from. /etc/update-motd.d/50-motd-news is writing to /var/cache/motd-news | 00:32 |
| blackflow | hmm... /var/cache/motd.news is just _part_ of entire motd which shows disk usage, memory, updates...... now I'm intrigued, what is creating that? | 00:34 |
| blackflow | okay, update-motd(5) explains how /run/motd.dynamic is constructed, but it states it's done by update-motd! that package is NOT installed, and the manapage is part of libpam-modules package. wth is this witchcraft! :) | 00:38 |
| blackflow | installed update-motd package. now I have /run/motd (as a straight file) but still no /run/motd.dynamic LOL? | 00:40 |
| blackflow | okay update-motd is red herring, it just sources all the scripts in /etc/update-motd.d/ and outputs to /run/motd (but not /run/motd.dynamic) | 00:42 |
| cryptodan_mobile | blackflow: did you read my link | 00:42 |
| blackflow | I skimmed through it | 00:44 |
| blackflow | there's a serious disconnect here between sanity, consistency, manpages and what's actually going on :) | 00:45 |
| blackflow | I think something is wrong with systemd/session/sshd/pam integration in this server that was debootstrap'd. libpam-sytemd IS installed tho'. Thing is, unlike on the server that DOES show dynamic motd, this server does not seem to log same entries of User slice creation | 01:00 |
| blackflow | I think that's it, pam_motd(8) manapge states it's a session module type. | 01:01 |
| blackflow | hrm....... one forced reinstall of libpam-systemd and one reboot later it appears to be working now. | 01:09 |
| blackflow | I'm sure dpkg showed libpam-systemd installed, and I'm sure I rebooted.... a ton of times since enabling PAM for sshd.... | 01:10 |
| blackflow | rbasak: sarnold: thanks for suggestions! | 01:14 |
| sarnold | blackflow: are you all sorted? or just sick of it for the day? :) heh | 01:16 |
| blackflow | sarnold: sorted, I got it working. Now I'll disable it because it's friggin annoying, but the point was to understand how it works. :) | 01:17 |
| Goop | cryptodan_mobile, sarnold, blackflow I think I'm going to try Keycloaker. Only thing is that I have never really used docker before. | 01:17 |
| sarnold | blackflow: hahahaha <3 that's awesome :) | 01:17 |
| Goop | Keycloak, sorry. | 01:18 |
| sarnold | Goop: ooh. it looks neat. | 01:19 |
| blackflow | sarnold: it started with people complaining that it uses bit.ly links and I was like "Wait, I don't have that at all"... so naturally, me hating blackboxes, I had to get into that and figure out why I wasn't getting it. UsePAM no. Part of my sshd_config template for years. | 01:19 |
| sarnold | blackflow: do you recall why you set usepam no in the first place? | 01:20 |
| blackflow | (and there was some derp with this particular machine as reinstallation of libpam-systemd fixed it -- I managed to replicate and confirm by removing the package and reinstalling, UsePAM it is) | 01:20 |
| blackflow | sarnold: not really, it was long time ago | 01:20 |
| sarnold | I've got memories of setting it once upon a time too, back when it was first introduced.. | 01:20 |
| sarnold | probably under the "pam looks complicated and underdocumented" gripe :) | 01:21 |
| blackflow | I think it's really jsut about it not being needed on my machines back then and for reasons of simplicitly and less moving parts, I disabled it. | 01:21 |
| blackflow | Truth be told.... I still don't need it, lol. :) it's jsut that now I understand fully the relationship between ssh, pam and motd. | 01:22 |
| blackflow | sarnold: yah that'd be the same gripe that got me disabling it :) | 01:22 |
| sarnold | blackflow: quick, write it down, before you forget it all again, like me :) | 01:22 |
| blackflow | did, added a comment in my standard sshd_config template :) | 01:23 |
| sarnold | :D | 01:24 |
| blackflow | and speaking of bit.ly complaints (like bug #1789850) .... I saw http://ubu.one/ shortened link in today's motd. why not use that? | 01:26 |
| ubottu | bug 1789850 in update-motd (Ubuntu) "Advertising in the MOTD" [Undecided,Confirmed] https://launchpad.net/bugs/1789850 | 01:26 |
| sarnold | that's just another name for bitly iirc | 01:30 |
| blackflow | what is? ubu.one is owned by canonical, bit.ly is..... LYbian Telecom. | 01:34 |
| mybalzitch | I'm never clicking on another bit.ly link | 01:35 |
| sarnold | host ubu.one --> 67.199.248.12 ; whois 67.199.248.12 | grep Organization --> Bitly Inc (BITLY) | 01:37 |
| blackflow | huh, the plot thickens! | 01:38 |
| blackflow | there's still that lybian telecom in the play and that's why people dislike it. using just ubu.one would aleviate that. | 01:39 |
| Glorfindel | blackflow: using ubu.one will not alleviate that until canonical stops relying on bitly for their link shortening service :P | 01:54 |
| blackflow | Glorfindel: which is what I sad. use ubu.one instead of bit.ly | 01:56 |
| blackflow | *said | 01:56 |
| mybalzitch | still bitly infrastructure | 01:59 |
| Glorfindel | blackflow: but using ubu.one is still using bitly, like I just said | 02:00 |
| sarnold | how on earth libya's telecom came up with such a useful service I'll never understand :) | 02:00 |
| blackflow | the telco is registrar only here, bitly is a US company | 02:02 |
| blackflow | I personally don't have an issue with it because it doesn't matter. gubermnts hijacking traffic via faek BGP, TLS CA security based *solely* on those CAs sayin "we swear we won't abuse". shrug. | 02:04 |
| blackflow | but I get it why someone would dislike .ly in their MOTD feeds. | 02:04 |
| cpaelzer | good morning | 06:17 |
| kstenerud | I've gotten myself stuck with dput. I uploaded a ppa but not to the right place, and I got a rejection email as a result. Now when I try to upload to the right place: | 09:51 |
| kstenerud | $ dput ppa:kstenerud/xenial-tomcat-resource-names-1606331 tomcat8_8.0.32-1ubuntu1.9_source.changes | 09:51 |
| kstenerud | Package has already been uploaded to ppa on ppa.launchpad.net | 09:51 |
| kstenerud | Nothing more to do for tomcat8_8.0.32-1ubuntu1.9_source.changes | 09:51 |
| DK2 | im wanting to apt-get upgrade my mailserver: https://paste.ee/p/6h6P4 | 09:59 |
| DK2 | im a bit afraid because it says: following new packages will be installed: dovecot-core {..} etc. | 10:00 |
| DK2 | however they will also be upgraded | 10:00 |
| DK2 | is smth broken here? | 10:00 |
| ahasenack | good morning | 11:13 |
| ahasenack | rbasak: cpaelzer what shall I do when there have been other seed changes besides the one I did for server? They are surfacing when I re-generate the ubuntu-meta pacakges: https://pastebin.ubuntu.com/p/7VKt2zzWsh/ | 12:46 |
| ahasenack | (d/changelog automatically updated by the update script) | 12:46 |
| xnox | ahasenack, that debdiff looks incomplete. first upload to new series must also include updates to update.cfg to switch to the new series. | 12:48 |
| xnox | ahasenack, cause you want this for disco, right?! | 12:48 |
| ahasenack | yes, I did update and commit update.cfg in the git branch, I just didn't update d/changelog with it. It's my first run ever of that script and I didn't know all that it would do | 12:50 |
| xnox | ahasenack, e.g. see the end of http://launchpadlibrarian.net/370085872/ubuntu-meta_1.417_1.418.diff.gz | 12:50 |
| xnox | ahasenack, well, you should paste complete debdiff then =) and you should mention that you are switching to new series. | 12:51 |
| ahasenack | sure | 12:51 |
| xnox | (in debian/changelog that is) | 12:51 |
| ahasenack | I was specifically asking about the results of the update script, which showed to be there are other seed changes | 12:51 |
| ahasenack | i.e., others made seed changes and didn't update the ubuntu-meta package (yet?) | 12:52 |
| ahasenack | I should probably wait for disco to be opened | 12:52 |
| cpaelzer | ahasenack: it is usually fine and correct to pick those changes up | 12:58 |
| cpaelzer | ahasenack: the question is more why they ahve not yet been picked up, but for disco the reason is clear if changes are recent | 12:58 |
| ahasenack | yeah, I didn't check how recent | 12:58 |
| ahasenack | but since the archive is still closed | 12:58 |
| ahasenack | it's moot anyway | 12:58 |
| ahasenack | I'll leave that card in the doing lane together with all the others (or TODO) | 12:59 |
| rbasak | I think people generally leave ubuntu-meta updates to be batched. | 13:05 |
| rbasak | (unless they want to see the results immediately) | 13:05 |
| ahasenack | but there is no way to upload it with just your changes, right | 13:09 |
| ahasenack | ignoring the other seed changes | 13:09 |
| ahasenack | it's not how it's supposed to be done, I mena | 13:09 |
| ahasenack | mean | 13:09 |
| rbasak | Right | 13:10 |
| shubjero | # timedatectl | 15:29 |
| shubjero | Failed to create bus connection: No such file or directory | 15:29 |
| shubjero | anyone see this before? just trying to use timedatectl on a vanilla ubuntu 16.04 server. no docker containers or anything going on... | 15:29 |
| setuid | connect(3, {sa_family=AF_UNIX, sun_path="/run/dbus/system_bus_socket"}, 29) = 0 | 15:30 |
| openfire | shubjero: apt install dbus | 15:31 |
| setuid | ^^ shubjero | 15:31 |
| shubjero | it appears to be installed already | 15:32 |
| shubjero | dbus: | 15:32 |
| shubjero | Installed: 1.10.6-1ubuntu3.3 | 15:32 |
| shubjero | Candidate: 1.10.6-1ubuntu3.3 | 15:32 |
| openfire | Is it running? | 15:33 |
| setuid | strace timedatectl | 15:33 |
| setuid | See what it's failing on | 15:34 |
| TJ- | shubjero: check /var/log/syslog - look for the dbus service starting, and any apparmor messages (especially DENYs) | 15:34 |
| shubjero | https://paste.ubuntu.com/p/BvVBCX38tt/ | 15:35 |
| shubjero | yeah the service wont start and i dont see any apparmor about denying it | 15:35 |
| TJ- | shubjero: you'd expect something like this https://paste.ubuntu.com/p/G3fCmQc2ns/ | 15:35 |
| openfire | "failed to create bus connection" generally implies that dbus either isn't installed or failed to start. | 15:35 |
| TJ- | shubjero: does "/var/run/dbus/system_bus_socket" exist? | 15:36 |
| openfire | You could just check dbus' service status via systemctl. | 15:36 |
| shubjero | hmm yeah dbus isnt started and wont let me start it | 15:36 |
| openfire | journalctl -u dbus |& curl -F c=@- https://ptpb.pw | 15:37 |
| shubjero | https://paste.ubuntu.com/p/QCcK2t3tpX/ | 15:37 |
| openfire | Or that. | 15:37 |
| openfire | Any error messages in 'journalctl -u dbus'? | 15:38 |
| shubjero | no entries | 15:38 |
| shubjero | i have a fleet of compute nodes and two of them are behaving this way.. so its nice to be able to compare working / not working | 15:39 |
| shubjero | just not sure why these two are misbehaving :) | 15:39 |
| TJ- | shubjero: maybe there's a dbus system-local.conf ? | 15:42 |
| shubjero | TJ-: /var/run/dbus/system_bus_socket exists on systems where i dont have an issue, but not on the two systems that dont appear to be working properly | 15:55 |
| shubjero | infact on the broken systems i dont even have /var/run/dbus | 15:57 |
| TJ- | shubjero: right; that socket is created by the service to listen on. You need to deep-dive into the dbus service/config | 16:02 |
| shubjero | yeah, fun | 16:03 |
| shubjero | lol | 16:03 |
| TJ- | shubjero: are the hosts supposed to be identical clones? | 16:04 |
| shubjero | pretty much | 16:11 |
| TJ- | which means "no" :D | 16:14 |
| shubjero | haha well you know how things go | 16:15 |
| shubjero | we try to manage configs with ansible so things should be changed and configured in the same way | 16:15 |
| TJ- | shubjero: right, so can you do a diff of a good and bad system's /etc/dbus-1/ directories? maybe start with a simple 'md5sum' complare | 16:19 |
| TJ- | shubjero: also /usr/share/dbus-1/ | 16:20 |
| TJ- | shubjero: as in "find /usr/share/dbus-1 /etc/dbus-1 -type f -exec md5sum {} \; " | 16:21 |
| shubjero | apt-install --reinstall dbus | 16:23 |
| shubjero | fixed :) | 16:23 |
| shubjero | TJ-: thank you for your help & support :), cheers | 16:23 |
| shubjero | and others :) | 16:23 |
| smoser | rbasak: how do the 'Approved' branches get landed? | 17:28 |
| rbasak | smoser: I merge and push manually. Then from master I grab the snap from the Jenkins nightly job and upload it. | 17:36 |
| rbasak | smoser: I was planning to build a snap from the MPs I approved and do a bit of testing first. | 17:37 |
| rbasak | (so using "Approved" as a holding place really) | 17:37 |
| ahasenack | has anybody else seen this pattern in a cosmic server: https://pastebin.ubuntu.com/p/yHMdN8Dgxc/ | 19:45 |
| ahasenack | the stuck "mount" call | 19:45 |
| ahasenack | it's in this line of code: | 19:46 |
| ahasenack | ext_partitions=$(mount | awk '$5 ~ /^ext(2|3|4)$/ { print $1 }') | 19:46 |
| ahasenack | from, rather | 19:46 |
| ahasenack | which I can call interactively just fine: | 19:46 |
| ahasenack | root@duo:~# mount | awk '$5 ~ /^ext(2|3|4)$/ { print $1 }' | 19:47 |
| ahasenack | /dev/sdb2 | 19:47 |
| sarnold | ahasenack: try mounting an nfs server and then take away the nfs server | 19:47 |
| ahasenack | but it's in R state | 19:47 |
| ahasenack | not D | 19:47 |
| sarnold | oh. good point. | 19:48 |
| sarnold | how'd it get stuck there? :) | 19:48 |
| ahasenack | fd 0 is /dev/null, 1 is pipe, 2 is /dev/null | 19:48 |
| sarnold | do you have one *currently* stuck? | 19:48 |
| ahasenack | yes | 19:49 |
| sarnold | YES | 19:49 |
| ahasenack | it's that pastebin | 19:49 |
| ahasenack | using 100% cpu | 19:49 |
| sarnold | strace it? | 19:49 |
| sarnold | what syscalls is this stupid thing doing? :) | 19:49 |
| ahasenack | shows nothing | 19:49 |
| ahasenack | root@duo:~# strace -f -p 6395 | 19:49 |
| ahasenack | strace: Process 6395 attached | 19:49 |
| ahasenack | (stuck) | 19:49 |
| sarnold | neat. not at all what I expected. | 19:49 |
| sarnold | ltrace? | 19:49 |
| ahasenack | and ctrl-c doesn't kill strace | 19:50 |
| ahasenack | I have to ctrl-z and do some kill %1 | 19:50 |
| sarnold | hah. that sure smells like hung NFS.. | 19:50 |
| sarnold | but again, R. you've got a crazy problem there :) | 19:50 |
| ahasenack | ltrace is also silent | 19:50 |
| sarnold | I bet that strace is still there? | 19:50 |
| ahasenack | I killed it with -9 | 19:51 |
| ahasenack | and it died | 19:51 |
| sarnold | okay, that feels like a good sign | 19:51 |
| ahasenack | I can probably kill that mount in the same way | 19:51 |
| sarnold | don't :) | 19:51 |
| sarnold | perf top? | 19:51 |
| ahasenack | I've seen it a few times already, I don't think it will be hard to reproduce | 19:52 |
| ahasenack | what's perf top? | 19:52 |
| ahasenack | from linux-tools-common? | 19:52 |
| sarnold | ahasenack: yes | 19:54 |
| sarnold | ahasenack: here's my favourite perf guide http://www.brendangregg.com/perf.html#OneLiners | 19:54 |
| sarnold | perf top is just the easiest thing to recall off the top of my head :) | 19:54 |
| ahasenack | sarnold: what it is showing: https://pastebin.ubuntu.com/p/HBsbSDHRCy/ | 19:55 |
| sarnold | ahasenack: hrm. I expected it to be spining in userspace.. since strace didn't show an *entry* into a syscall, I assumed it wasn't *in* the kernel. but that perf top output sure looks like something in the kernel is spinning madly | 19:56 |
| ahasenack | https://pastebin.ubuntu.com/p/tbH4t2P2rF/ that's the mount process | 19:56 |
| ahasenack | wonder what branches is | 19:57 |
| sarnold | ifeq, ifneq, etc | 19:58 |
| ahasenack | machine load is 1 | 20:01 |
Generated by irclog2html.py 2.7 by Marius Gedminas - find it at mg.pov.lt!