tewardRoyK: running memtest86 right now, but I don't think there'll be anything failing, this thing's passed memtest in the past01:05
tewardmemtest86+ *01:06
tewardRoyK: so far, 41% of the tests are done, and it's passing them all.  I'm fairly sure it's not a memory or CPU problem, this happened on an 18.04 plain install too, kernels updated and then *wham* dead with panics on boot01:14
tewardmaybe i need to go yell at #ubuntu-kernel...01:14
teward(memtest also takes some time when the system as 24GB of RAM :| )01:15
RoyKteward: make sure you have netconsole running in case something happens01:33
tewardRoyK: again, how do I enable netconsole when I can't even get into the system with any boot process?01:37
tewardthe unanswered question from before there remains01:38
tewardliterally the moment after Grub the system panics01:38
tewardnot enough time to get in to do anything01:38
tewardsame from LiveUSB01:38
tewardRoyK: yeah, 100% memtest pass.  so unless you know how to enable netconsole before the system boots...01:42
tewardah there it is01:43
tewardhmm i'll work on this tomrrow after work01:43
tewardtired now :p01:43
RoyKteward: you'll need to reboot01:45
tewardwell that's a given01:45
tewardbut i'll deal with this tomorrow, CBA to set up a syslog receiver at the moment01:46
teward(and i'm dead tired and have to be up early so... bed time.)01:46
RoyKthe only issue is that syslog won't send a panic01:47
Checkmatei've this problem with mod_evasive04:01
Checkmateeverything work fine but if i refresh the url page i can see it and the forbidden message disappear04:02
Checkmatei've tryed mod_qos and mod_evasive nothing work good04:28
tomreynCheckmate: mod_evasive can work ok if properly tuned. but it's kind of a last resort measure, it's much better to optimize the application / service configuration to cause less load (improve application source code, use caching, replace outdated software components by newer ones). alternatively, you can increase system resources and use load balancing, too.04:44
Checkmate@tomreyn i'm trying to stop multiple POST request04:48
Checkmatei've tryed all kind of things but seems not working04:48
tomreynCheckmate: you said so some hours ago. if you're looking for assitence with this you'll need to provide details (but there may be better places to ask this, such as in #httpd if you use apache httpd, as 'mod_evasive' suggests)04:49
tomreynthere is also mod_security which can be an option if you cannot modify the application source.04:52
CheckmateMaxKeepAliveRequests not work with me too04:52
tomreynCheckmate: maybe sztart by discussing the web application and versio which is under attack, the ubuntu version and web server you run there.04:54
tomreynthen discuss the path to the file which is being attacked, and how you can tell (logs). you can rewrite any ip addresses by 127.0.x.1 where you replace x by a different number (1-255) for every new ip address.04:56
Checkmate@tomreyn i'm under the latest version of Ubuntu getting attack at index.php05:00
Checkmatemultiple post request05:00
tomreynCheckmate: and index.php is part of which standard web application, or is it custom code? what do these requests look like?05:01
tomreynthe latest released version of ubuntu is 18.10. are you sure that'S the one you run? "lsb_release -ds" will tell.05:01
Checkmate@tomreyn yes apache205:03
tomreynthis doesn't answer an of the questions i asked :)05:04
Checkmateubuntu are you sure = yes  which standard web application = apache205:08
Checkmatemaybe i'm losing time here05:08
tomreynapache2 is the package name of the Apache HTTPd web server, version 2.x. "index.php" suggests this is PHP code. this PHP code will be part of a web application such as workpress, drupal, typo3, or something you wrote yourself. i'm asking which one it is.05:10
tomreynmaybe i'm losing time here05:10
Checkmateyeah my bad its personnel php encoding05:11
tomreynthen my bet is that this application cariies out too many tasks before authentication is required, allowing anyone to keep the server busy.05:13
tomreynyou shoould change it so that those actions which cause increased server load are only available after authentication, or for a given set of source ip addresses.05:14
Checkmatei dont wanna use a database05:16
Checkmatei want to limit connection request for each ip05:17
tomreynif you can share the source code, feel free to do so, i can take a quick look later, but this only makes sense if you can also provide more information on the POST requests which are sent. you can capture them using tcpdump, a proxy server, mod_dumpio (but be casreful there since it will drive up server load further).05:17
lordievadergood morning06:20
muhahaIs possible to download signing key from keyserver with curl ?  instead of this apt-key adv --keyserver keyserver.ubuntu.com --recv-keys 93C4A3FD7BB9C367 ?07:45
TJ-muhaha: it should be possible with gpg, using its --dry-run option *or* by setting an empty or null keyring so it doesn't write the retreived key into the trust store07:50
muhahaTJ-: like this? gpg --dry-run --keyserver keyserver.ubuntu.com --recv 93C4A3FD7BB9C36707:54
muhahaI can not pass internet proxy. It has some kind of useragent whitelisting... curl user agent works, thats why I need to download it with curl directly...07:55
TJ-muhaha: try using hkps://keyserver.ubuntu.com (uses TLS)07:58
Skuggenmuhaha: Have you tried adding --keyserver-options http_proxy=$value?07:58
muhahayes, it does not work07:59
muhahalike I said, I need to pass curl user agent to access internet07:59
lucyluHi guys.. I installed a second WP site on my ubuntu server. On the second site I was able to get the WP install screen and istall it. However when I went to change the theme the site disconnected and I cant connect to it : ERR_conn_timed_out .. Any way to check how to debug this?08:46
tomreynlucylu: see #ubuntu09:02
tomreynmuhaha: curl 'https://keyserver.ubuntu.com/pks/lookup?op=get&search=0x93C4A3FD7BB9C367'09:04
tomreynyou may need to strip the surrounding html09:05
cqshello, I'd like to ask whether there is some known issue while installing postgresql db on ubuntu server11:15
cqsit just hangs when setting up alternatives..11:15
cqsit's basically a clean installation11:15
cpaelzercqs: I'd not know of one11:16
cpaelzerI have seen an issue that sounds similar, but it was "update alternative ... already owned by <pkg>"11:17
cpaelzerand said pkg was not in the archive11:17
cqsupdate-alternatives: using /usr/share/postgresql/10/man/man1/postmaster.1.gz to provide /usr/share/man/man1/postmaster.1.gz (postmaster.1.gz) in auto mode11:17
cqsthis is where it ends up11:17
cpaelzerok, you have a different file at least11:17
cpaelzermaybe it makes sense for this one (other than the bug I checked a while ago)11:17
cqsi see11:18
* cpaelzer is checking ...11:18
ahasenackgood morning11:19
cpaelzerhi ahasenack11:20
ahasenackhi cpaelzer11:20
cpaelzerhave you seen above already (before I check too deep and you just know it) ^^11:20
cqsthanks for you time11:21
cpaelzercqs: I'm pretty sure you are "after" that update alternatives11:21
cqsyes i am11:21
cpaelzerwhat is an "ps axlf" showing you11:21
cpaelzermaybe pastebinit ?11:22
cpaelzerjust retried postgresql on fresh bionic and cosmic - works as expected11:22
cpaelzeris that the systemctl start hanging ?11:23
cqshard to tell u11:23
cpaelzeryeah looks that way, and that would be the last command of configure_version in /usr/share/postgresql-common/maintscripts-functions11:25
cpaelzercqs: what is "systemctl status postgresql.service" showing you atm?11:25
cqsloaded and active11:26
cqslooks normal11:26
cpaelzercqs: how long is that hanging already?11:27
cqsall looks fine, it's just apt stayed locked11:27
xnoxI have questions about git-ubuntu.... sometimes the merge proposals diffs look odd in launchpad, is that normal?11:28
ahasenackxnox: we recon it's a but in python-git11:28
RoyKcqs: can you connect to the db with 'psql' with user postgres?11:28
xnoxsecond question - there is nowhere one should push things? one should just dput & close MP right?11:28
xnoxthere is no automagic marking things as merged?11:29
cpaelzerxnox: there is, if you follow the tagging procedure that we do11:29
cpaelzerxnox: but I'm not sure if this is reasonable for everyone11:29
cpaelzerxnox: that would also retain your git commit history on the import11:29
xnoxi thought i tagged stuff, but all pushes from me were declined.....11:30
cpaelzerand the importer sees your dput being published, realizes it is the same and then closes things11:30
xnoxand if i can't push things back to origin.... where does one supposed to push?11:30
xnoxare there docs, i did not find/read?11:30
cpaelzerxnox: the repo "pkg" that is set up by default is the target11:30
cpaelzerbut push access to that is rather restricted11:31
cqsRoyK: yes i can connect to psql11:31
cpaelzerespecially as we long term don't want to require manual tagging anyways11:31
cpaelzerwe want the importer to find approved MPs and use those11:31
cpaelzerxnox: it would be in (outdated I think) https://wiki.ubuntu.com/UbuntuDevelopment/Merging/GitWorkflow#Detailed_workflow11:32
xnoxcpaelzer, tah, i'll read up on that, and will try to use that again.11:32
cpaelzerxnox: I'd ask you to get in touch with rbasak - I think we need to decide what we should do until the intended future workflow (with just approved MPs) can happen11:32
cpaelzerhe is coordinating the related efforts11:32
cpaelzercqs: so it seems all is fine, but the systemctl start did not return and due to that blocks11:33
cpaelzerI unfortunately have no former case like this to base the next steps on :-/11:33
cqsso shall i just kill that stuck process and unlock apt?11:34
cpaelzerI guess, so - that will most likely make it fail11:35
cpaelzeryou might apt install --reinstall afterwards just to be sure11:35
cqsit wants to run 'sudo dpkg --configure -a' again but i guess it will hang again11:36
cpaelzertry it11:36
cqsbtw i was installing postgresql-contrib in one shot11:38
cqsmaybe this one is the guilty one11:38
cqs  Package postgresql-10 which provides postgresql-contrib-10 is not configured yet.11:38
rbasakxnox: since Launchpad publications are the single source of truth for Ubuntu, we can't allow uploaders to push things to origin since those might mismatch Launchpad publications, and the git view is supposed to be an exact reflection of what Launchpad published. So only the importer pushes after verifying correctness. But then uploaders need to supply rich history to the importer somehow. We don't11:39
rbasakhave a good workflow for this yet. As a stop gap we push "upload tags" but right now they have to be processed manually.11:39
rbasakWhen Launchpad per-ref ACL support arrives, we will be able to give uploaders access to push upload tags directly (but that'll still be a stop-gap)11:39
rbasakBefore then, if you want rich history preserved, ping someone in ~usd-import-team to do it.11:39
xnoxah i see, push a tag, rather than push branch refs.11:39
xnoxfair enough.11:39
xnoxto be fair, i don't mind either way, just didn't know what i should be doing today.11:40
rbasakThe importer will only adopt tag if the upload matches in Lanchpad.11:40
xnoxand if i did `something wrong, by not pushing something somewhere`11:40
rbasakIt's fine to dput without using git. The importer will cope by synthesizing a commit, which will effectively be the squashing of your commits.11:40
cpaelzercqs: apt install postgresql postgresql-contrib does not fail for me either11:48
cqsthe only thing that comes to my mind is that i didnt have locales properly set, so it defaulted to som utf811:48
cqsbut that's all11:48
rbasakxnox: that was becoming an FAQ, so here you go: https://askubuntu.com/q/1086094/780812:36
mad_mosesHi, can I use NFSv4 securely in the internet? (Enabling encryption?)12:49
UssatI would NEVER mount a NFS over the net unless it was over a VPN12:50
Ussatand then only as a last resort12:51
mad_mosesUssat: okay, just read that NFSv4 also supports encryption. So I thought it might be safe12:52
sdezielmad_moses: you could look at SSHFS maybe12:52
mad_mosessdeziel: sshfs is doing strange things. I mounted a server folder and try to work remotely with my pycharm on my python project but git and pycharm are doing strange things (refereshing every second)12:53
ahasenackcpaelzer: if you do a "lxc launch ubuntu-daily:trusty trusty-foo", and then enter that container and run "hostname -f", do you get trusty-foo.lxd?13:45
RoyKmad_moses: nfs4 uses kerberos for authentication, but sends the data in cleartext13:47
RoyKmad_moses: that is - nfs4 may use sec=sys and then only checks the client's ip address13:48
cpaelzerahasenack: seeing the quesiton just now13:51
cpaelzerjust a sec13:51
ahasenackno rush13:51
cpaelzerahasenack: yes I see the same13:52
ahasenackok, thx13:52
mad_mosesRoyK: okay, vpn is the answer ...14:16
mfoxnox, hey :) so, i noticed there are some regressions in systemd rdeps in pending-sru page.  i gone through them, and apparently none are related to the LP/patch I submitted (LP 1795658), but wondering if you'd like me to review any of them.14:37
ubottuLaunchpad bug 1795658 in systemd (Ubuntu Xenial) "xenial systemd reports 'inactive' instead of 'failed' for service units that repeatedly failed to restart / failed permanently" [Medium,Fix committed] https://launchpad.net/bugs/179565814:37
xnoxmfo, well, your patch is not the only thing in that upload.14:37
xnoxthere are 6 bug fixes.14:37
xnoxas you can see on the pending-sru page14:38
mfoxnox, yes, I see. that's why I'm asking :)  i could not track the test errors to _that_ part of the upload, but you know, maybe there are symptoms that are not that clearly linked.14:39
mfoi offered checking some of them in case you were suspicious they were not from the other uploads either.14:39
mfoxnox, well, and sorry to bother, but I just wanted to offer help in case that helped. no worries :)  if you find something for me about that, please just let me know and I'll take a look.14:41
xnoxmfo, i've hit retry on all of the regressions, to see if any of the flaky ones would clear.14:41
xnoxmfo, and then i'd be writting up to release team for things that are clearly broken.14:41
mfoxnox, ok. iirc there are a few ones after "rebooting" in autopkgtest, which seems like a couldn't connect post-reboot type of error, not really pkg code itself.14:42
mfoxnox, thanks!14:42
xnoxmfo, well, or a VM failing to boot with new systemd =/14:42
mfoxnox, oh, indeed. hopefully not the case!14:45
Checkmateguys my ip server down after typing this command iptables -L -n16:22
Checkmatewhats i do now ?16:23
tomreynwhat is an ip server?16:23
tomreynuse your out of bound access to flush iptables, or if you have none, power cycle it.16:24
Checkmate@tomreyn my vps down its like all opened ports closed do i need to restart server?16:26
TJ-"iptables -Ln" isn't going to add a rule, it just lists with numerics16:26
tomreynyes it should not cause this, but apparently somehow Checkmate was working on iptables and locked themselves out.16:27
Checkmatewell i put this command and same time the vps down16:27
TJ-Checkmate: did you issue any commands before that one?16:27
tomreynCheckmate: if you have no other way to access the system, such as through a virtual serial console or KVM your VPS host provides through e.g. their web panel, you will need to have them power cycle it (maybe also through a self service web panel).16:28
Checkmate@tomreyn yes iptables -F16:28
TJ-Checkmate: -F flushes the tables; if the default policy is DROP then you've cut yourself off16:29
Checkmatedamn what i need to do now @TJ-16:30
Checkmatei have a web shell access please provide me how to get my vps server back16:31
TJ-Checkmate: tomreyn has told you twice; reboot the server and hope there are saved rules in place to open ports. Otherwise use your host's out-of-band console access to fix it16:31
tomreynyou need to undo what you did last, or reboot which will undo non permanent configuration changes16:31
Checkmate@tomreyn after restarting everything work fine thx16:34
tomreyngood. keep the web shell open while you work on iptables, make sure you know how to undo changes before you make them16:36
Checkmate@tomreyn do u know how to clear all banned ip fail2ban service16:43
masonCheckmate: man fail2ban-client - there's probably some sleek way to do it all at once, but this will let you do it granularly.16:48
tomreynCheckmate: not of the top of my head, no16:48
Checkmatei see something about removing fail2ban.sqlite3 its safe?16:49
tomreynsqlite is a flat file database, often used for storing configurations. removing this is probably not what you want.16:50
RoyKCheckmate: if the server goes down with an iptables command, contact the people you got it from - it shouldn't panic or shut down for something like this17:37
RoyKCheckmate: where do you have this VPS?17:38
RoyKtomreyn: serial consoles are a bit hard to work out on virtual machines ;)17:38
tomreynRoyK: why so?17:39
tomreynboth KVM + Xen support (virtual) serial consoles just fine.17:39
tewardRoyK: so it did a full blown panic immediately when attempting to boot17:39
RoyKbecause there's no serial port, for starters, at least if you rent a VM from some company abroad17:39
tewardon my HP Z400 workstation17:39
tewardi'll have to picture the screen when I get the data up to share the panic error17:40
RoyKnetconsole should work, though17:40
tomreynRoyK: you'll just need to add the virtualized equivalent of a serial console switch17:40
RoyKbut then again - if a VM panics just after iptables -Ln, something is very, very wrong17:41
RoyKtomreyn: something you can't do if someone else controls the hypervisor17:42
tomreynRoyK: right17:45
masonI'm curious how libvirtd presents microcode to VMs. Which is to say, with Ubuntu 18.04, my qemu-kvm VMs show vulnerable to Spectre 3a and 4 where the host does not. Guests tried include Ubuntu 16.04, RHEL 7.5, Debian 9.5.20:46
JanCI doubt showing microcode to VMs makes any sense at all20:47
masonJanC: The facility has to be there in the CPU presented, no?20:48
masonOkay, could you explain how it's supposed to work, then?20:48
ahasenackmason: can you setting up the cpu in the vm to be a copy of the host?20:50
ChmEarlmason, qemu-system-x86_64 -cpu ?  , then try some CPUID/CPU combinations20:50
ahasenackI don't know how to do it with virsh, but in virt-manager you have a dropdown menu in the cpu tab20:50
masonI've tried a couple CPU combinations, all with IBRS.20:50
JanChow microcode & all those controller CPUs work isn't really documented20:51
JanCso there really isn't any way you could implement them virtually20:51
masonWell. Microcode application, right. But the guests themselves should believe they've got the correct facilities, as I understand it.20:52
JanCso, did you test if those guests are actually vulnerable?20:53
masonJanC: Yes. That's why I'm asking.20:54
JanCI mean, did you test if they are actually vulnerable, or did you just test a kernel flag or something?20:55
masonJanC: I ran https://github.com/speed47/spectre-meltdown-checker which I believe looks at more than flags. I'm open to suggestions for testing with PoC somehow.20:56
ahasenackrbasak: still around?20:56
masonIf it's a false positive, that's fine.20:56
masonI'll migrate the guests to a RHEL hypervisor and test there too I guess, for comparison.20:57
tyhicksmason: hi - variant 4 mitigations are only enabled when there's a seccomp filter in place for the process in question or the process has opted into mitigation by making a certain prctl(2) systemcall20:58
tyhicksmason: this is documented in the Mitigations section of https://wiki.ubuntu.com/SecurityTeam/KnowledgeBase/Variant420:59
tyhicksmason: as far as variant 3a goes, that's something that is entirely mitigated in microcode20:59
masontyhicks: Thank you. Odd that the spectre-meltdown-checker script doesn't set this. I'll have to read what they're doing.20:59
tyhicksmason: VMs don't load microcode - it is something that the host is in charge of20:59
masontyhicks: That seems fine, but I thought the CPU then presented would have all the bits needed for the kernels to mitigate things. I'm clearly a bit confused.21:00
tyhicksmason: there's a lot of different ways that your virtual CPUs can be configured, you'd need to paste the contents of /proc/cpuinfo for me to start understanding what's going on inside your vm21:01
masontyhicks: kk, coming up21:01
ahasenackrbasak: n/m, filed a bug :)21:01
tyhicksmason: going back to variant 4... you can use the ssbd-exec tool that I wrote to enable mitigations and then run the checker script (replace the grep command in the second example with the checker script): https://github.com/tyhicks/ssbd-tools#using-ssbd-exec21:02
masontyhicks: https://bpaste.net/show/e19b21dd662e - the first one is https://bpaste.net/show/e5b63791427f on the host, and the second is https://bpaste.net/show/154b82e83888 on the host21:03
masontyhicks: ty, will try that21:03
tyhicksmason: I'm starting to get cloudy on the details but the kernel reports a bogus microcode revision inside of VMs (which is why you see 0x1 reported as the microcode revision in the cpuinfo pasts)... I wonder if the checker script is relying on that revision to be correct...21:06
* tyhicks checks the source21:06
JanCmason: microcode is never enough to prevent all spectre-style vulnerabilities AFAIK21:07
masonThere's one of the more recent ones that requires guest cooperation, yeah.21:07
masonI want want the fence to be as high as possible.21:08
JanCor if it would be enough, it would likely slow down the CPU to the point where you could be using your smartphone instead of that high end Xeon  :P21:08
masontyhicks: # ./ssbd-exec -- ~/spectre-meltdown-checker.sh21:09
masonERROR: Speculation cannot be controlled via prctl21:09
masontyhicks: That's on the second system from that initial bpaste, which has a Skylake Xeon box as a host that shows all green with the script.21:11
tyhicksmason: the correct command is: $ ./ssbd-exec -p disable -- ~/spectre-meltdown-checker.sh21:12
tyhicks(but I think you're going to hit the same error)21:12
masonah, trying again21:12
masonClose: https://bpaste.net/show/4efbd9fc3bd821:12
tyhicksmason: what kernel version are you running? (cat /proc/version_signature)21:15
masontyhicks: Host or guest?21:15
masontyhicks: I'll give you both.21:15
masontyhicks: 3.10.0-862.14.4.el7.x86_64 (RHEL 7.5, guest) seems not to offer this. The host is 4.15.0-36.39-generic.21:17
tyhicksmason: did you run ssbd-exec in the guest or host?21:17
masonI didn't bother running it on the host, because the host makes the script happy.21:18
tyhicksmason: oh, RHEL must not have backported the fix21:18
tyhicksmason: err, they must not have backported the prctl portion of the fix21:18
masontyhicks: I'll try again on an Ubuntu guest.21:18
JanCthere is a big spread between fixing those vulnerabilities entirely and keeping your CPU performant  :)21:31
JanCI suppose at some point there will have to be host & guest profiles where admins will have to make their choices...21:32
tyhicksmason: that checker script has become pretty complex - I can't say for sure but I'm pretty confident that the variant 3a failure is a false positive most likely caused by the microcode revision being obscured inside of the VM21:33
masontyhicks: I'll look around for other ways to test - a PoC would be the best way I imagine.21:35
JanCtesting 8.04 guest on 8.04 host would probably be a good test too21:37
masontyhicks: Thank you very much for your time. Also JanC and ahasenack21:37
masonJanC: Oddly, I don't actually have an 18.04 guest, only hypervisors. I'll have to spin one up.21:38

Generated by irclog2html.py 2.7 by Marius Gedminas - find it at mg.pov.lt!