=== freyes_ is now known as freyes [05:14] good morning === Nafallo_ is now known as Nafallo [06:08] Good morning === ashleyd is now known as ashd [12:15] hello [12:15] i have just installed ubuntu-server 18.04.1 with mass controller [12:15] with default settings with lvm [12:15] it created a 4GB root partition and 16GB /dev partition [12:16] why?? [12:16] the /dev usage is 0%, and root partition is already full [12:18] boritek: can you paste the url returned by "df -h | nc termbin.com 9999" [12:33] boritek: can you paste the url returned by "df -h | nc termbin.com 9999" [12:39] boritek: that's a known bug [12:39] let me fetch it to show you [12:40] boritek: https://bugs.launchpad.net/subiquity/+bug/1785321 [12:40] Launchpad bug 1785321 in subiquity "LVM Entire Disk option does not use entire disk" [Undecided,New] [12:40] if you want lvm, for now you should create the layout manually [12:40] or install with 4Gb, and resize later [12:41] maybe add an LV for /usr, then you won't need to resize / [12:41] that can be done without booting into rescue mode, since you won't be resizing / [12:43] cpaelzer: do you know if this d/t/control syntax works for Depends? : [12:43] squid3 | squid, [12:43] n/m, it does, it installed squid [12:45] hah [12:46] that kylin test is failing because it can't find python-imaging:amd64 [12:47] last uploaded to precise :/ [12:52] ok, it became pillow [12:59] ahasenack: yes it can have alternatives, but you foudn that already [13:00] I'm unsure what happens if both are available, but as long as there is one of the two it should work [13:00] the ordering matters [13:00] at least in d/control [13:00] should be the same [13:01] ack for d/control behavior, and I'd hope so as well - but expect/hope != knowledge, so I'm still unsure :-) [13:06] Hello, does anyone here familiar with mysql on ubuntu server? I have problem setting mysqld.cnf on /etc/mysql/mysqld.conf.d/ , I set sql_mode = '' but it won't affect the sql server, what cause it? I think I've set the write things. Is there anyone could explain maybe, or maybe I'm wrong. Thanks [13:07] I have no idea, sorry [13:07] for starters I don't even know what's the "good" config file, there are so many under /etc/mysql [13:08] It works like regular .d/ directories [13:08] Drop any number of .cnf into /etc/mysql/mysqld.conf.d/ to customise your configuration [13:09] Changing the supplied ones will result in conffile prompts (may be good or bad depending) [13:09] Use /etc/mysql/conf.d/ for things that affect the client (eg. libmysqlclientX but also libmariadbclientX). That's therefore shared between MySQL and MariaDB [13:09] And /etc/mysql/mariadb.conf.d/ for MariaDB-specific configuration. [13:09] (on the daemon side) [13:10] As for sql_mode specifically, I don't know, sorry. [13:11] The only reason I can think of that it won't affect the server is if you've messed with other config files in /etc/mysql/*.cnf or if you're actually using MariaDB. [13:12] You should be able to trace the config files that are active by following through from the service configuration. [13:12] rbasak: maybe ordering in the dir, and a later conf file in ther eoverwriting his change? [13:12] That could do it. [13:12] By default I think we only ship one file in the dir though? [13:13] rbasak: Yep, I've but it's not reading it I dunno why, I means I set some param and still when the service is running the configuration doesn't change [13:13] There's a parameter you can call to mysqld to have it print the configuration it's picked up I think. [13:14] rbasak: I use MySQL-server package [13:14] rbasak: aaaa, I don't get what it means with your last statement, could you explain it to me? === ashleyd is now known as ashd [13:50] rbasak: I want to ask, anything under /var/mysql/mysql.conf.d/ is loaded by my.cnf right? [13:51] benl90: should be, by default, yes. [13:51] rbasak: Then what means with ! and without ! [13:51] If you have mysqld installed [13:51] And you don't have mariadb installed [13:51] rbasak: yes.. I've mysqld. mysql server 5.7.23 [13:51] A common error is to have tried to install MariaDB and then not removed it completely. [13:52] rbasak: !includedir /etc/mysql/conf.d/ means it load from the folder right? [13:53] Yes [13:53] rbasak: Hmm.. why it's not working, when I force write on the my.cnf it's working -_- [13:54] I don't know, sorry. Can you provide steps to reproduce the problem on a fresh installation? [13:57] rbasak: Yep, I tried to edit the mysqld.cnf on the mysql.conf.d folder [13:58] rbasak: then after that, I add sql_mode = '', after that I restart the service and check using SELECT @@sql_mode; still, the mode is there and not overwrited. [13:59] Skuggen: you might know off the top of your head? ^ [14:00] benl90: have you verified this still happens on a fresh installation? [14:08] rbasak: Hmm... Anyway how to delete the mysql and it's configuration and start from scratch in ubuntu [14:13] benl90: depends on what's wrong, and we don't know that. You can try https://askubuntu.com/a/1062707/7808, but that's no replacement for trying to reproduce on a fresh installation. You can use a container or a VM, so this should not be difficult. If you're not familiar, it's good to learn anyway - being proficient with lxd containers massively speeds up so many sysadmin tasks, because you can try [14:13] something on a fresh Ubuntu very easily. [14:20] rbasak: ahasenack: did you experience on git ubuntu submit that you got rejected for the branch not being found? [14:20] it is correctly pushed [14:21] I think on a totally new branch LP sometimes takes a while for processing [14:21] later on things work [14:21] I never used git ubuntu submit [14:22] so you open the MPs just manually then [14:22] well that is my fallback when it fails [14:22] I was just wondering if we should print something (if it is a geenral issue) [14:23] yeah - merge ID hattrick [14:23] I don't use submit either [14:23] thanks postgres MRE :-) [14:23] oh, no wonder I'm the only one facing it then :-) [14:26] I think this is the fix for the ubuntu-kylin-sso-client dep8 tests: https://pastebin.ubuntu.com/p/C872JtdJD3/ [14:26] at least one of the fixes [14:27] because: [14:27] 2018/08/14 10:11:02| ERROR: cannot change current directory to /tmp/autopkgtest.DUUfBX/build.7Km/real-tree/_trial_temp/squid/spoolsquid: (2) No such file or directory [14:27] but we'll see [14:28] ack on the added / [14:28] but would that dir then exist? [14:28] I mean the path is broken without the / for sure, but it might still get a "no such file or dir" [14:28] if no one created it [14:29] ahasenack: FYI the postgresql MPs are up for you [14:29] ...../squid/spool exists [14:29] it is created if it does not exist [14:29] I don't know yet if squid itself will create the missing bit, maybe not [14:30] the argument is called "dir" [14:30] so if it would create on demand it would have created the dir "spoolsquid" [14:30] agreed [14:31] I know where to look, I'll troubleshoot that later [14:31] let me get the finishing touches up for squid itself and put that up [15:27] Quick question for you guys. So I am trying to create a network install script for about 100 servers. Pxe boot and all. I have a ubuntu livecd image that I moved to pxe and it can netboot into a livecd and ssh in. Then I have some scripts that check for hardware like cpu / ram and configure LSI raid arrays. Then I want to automatically install ubuntu after that. [15:27] I want to be able to launch ubuntu installer with kickstart file after lsi raid arrays are configured automatically. How would I acomplish this? [15:28] I don't want to "netboot" and pass KS=bla for kickstart file and install that way. I want to install from the livecd after its done running my scripts. [16:10] jonfatino: why not run the script as part of the kickstart? [16:31] anyone fluent in how systemd units work and 'network readiness' within SystemD units can be determined so you can decide whether to start a service or not? https://bugs.launchpad.net/ubuntu/+source/nginx/+bug/1786974 is related to some race condition where nginx starts before IPv6 is ready, and it causes problems, but that's a SystemD race condition if i'm not mistaken? [16:31] Launchpad bug 1786974 in nginx (Ubuntu) "Nginx is starting before IPv6 address is ready" [Undecided,New] [16:35] leftyfb: good idea but ideally we want to do it in stages and report back to a centralized database for deployments [16:36] then proceed with next step and install [16:36] jonfatino: that could be done [16:36] jonfatino: just gotta script it as part of the kickstart [16:37] The tools needed for megacli and a few audits / stress test and what not are not part of netboot image. [16:37] I'm trying to find the exact commands ubuntu uses to run the install script so I can start it manually or when ever I want [16:42] jonfatino: maybe start looking into subiquity [16:43] leftyfb: have you ever used debootstrap? [16:44] I have transparently. But not really [17:05] teward: systemd.special(7) -> network-online.target [17:07] tomreyn: would that solve a race condition like bug 1786974 seems to be having? It was my understanding that you should require network.target not network-online.target [17:07] bug 1786974 in nginx (Ubuntu) "Nginx is starting before IPv6 address is ready" [Undecided,New] https://launchpad.net/bugs/1786974 [17:07] (so i'm trying to clarify before doing anything) [17:07] thanks by the way [17:07] teward: i dont know that, sorry [17:08] hmm [17:08] i'll have to prod further, but thanks for the pointer, tomreyn. I'm not 100% fluent with SYstemD :P [17:08] i was actually wondering how this mechanism in systemd prevents / handles race conditions, but never spent time on researching it. [17:08] i'm neither, just catch things up here and there [17:10] teward: if the user binds to something else than [::]:80, adding the network-online.target dependency makes sense [17:10] sdeziel: their error is :::80 [17:10] theirs is erroring with a default setup it seems by the error msg [17:11] though since they didn't file the bug the proper way I have no config data to use [17:11] the IPv6 address was obfuscated [17:11] sdeziel: whether it's a link local or not is unclear, and probably would help determine the answer and solution [17:12] binding to the wildcard shouldn't fail even if the network isn't online [17:12] sdeziel: the question is whether the fix for this should be applied on the one user's system or as part of the package. [17:12] sdeziel: it can if v6 is disabled which some users are doing (stupidly I might ad) [17:12] sdeziel: if it makes sense to bind to network-online.target that's one thing, if we're running a default install should we be delaying until network-online.target *just* to handle edge cases [17:13] since the default nginx config runs on [::]:80 (and also 0.0.0.0:80) [17:13] teward: right, that's good question. The man page recommends to stick with the network target for networking daemons [17:13] sdeziel: then the answer is "Won't Fix" for the bug, because we already want on network.target [17:13] at least as i understand the systemd unit in use [17:14] by nginx [17:14] DAMN KEYBOARD! *throws keyboard into the river and goes to get a spare from the supply bin* [17:14] teward: that would be my feeling as well but I'd check if the user's config does indeed specify a specific IPv6 [17:14] teward: note also that there's also 'network.target', and that systemd.special(7) states "Usually, network.target is part of the boot of most systems, while network-online.target is not, except when at least one unit requires it." [17:15] so maybe network.target is the better choice, if its good enough [TM] [17:16] tomreyn: indeed. that's what the service uses currently [17:16] but it seems that this 'race condition' is not new [17:16] and I did note that bit in the manpage that tomreyn pointed out [17:16] i'm sure tomreyn appreciates this [17:17] :P [17:18] sorry i'm still recovering from illness, i'm confusing who i'm messaging :| [17:18] ;-) no worries, i'm just having fun here [17:21] sdeziel: the only reason I'm hesitant to tell the user to alter their service file for systemd is because that gets clobbered next-update whenever a package is updated with a newer version IIRC [17:21] then they'd have to reapply that workaround again [17:29] teward: not with "systemctl edit nginx" overrides [17:30] teward: the user then only need to put "[Unit]\nAfter=network-online.target\n" in there and be done with it [17:31] see, now I'm learning more about SystemD than I would on my own heh [17:31] sdeziel: if they come back and indicate they're using an actual v6 address then I'll provide that workaround, but document that the bug itself won't be fixed in the package because network.target is the 'typical target' for networking daemons... [17:32] assuming we include web servers in the category of networking daemons? [17:35] teward: the scripted way would be: https://paste.ubuntu.com/p/YfD5TYdw2y/ [17:36] sigh, both After= and Wants are needed: https://paste.ubuntu.com/p/m64cXMRbhP/ [17:37] teward: see https://www.freedesktop.org/wiki/Software/systemd/NetworkTarget/ [17:37] teward: waiting for network-online.target is incorrect [17:37] rbasak: that's what I had assumed === LastTalon_ is now known as LastTalon [17:38] https://bugs.launchpad.net/ubuntu/+source/apache2/+bug/1786675 came up the other day [17:38] Launchpad bug 1786675 in apache2 (Ubuntu) "Apache will not start on boot if bound to a fixed IP address" [Undecided,Invalid] [17:38] rbasak: so we should consider this a "Not a Bug" and blame a race condition between the software and the underlying networking management system/infrastructure? [17:39] teward: the reporter of the nginx bug report says "...emerg] bind() to [xxx]:80 failed" so I think they're binding to a specific IP. [17:39] which i'm asking them to confirm in the comment I made [17:39] For Ubuntu, I believe this is Invalid/Won't Fix because the correct thing to do if customising local configuration is to also override systemd as discussed above. [17:40] For upstream, I think it's a valid feature request that daemons handle this gracefully [17:40] rbasak: ACK. I'll have to add that to our Server Team documentation. [17:40] (for the override part) [17:40] The upstream systemd page should describe it well enough for upstream - either use IP_FREEBIND on Linux, or better, respond dynamically to network configuration changes. [17:43] thanks rbasak, i've marked the bug accordingly. [17:43] rbasak: while you're around, can you confirm that my last message to the list regarding the proposal to jump to NGINX mainline at least for 18.10 and 19.04 made its way to the mailing list? [17:44] i have no confirmations on my end whether it did or not :| [17:44] ah there it is in the archive, nevermind [17:44] (it wasn't showing for me, I blame caching) [17:44] https://lists.ubuntu.com/archives/ubuntu-server/ [17:44] rbasak: yep, it wasn't showing in there, I blame caching :P [17:45] initially I thought I was stuck in the mod queue again heh [19:28] what do you guys do to store passwords for servers/apps/ i.e. admin passwords for mysql ; passwords for basic auth for sites etc; i.e passwords taht are not your user password [19:28] right now i have a pw.txt somewhere on my laptop ; but wondering if theres a better way [19:31] keepass database with the passwords in them. Or a PGP encrypted text document synced to several locations that contains passwords and requires my exact private key to decrypt [19:31] also 1password but that's just me :P [19:32] arooni: ^ [19:33] hmm i already use lastpass [19:33] i wonder if i can have a text doc there [19:33] -1 because LastPass has had breaches in the past :P [19:33] it just seems sloppy to have it in text format; even though i connect to the server with ssh keys etc [19:33] keeping it encyrpted as a document seems smart too; i have dropbox [19:34] is there an easy way to decrypt/encrypt from command line or apps on ubuntu/mac [19:35] gpg --encrypt --armor -r YOURKEYID foo.txt [19:35] makes foo.txt.asc [19:36] you can decrypt that later with gpg --decrypt foo.txt.asc [19:36] just make sure you don't lose your privkey :p [19:36] truth [19:36] and don't share the plaintext version across cloud storage either :P [19:36] i kinda like storing them in lastpass; because i already have 2 factor auth setup there [19:37] up to you :P [19:37] thx for the brainstorm [19:37] good talk ;P [20:45] hello [20:48] I have a Digital Ocean Ubuntu 16.04 server and SSH (PuTTY) was reporting the host key changed. So I replaced the keys and shut off SSH access. Chkrootkit says Possible Linux/Ebury - Operation Windigo installetd [20:48] Reviewing the configuration doesn't seem to show it is infected but I want to be sure [20:49] Is Possible Linux/Ebury - Operation Windigo a false positive on Ubuntu 16.04 as I've read from several sources? [20:50] Chkrootkit also lists fail2ban as suspicious files [20:53] last time I looked at one of those rootkit tools I found it was prepared to label the upstart /sbin/init as malicious [20:53] it's probably best to just ignore all those [20:56] sarnold, I see no other IOC on there and have scanned several other ways, locally, remotely, etc. [20:57] sarnold, Why would a host key change? [20:57] sarnold, would a patch do that or if Digital ocean made an infrastructure change? [20:58] evit: I think I've seen instances where a host brought up with just an rsa key will generate an ecc-based key on a later reboot, and thus suddenly have a *new* key. I dont know why :( [20:58] How did you get to the server in order to replace the host keys? [20:58] rbasak, Via the web console login [20:58] rbasak, I've since disabled SSH entierly [20:59] and replaced they keys [20:59] If the host key appears changed, then that's a sign of a MITM, not of a host compromise in itself. [20:59] It seems unlikely an attacker would change the real host keys, since there's nothing to gain from that if somebody already has root. [21:00] (and it'd reveal the presence of the adversary) [21:00] So if I changed they keys and it looks clean should I be confident? [21:00] You should properly check the host key fingerprints before accepting them. === miguel is now known as Guest77219 [21:00] yes, that is why I re-keyed [21:04] This article states that ssh -G is an IOC. https://www.welivesecurity.com/2014/04/10/windigo-not-windigone-linux-ebury-updated/ [21:04] Is that correct? A lot of other sources say otherwise and I'm not sure what to think... [21:06] Eventually, I will rebuild with public key auth and OTP MFA [21:09] Should SSH -G be considered an indication of Windigo compromise or is that normal behaviour === guntbert_ is now known as guntbert [21:12] evit: i'm not sure an article from 4 years ago is relevant to security concerns today [21:12] evit: also -G is a valid `ssh` option, and they are referring to running it on the compromised system in particular, i think [21:13] nacc, yes, agreed. Seems like they are assuming that is an IOC [21:13] evit: in 2014. [21:13] evit: which is not the current year, you know :) [21:14] nacc, =P [21:15] nacc, I saw my host key change and I freaked out. I'm not engaging in baseless fears here. Either MiTM, host compromise or some system change at Digital Ocean [21:19] evit: https://bugs.launchpad.net/ubuntu/+source/chkrootkit/+bug/1488256 [21:19] Launchpad bug 1488256 in chkrootkit (Ubuntu) "chkrootkit 0.50 shows Windigo/Ebury false positive due to new '-G' option in openssh 1:6.9p1-1" [Low,Confirmed] [21:19] ahasenack, Thank you! I just wanted the wisdom of the channel to clarify I'm not going nuts. =0 [21:20] that doesn't answer why the ssh host keys changed, though [21:21] ahasenack, I figured it was MiTM but traceroute says otherwise. [21:23] ahasenack, What other factors can change host key? [21:38] evit: the 'host changed it' seems pretty likely to me if it's a VM and you have not taken measures to prevent them from accessing it. [21:39] i know of a couple cloud hosters who insist on having file system accesss not just to containers but also to VMs. [21:39] (and may not tell customers about this practice) [21:40] i do not know baout how DO handles it [21:40] tomreyn, Since they have physical access and its a cloud instance I don't think anything I do will prevent them [21:44] tomreyn, Are you saying I should encrypt the cloud instance? [21:44] evit: right, you cannot prevent it. but you can raise the bar, make it harder for them, break their default processes, and increase your chances to know this is what happened when it happens. [21:45] doing so also makes it harder for them to provide a good service to you, though, so YMMV [21:46] tomreyn, The cloud is often worse than self hosting. The beancounters are bad at math and security. =P [21:47] i'd still prefer own hardware whereever there is risk. [21:47] (and ideally own network, too) [21:50] tomreyn, I'd prefer too. I can own it if I mess it up! =P [22:12] Hello, can anyone explain me why mysqld.cnf is readed by mysql but none of the directive is followed and set globally on mysql? Thanks [22:13] I have moving through ubuntu server documentation, no luck, and I'm on mysql 5.7.23 on ubuntu 18.04. Is there maybe opened issue about this? Thanks. [22:17] on my ubuntu 16.04 with mysql 5.7.13 it's honoring the configuration when I run sudo mysqld --print-defaults it will show the additional parameter that set by editing mysqld.cnf [22:19] benl90, the newer mysql reads the config files is a certain order, including whats in the /etc/mysql/conf.d folder [22:19] its probably not seeing your file at all [22:19] compdoc: Wait a second... I will look into it again. compdoc it's reading, when I accidentally make an error on configuration, it won't start. It's confusing hahaha... [22:20] ah. yeah, it wont start if you get it wrong [22:21] compdoc: Yep, that prove that the conf is readed right? [22:21] yes, I would think so [22:22] compdoc: ah the rule that written on all my.cnf and it's sub dir should be run as global variable right? [22:23] I think so [22:24] compdoc: It seems not honoring my cnf... on show variables; it won't change... still same -_- [22:25] I think theres a mysql help channel [22:26] compdoc: #mysql? [22:27] yup [22:27] thanks compdoc. Let me checkout :) === kevr_ is now known as kevr [23:19] I have question how to clean all configuration of an removed packaged on ubuntu [23:21] I tried purge nothing happen [23:26] benl90: what package and what did you see before and after to indicate "nothing happened"/ [23:26] benl90: and what version of ubuntu, etc. [23:29] nacc: Mysql-server, on ubuntu 18.04. I tried to fix broken packages, because on this server mysql doesn't obey mysql.cnf, it frustate me alot, I dive through documentation on mysql and mysql board no bug or such, also I've other sevrer that working perfectly with same installation and same conf [23:31] benl90: ok and what did you purge? [23:31] nacc: mysql-server? Let me try dpkg -P [23:31] benl90: well, that won't be what you want, that's just a metapackage [23:31] it doesn't own any conf files [23:32] nacc: Then how to delete the conf and completely clean reinstall it. I really desperate of it :'( [23:33] benl90: you need to purge the actual package in question (the one that mysql-server depended on) [23:33] benl90: mysql-server-5.7 i think [23:34] nacc: it show this Purging configuration files for mysql-server-5.7 (5.7.23-0ubuntu0.18.04.1) ... dpkg: warning: while removing mysql-server-5.7, directory '/etc/mysql/mysql.conf.d' not empty so not removed, then how to clean it all [23:35] benl90: is it empty now? if it's not empty now, recursively delete the directory? [23:35] not empty, the folder also not empty /etc/mysql [23:36] benl90: well, of course /etc/mysql is not empty /etc/mysql/mysql.conf.d exists [23:37] ah then how to clean it. or it's there since I do fresh install [23:37] ? [23:37] benl90: you can use `dpkg -S` to see what, if any package, owns the paths in question [23:37] if none do, you can safely (imo) remove the directories. Reinstallation will recreate them. [23:37] rbasak: --^ may know better than I, though [23:37] nacc: No path found, so it will be safe to remove them? [23:40] nacc: this happen https://privatebin.net/?1440a70e4ed74582#0GP3btqDOTFTo8GTdXffcpP3kjf1fv+rnSDZHFl1vlc= [23:40] benl90: i think so; i mean, it seems like you don't want mysql on your server [23:40] nacc: no, I want to clean reinstall it, because it have wierd behavior [23:40] benl90: right, so you 1) want to fully remove it first, 2) properly reinstall it? [23:41] benl90: it's not particularly helpful to truncate the commands you ran [23:41] nacc: I've using apt install right? [23:41] Make sure mysql-common is purged before cleaning /etc/mysql/. [23:41] benl90: i have now way of knowing that :) [23:42] Since it uses update-alternatives, it won't show up in dpkg -S [23:42] benl90: as you didn't paste the command you ran [23:42] rbasak: oh right, sorry! [23:42] But apart from that, if it's purged, I think it should be safe to wipe it, since anything that uses /etc/mysql depnds on mysql-common and you can't purge mysql-common unless they are also removed. [23:42] okay my command is apt install mysql-server [23:42] rbasak: right, that's what i was thinking, just had the wrong package in mind [23:42] However, make sure other packages aren't shipping in /etc/mysql that are removed but not purged, like nacc said I think dpkg -S will do it but I'm not sure. [23:43] benl90: so the problem is probably, currently, you still had mysql-common installed [23:43] benl90: and so mysql-server, which depends on that, broke a bit :) [23:58] nacc: SUCH A WOW... it works.. seems the mysql-server package broken that cause wierd beavior