=== kallesbar__ is now known as kallesbar === kallesbar_ is now known as kallesbar [05:18] I'm getting a strange error on ubuntu bionic server install; it's attempting to run zpool and failing because it's not installed [05:20] https://pastebin.com/CpiyUrMy [05:20] anyone else run into this? [05:20] relevant line starts at 162 [05:26] NickZ: at a guess, you have an existing zfs pool there and curtin doesn't know how to handle it. Disconnect the drives? [05:31] nope, these are completely clean drives [05:32] manually partitioning the drives prior to install seems to resolve the issue === cpaelzer_ is now known as cpaelzer [06:30] <^Squirrel^> hi, can I get help for ubuntu gere? [06:32] !ask | ^Squirrel^ [06:32] ^Squirrel^: Please don't ask to ask a question, simply ask the question (all on ONE line and in the channel, so that others can read and follow it easily). If anyone knows the answer they will most likely reply. :-) See also !patience [06:33] <^Squirrel^> I have a folder /proc/4838/cwd that is 9.4G, process 4838 identified as Samba - I think it is killing my server - is it normal that it should be so big, and if not, how to I get it back to normal size? [06:33] <^Squirrel^> my server being the server on which that ubuntu runs, of course [06:36] Good morning [06:37] <^Squirrel^> alternative question: I have a partition for /, one for /home and one for /var. I would like to find the larger files for the / partition, a way that doesn't scan the other (mounted) partitions even if linked to it - is there a way to do that? [06:37] <^Squirrel^> morning lordievader [06:39] Hah, just ran the 16.04 -> 18.04 upgrade on a server with molly-guard installed, and molly-guard cancelled the automatic reboot afterwards [06:40] ^Squirrel^: `du -x` skips directories on different file systems, as per the man page. [06:42] <^Squirrel^> thanks [06:42] <^Squirrel^> so "du -x /" is what I should run? [06:52] <^Squirrel^> ok that works, still doesn't tell me what is filling my 10G of /root folder [06:52] <^Squirrel^> folder/partition [06:56] rbasak: are you planning on working on that mysql merge? [06:57] rbasak: or is lars doing it? [06:58] mdeslaur: we're taking care of it. Lars has a branch up for review. I'm hoping to get to it today, but I'm not sure. [06:58] rbasak: ok, cool, thanks! [07:04] <^Squirrel^> lordievader, is there a way to check what size the folders are (the sum of the files in that folder and subfolders) on a server (no GUI) [07:07] ^Squirrel^: du. See the du manpage for details of -s and -c. [07:07] Or you can use baobab on a desktop machine connecting over ssh. [07:07] ^Squirrel^: du -shx /* may be useful [07:08] <^Squirrel^> thank, will try. Root space free is 0 I can't install anything new [07:08] <^Squirrel^> I have ssh access too yes [07:13] du is installed by default. [07:15] <^Squirrel^> rsalveti, sarnold, I use these thanks [07:15] <^Squirrel^> lordievader, yes, of course [07:15] <^Squirrel^> this is weird [07:16] <^Squirrel^> df reports that my root partition is 0, but sudo du xhc reports 3.6G is used. / is a 10G partition [07:17] <^Squirrel^> I am using LVM, can it be the reason? [07:19] Might simply be removed files in use. df sees those, du doesn't. [07:22] <^Squirrel^> so what does "removed files in use" mean? [07:23] <^Squirrel^> I have rebooted the server, this partition is still full [07:23] <^Squirrel^> that's 5G worth of dark matter [07:24] ^Squirrel^: no, lvm is not the reason. Look for unlinked files still in use by opened processes. [07:25] a reboot is a good way to fix that though :/ [07:25] <^Squirrel^> I rebooted already, twice, to no avail [07:25] <^Squirrel^> jelly, how do I do this? [07:25] lsof or fuser are the usual tools [07:25] you might have files hiding under a mountpoint below this one [07:25] lsof -n +L1 [07:26] jelly: ooh +1 [07:26] but if you rebooted that would have cleared the processes [07:26] <^Squirrel^> no output of this lsof [07:26] <^Squirrel^> lsof -n +L1 [07:27] how many mountpoints do you have other than / ? [07:27] with real filesystems or bind mounts [07:27] (also, doing anything with /* is bad form, avoid) [07:28] <^Squirrel^> expcluding tmpfs and udev, 16 [07:28] umount everything else under / and retry du -x / [07:28] jelly: how else would you get a list of all top-level files and directories? [07:28] siwtch [07:28] <^Squirrel^> in reality, about 8 I explicitly mounted [07:28] sarnold: ls -ld, not du [07:29] or echo [07:29] <^Squirrel^> including /var [07:29] <^Squirrel^> and /home [07:29] jelly: ls and echo do not summarize the sizes of all files linked in all directories underneath those top-level directories [07:29] sarnold: what use are sizes of files in /proc [07:30] jelly: good point. [07:31] <^Squirrel^> ok jelly, this might be getting somewhere - umounted -a then I get a total which corresponds more to df outuput [07:31] (procfs is slow, in might take half an hour just traversing that on a busy system) [07:33] du can very well traverse directories on its own, no need to help it with * [07:34] ^Squirrel^: look at directory contents previously hidden under other mountpoints [07:35] ^Squirrel^: btw, once you've got some space to install new packages, ncdu may be to your liking :) [07:35] <^Squirrel^> I tried to install ncdu, can't :P [07:36] <^Squirrel^> but once I find the problem here, I will [07:36] <^Squirrel^> I need to create a root user now - I can't unmount /home otherwise [07:36] I have a problem, I have a 16.04 server that lose all network connectivity from time to time. I can see the DHCP addresses in ip a, but I cannot ping it and it cannot ping anything [07:37] sure you can, just get your CWD ass out of /home [07:37] Network come back with sudo systemctl restart networking [07:37] (but having root is always nice, for other reasons) [07:38] <^Squirrel^> lol I could just CWD out indeed [07:38] or as they call it in sh, "cd" [07:38] lsof -n|grep /home [07:39] <^Squirrel^> but I can't umount home, cause I'm still logged into a user using /home... [07:45] <^Squirrel^> ok /home umounted... [07:47] <^Squirrel^> jelly, du -x / still reports some of the umounted folders... [07:48] <^Squirrel^> yup. that's it :) [07:48] <^Squirrel^> I knwo the problem now [07:48] <^Squirrel^> Yay! [07:48] <^Squirrel^> thanks guys! [07:50] <^Squirrel^> Solved! [07:52] <^Squirrel^> thanks jelly especially :) [08:10] What can I do to investigate losing connectivity after a reboot? [08:11] The system has the correct IPs, but there is no connectivity at all without rebooting networking [08:11] from syslog and apache, I can see the machine rebooted around the time of the problem [08:15] sylario: maybe the dhcp server wasn't working by the time your system booted up again? [08:15] oh you saiid it got the correct ips, sorry [08:15] It's some kind of xen hosting I think [08:16] so does resolving not work then, or are remote (and what about local) targets not reachable if addressed via ip address either? [08:17] * lacal targets, such as you gateway [08:18] I have access via an emergency console, I cannot ping google or the french taxes office from the problematic server, I cannot ping any of the server's IPs [08:18] sudo systemctl restart networking solve everything [08:18] the servers' LAN or WAN IPs? [08:19] and where from [08:19] It has only public IPs, not addresses in private ranges [08:21] ok so i'm wondering whether while this problem occurs, you could, from the server, ping 1.1.1.1 or 8.8.8.8 or the servers' gateway (as listed by ip route | grep ^default) [08:21] I try to ping it from public IPs from two different ISP, and I connect on it with an emergency console provided by the hosting company [08:21] I did not try that [08:22] in case you wont be able to ping the gateway when it happens again, talk to your hosting company. [08:23] in the meantime review /var/log/syslog to get a better idea of why the network didn't come up fully after the reboot. [08:27] tomreyn: thx for your help, i'll try that as soon as I can test on the server. [08:27] It will go in the next ticket [12:18] hello! I just rented an Ubuntu VPS but I see some weird traffic spikes, I wondered if any one had some recomendations for some good traffic montioring tool, preferably that saves traffic so I can see when traffic spiked and what caused it, a web gui would be nice but terminal works too. preferably open source [12:20] vimes: perhaps ntopng is the tool you want [12:21] thanks! I'll check it out sarnold [12:58] vimes: iftop for realtime viewing. Not sure about looking later though. It would make sense for it to have a pcap replay feature but I don't see that. === cpaelzer_ is now known as cpaelzer [16:31] jamespage: heads up on possible upgrade path bug for keystone, dont know how to add ubuntu tracking [16:31] https://bugs.launchpad.net/keystone/+bug/1793347 [16:31] Launchpad bug 1793347 in OpenStack Identity (keystone) "keystone upgrade fails q->r oslo.log requirement to low" [Undecided,New] [16:34] coreycb: ^ [16:43] are auto-upgrades enabled by default? I have a vm that was provisioned and it was, and I'm trying to determine if this is a provisioning problem or a distro problem. [17:00] bin_bash: I don't know for sure, but you can easily find out by spending the 1 minute it takes to do the server install from usb onto a device [17:00] sure if i had a device to install it on [17:00] but i dont so i'm asking here [17:10] jamespage: coreycb and another possible one https://bugs.launchpad.net/nova/+bug/1793353 [17:10] Launchpad bug 1793353 in OpenStack Compute (nova) "broken upgrade path q->r requirement for oslo.db" [Undecided,New] [17:13] bin_bash: Virtual Box is a decent way to test things with temporarily [17:17] based upon a bionic lxd, it seems like some sources are enabled. That is the cloud image, though, not necessarily the same as the base server install [17:19] I thought server enabled root and -security by default... I just can't find the documentation on it. [17:22] pragmaticenigma: sure if that was a viable option, but it's not unfortunately. [17:22] nacc: that's pretty terrible... [17:23] bin_bash: what do you mean by terrible? [17:23] pragmaticenigma: if i install a package of a specific version on a freshly provisioned VM, and then come back the next day, the package shouldn't have been upgraded. [17:23] but that's exactly what happened. [17:25] bin_bash: If it is truly the default of -security having been enabled, then the version didn't change, a patch was applied to mitigate a security vulnerability. What package was auto-updated on you? [17:25] nodejs [17:25] APT::Periodic::Update-Package-Lists "1"; [17:25] APT::Periodic::Unattended-Upgrade "1"; [17:26] this was the content of /etc/apt/apt.conf.d/20auto-upgrades [17:27] bin_bash: what about the 50* one? [17:28] do you want the whole thing in a paste? [17:28] it's actually mostly commented out [17:29] pragmaticenigma: https://0bin.net/paste/rLMnSMn5X4TrAjtu#8GTWJhVLB7ukZuOZw8G5nCySn5G7pdsV6ZcAGrf+I1Z [17:30] bin_bash: If what I've found so far is true, only the first two are not commented out... the rest are... specifically -security isn't commented out [17:30] bin_bash: the link says there is no paste there [17:30] then someone else must have clicked it [17:30] is there a bot that crawls links and opens them? [17:31] anyone in this room could have clicked that before I got the chance to [17:32] Unattended-Upgrade::Allowed-Origins { [17:32] "${distro_id}:${distro_codename}"; [17:32] "${distro_id}:${distro_codename}-security"; [17:32] "${distro_id}ESM:${distro_codename}" [17:32] thats the only uncommented part [17:32] including the bots... with the spam attack on freenode [17:32] oh and this [17:32] Unattended-Upgrade::DevRelease "false"; [17:32] i mean it was mere seconds between pasting and you trying to click, so it must have been something automatic. never had that problem in another channel, most bots just fetch metadata [17:33] okay, from what you have posted so far then... your instance is setup to only auto update security patches. [17:33] hm [17:33] even this though? [17:33] "${distro_id}:${distro_codename}"; [17:33] that doesn't have -security [17:34] that is your "root" store... the items in there do not change until a point release occurs, suchas 18.04.1 to 18.04.2 [17:34] plus if i specifically install a version (which I did), that should add some kind of flag, right? [17:34] bin_bash: no... those are defined in another file, explicately by the person doing the sysadmin [17:35] =/ [17:41] I wish I could find release notes to verify the enabled by default for you. Short of installing (which I can't do at my present location) I can't find anything online to verify other than a bug that 16.04 ignored a users selection of no unattended updates [17:42] yeah thats what i was looking for as well [17:42] but didn't find it [17:43] thank you for looking though [17:43] bin_bash: the closest that I can find is Ubuntu Desktop Gnome/unity have installed the package by default, and in Debian, the default configuration is to enable the main or root and the "-security" by default [17:44] hm interesting [17:44] bin_bash: I know that Ubuntu Desktop asks during installation, and by default has "Install security updates without confirmation" preselected [17:44] this is jsut a weird one-off package. gotta have node6 for this dumb thing -.- [17:45] bin_bash: I was partitially expecting you to mention something about GhostScript, as there were some significant security vulnerabilities patched in the last 24 hours [17:45] ahh [17:46] well i had a helluva time installing node6 at all. every other time ive done it on 16.04 i just added the source list and then apt install and it was fine [17:46] this time i had to specify the version [17:48] I wish I were more familiar with it... I know 18.04 some foundational pieces where significantly changes (netplan, etc) that have made some rather hard support questions [17:52] bin_bash: Found it! The default config file for the unattended-upgrades package has main and -security enabled by default in 50unattended-upgrades [17:52] it seems like something that should be documented [17:52] oooo [17:52] where did you find that [17:52] https://launchpad.net/ubuntu/+source/unattended-upgrades/1.1ubuntu1.18.04.5 [17:52] thank you! [17:52] rather: https://launchpad.net/ubuntu/+source/unattended-upgrades [17:53] perfect [17:53] i read that as "unintended upgrades" [17:53] hahaha which is more fitting -.- [17:54] yeah... I go back and forth on whether I want to enable it... sometimes I get really tired of the prompts on Desktop... at the same time, I run MythTV and don't want it to decide to apply updates in the middle of a recording and trigger the daemon to restart [17:55] I typically install all instances from the mini.iso release. In part because I can still install to 32 bit machines with it, and it's the same dialog prompts no matter what the final version I'm intending to install [17:55] and depending how far into the release we are, I don't have to spend an extra hour or two installing updates after installing the release [17:57] bin_bash: from this, I would assume the author of your vm included/installed the unattended upgrades package. Assuming they either clicked through accepting the default selections or installed the package after the fact, I don't think it was the vm's authors intent to misconfigure. As much as trying to follow the defaults offered by the original installation [17:58] bin_bash: it really should be better documented. it's even harder for server as most the documentation highlights the desktop installations more than server. I would assume Canonical would rather people installing server sign up for a support plan [18:06] it's just so frustrating. i dont think anything should automatically update on a server, that's just asking for problems. [18:08] bin_bash: Agree'd to a degree... the problem is the number of systems that get abandoned or aren't maintainted regularly that really could benefit such that they don't fall vicitm to someone's bot net [18:11] at minimum at least the security holes are plugged. If the unattended update broke your nodeJs application, that is really strange. unless it's the tenuous behavior of node6 (?) and ubuntu 18.04 you experienced?= [18:13] well the thing is that im installing it from nodesource [18:13] this is the official way of doing it according to nodejs [18:13] https://nodejs.org/en/download/package-manager/ [18:13] unfortunately something must have changed in 18.04 [18:13] because previously i just added the source, apt install, done [18:13] this time even after adding the source it STILL installed the one from the main repo [18:14] i had to do apt install nodejs=6.14.1-1nodesource1 [18:14] this time, it's weird [18:18] not too wiered... beneath the instructions for installing to Debian/Ubuntu... it mentions it only supports 16.04 and 14.04 [18:18] so I think that might be the root of your issue [18:39] ooh i didnt even see that [18:40] still though i dont think 18.04 was LTS when this was published [18:42] probably not, and there are a lot of under the hood changes with 18.04 that they might be trying to hammer out [18:43] yea [18:43] bleh [18:43] effed up to call it LTS when theyre still fixing things [18:43] absurd [18:47] bin_bash: depends on your perspecitve... If you take NodeJS out of the equation, 18.04 by itself is very stable (i'm currently running it) ... since nodejs team have to react to the change (they can't exactly see into what Canonical is planning for ubuntu) it takes a while before they get their dependencies and scripts polished and ready. There were some significant changes in 18.04 starting with the netowrk [18:47] management, and even the daemon managment [18:47] im not just talking about nodejs though [18:48] in general a server shouldn't do upgrades without intervention by default [18:48] think about php for example. there are many platforms that only work on php 7.0 or php 7.1 but not both. if php were security-upgraded to 7.1, that could cause a huge problem [18:58] bin_bash: release don't upgrade major versions [18:58] generally [18:59] bin_bash: "LTS" has nothing to do with "bug-free" [18:59] but thats exactly what happened with nodejs lol so i can't really expect it to not happen with other packjages [18:59] bin_bash: you weren't using an ubuntu version [18:59] bin_bash: so go complain to node, not here. [18:59] what? [19:00] the problem was with the core ubuntu repo. [19:00] bin_bash: you were using some external repository, right? [19:00] i was TRYING to [19:00] but it kept overriding it [19:00] bin_bash: what is 'it'? [19:01] apt/synaptic [19:01] bin_bash: i think maybe you just don't understand how packages work... [19:01] i think maybe you're not understanding what i'm saying [19:01] bin_bash: are you complaining that 18.04 has 8.10.0 while you wanted 6.14.1 ? [19:01] If I were to install weechat on debian from the weechat repos [19:01] full stop. [19:01] weechat repos? [19:01] debian? [19:01] apt wouldnt then override that for the core repos [19:01] it's called an example [19:02] if the weechat version in debian was greater than from weechat then yes it would. [19:02] bin_bash: i am fairly sure you just didn't check versions of packages via `apt-cache policy`, didn't bother to pin, and are complaining about that. [19:03] no, i'm complaining that the system made an internal, automatic decision to upgrade a package from another repository [19:04] bin_bash: what 'other' repository? [19:04] bin_bash: apt-cache policy nodejs, please [19:04] and for the record, i /DID/ check apt-cache policy [19:04] (if nodejs is the package you are worried about) [19:04] jesus fuck. i literally said I installed the package from nodesource. overnight, ubuntu upgraded that package from the extra repository [19:04] because the version in ubuntu is greater [19:05] you are choosing to run some third party repositroy [19:05] except it shouldn't do that bny default [19:05] and didn't bother to configure your apt sources appropriately to pin it [19:05] that's your opinion. [19:08] <_KaszpiR_> bin_bash apt-pin [19:10] _KaszpiR_: thats not a command. apt-cache search doesn't even return anything [19:10] <_KaszpiR_> https://jaqque.sbih.org/kplug/apt-pinning.html [19:10] regardless, if i install a package from one repo, it shouldn't be overriden by a package from another repo [19:10] ive literally never encountered that on any distro [19:10] <_KaszpiR_> not really [19:11] <_KaszpiR_> welcome to debian :/ [19:11] ive been using debian for years [19:11] and this is the first time running into this [19:11] ¯\_(ツ)_/¯ [19:11] <_KaszpiR_> heh, lucky you, then [19:11] <_KaszpiR_> got that many times [19:12] maybe because usually the other repo has newer versions rather than older [19:12] i'm used to running outdated packages on arch because nothing is automated, everything is deliberate [19:13] bin_bash: The repos are just storage containers... the repos themselves don't set a hierarchy. the packages themselves do with their naming convention since they sort alphanumerically. Debian happens to come earlier in the alphabet than Ubuntu... therefor an Ubuntu package is going to get precendence since it occurs later in the alphabet [19:14] assuming the package name is of 6nodejs-ubuntu-18.04 versus 6nodejs-debian-9 [19:14] pragmaticenigma: tbh mostly my upset is regarding a package being changed in any way without my deliberate action, and having this set as a default parameter and also not well-documented is porblematic [19:15] I assure you it's documented somewhere... but the joy of linux is... where? [19:17] i didnt say totally undocumented, i just said not well-documented. :P [19:18] I have the same premise [19:22] appears to have changed in ubuntu with https://git.launchpad.net/ubuntu/+source/unattended-upgrades/commit/?id=558990e4 [19:22] not 100% on that [19:22] it used to be -security only, though, and then the release pocket was added [19:23] nacc: that jives with what I found in the package from launchpad [19:24] in this case, it wouldn't matter, though, actually [19:24] a newer version is in ubuntu, period [19:25] and the -security case has been that way for quite some time [19:25] maybe unattended-upgrades became installed by default, dunno [19:27] nacc: I remember seeing it somewhere that server began installing it by default, but I just can't find a release note or documentation on it [21:33] hello, probably a dumb question: how to I get cloud-init to write the netplan file ? [21:37] i edited the clout-init file but not sure what command makes it generate the outputs [21:38] grandy: cloud-init runs once at boot [21:39] nacc: hmm, my ubuntu server install has a comment in the netplan that it was generated by cloud-init. I modified the cloud-init file in question and rebooted, but it did not update the netplan file. Just trying to add another network interface [21:42] grandy: which file did you edit? [21:45] nacc: /etc/cloud/cloud.cfg.d/50-curtin-networking.cfg [21:46] smoser: rharper --^ ? i don't know, tbh; does it need to be in the initrd instead? [21:48] nacc: i added the enx... interface, eno1 was already configured: https://pastebin.com/m6vyvDeJ [21:49] grandy: that will only be written once per instance. [21:49] so if the instance-id has not changed, changes to that will not get updated to the system. [21:49] * smoser has to go afk. [21:49] smoser: ahh, ok, so it's mainly for the intial config of the machine... where would i add a new network interface? [21:50] grandy: what do you mean 'add a new network interface'? You mean just the configuration for it, right? [21:50] smoser: yeah just to tell it to bring it up and use dhcp [21:51] grandy: i think you just want to put that in your netplan config file, no? [21:51] smoser: there is a configuration in /etc/netplan that is generated by cloud init, but it warns that it might be regenerated. [21:51] grandy: i mean, you can add another file in /etc/netplan, aiui [21:52] nacc: ahh ok, this is the contents of /etc/netplan/50-cloud-init... [21:52] https://www.irccloud.com/pastebin/RxCAvqeg/ [21:52] grandy: right, so leave that one alone (i think) [21:52] grandy: and add nother (see `man netplan`) [21:53] nacc: ahh ok, so then where would I change the config for eno1 ? a new file also? Just wondering in case i have to do that later. [21:54] nacc: it must be that cloud-init is meant for ephemeral instances, in which case it seems to make sense. [21:54] grandy: see the manpage, you can override settings with appropriately named files [21:54] grandy: cloud-init initializes an instance based upon cloud-provided data (among other things) [21:55] nacc: ok will do, yeah ok, this is starting to make sense now, just installed ubuntu server and it's been a few years since I have configured my own server so was not really up to speed on cloud-init ... thanks much for the help [21:56] grandy: sure, their docs are good toohttps://cloudinit.readthedocs.io/en/latest/topics/examples.html [21:57] nacc: yeah i was reading over them a bit when I thought it was meant for ongoing config updates and was thinking wow this looks like a great approach to config. [22:12] nacc: it worked. thanks again [22:13] grandy: cool, np!