| cpaelzer | good morning | 05:13 |
|---|---|---|
| lordievader | Good morning | 06:27 |
| === chmurifree is now known as chmuri | ||
| === gogbog is now known as Guest72809 | ||
| Epx998 | Is there a known issue on some hardware where debian-installer partitions a disk as sdc vs sda? | 15:19 |
| madLyfe | went to update my server via update && upgrade and it was running fine till the end. here is the end of the log: https://paste.fedoraproject.org/paste/Ib96aSM5fiupeAbW5-gaqQ/raw | 16:07 |
| blackflow | madLyfe: cp: error writing '/var/tmp/mkinitramfs_sr8t8I//lib/modules/4.13.0-41-generic/kernel/drivers/gpu/drm/i915/i915.ko': No space left on device | 16:08 |
| blackflow | the errors that follow result from that (no space left) | 16:08 |
| madLyfe | is there a way to see via command the drive breakdown? | 16:09 |
| blackflow | madLyfe: install ncdu | 16:11 |
| madLyfe | but space.. :P | 16:11 |
| blackflow | madLyfe: then du -sh /* and drill down from there | 16:12 |
| nacc | i would suggest du -h --max-depth=1 | 16:12 |
| nacc | on / | 16:13 |
| blackflow | well, you need -s(ummary) for each dir | 16:13 |
| nacc | no you do't | 16:13 |
| nacc | *don't | 16:13 |
| blackflow | I see, max-depth does the summary | 16:14 |
| madLyfe | is the last number the total? | 16:14 |
| nacc | blackflow: right, it's an alternative (du doing the dirlist vs. the shell) | 16:15 |
| rbasak | I like baobab for drilling down. It's a GUI but does support investigation remote machines over ssh. | 16:15 |
| rbasak | investigating | 16:15 |
| madLyfe | i went to root to see the total of everything but only printed ./ .nano and . | 16:16 |
| nacc | madLyfe: you don't need to cd, you can just pass the argument | 16:17 |
| nacc | madLyfe: and . means the directory | 16:17 |
| madLyfe | so go back to home/user? | 16:17 |
| nacc | madLyfe: i mean it doesn't matter where you run `du` from, you can tell it where to check disk usage | 16:18 |
| madLyfe | so drop the '=1' to search the whole drive? rather than just the dir i was in? | 16:18 |
| nacc | madLyfe: which version of the command did you use? | 16:19 |
| madLyfe | du -h --max-depth=1 | 16:20 |
| nacc | madLyfe: run from what location? | 16:20 |
| madLyfe | user | 16:20 |
| nacc | madLyfe: that's not a location | 16:20 |
| nacc | madLyfe: what filesystem path | 16:20 |
| madLyfe | home/user | 16:20 |
| nacc | madLyfe: /home/<user> ? | 16:20 |
| madLyfe | sorry, yes. | 16:21 |
| nacc | madLyfe: that will just tell you the usage of that directory | 16:21 |
| nacc | madLyfe: that's why I said "on /" immediately after. `du -h --max-depth=1 /` | 16:21 |
| madLyfe | oh. wasnt sure that was supposed to be a part of the command | 16:22 |
| blackflow | madLyfe: perhaps check the du(1) manpage. it's wise to check the manpage of commands you're given by random people on irc :) | 16:25 |
| madLyfe | https://paste.fedoraproject.org/paste/8K5JqNBszC2J5xtjevcwQA/raw | 16:26 |
| nacc | madLyfe: and how large is your disk (`df -h` can help) | 16:27 |
| madLyfe | 16gb | 16:28 |
| madLyfe | its a flashdrive | 16:28 |
| madLyfe | https://paste.fedoraproject.org/paste/Xz~tFEKAOKExJAym-vCjHg/raw | 16:29 |
| nacc | madLyfe: your / is full | 16:31 |
| madLyfe | do updates not overwrite? do they accumulate? | 16:32 |
| madLyfe | can that be pruned? | 16:32 |
| nacc | madLyfe: what do you mean? | 16:32 |
| madLyfe | like are old updated files kept around or are they overwritten? | 16:33 |
| nacc | madLyfe: 'updated' in what sense? | 16:33 |
| madLyfe | every time you run update && upgrade | 16:33 |
| nacc | madLyfe: update just updates the file lists locally | 16:34 |
| nacc | madLyfe: are you asking if it keeps around the old .debs? | 16:34 |
| madLyfe | ya | 16:34 |
| nacc | madLyfe: it keeps some amount in the cache | 16:35 |
| nacc | madLyfe: if you think that's a problem, use `sudo apt-get clean` | 16:35 |
| madLyfe | hmm i dont think that cleared anything up for me heh | 16:36 |
| nacc | madLyfe: usually it's not much space | 16:37 |
| nacc | madLyfe: right now your /opt and /var are relatively large | 16:37 |
| nacc | madLyfe: which might imply large logs | 16:37 |
| nacc | madLyfe: you can run the same du command with /var | 16:38 |
| madLyfe | ya its amdappsdk-3.0, amdgpu, amdgpu-pro, rocm. | 16:38 |
| madLyfe | in /opt | 16:38 |
| madLyfe | just not sure what i can get rid of | 16:39 |
| blackflow | anything your __server__ doesn't need :) | 16:39 |
| madLyfe | lel i dont know what it doesnt need. | 16:40 |
| madLyfe | i am its plaything. | 16:40 |
| nacc | madLyfe: that doesn't make sense | 16:40 |
| madLyfe | here is /var breakdown: https://paste.fedoraproject.org/paste/tf6vLUjOZIpEC40USrgtww/raw | 16:40 |
| nacc | madLyfe: is it actually your server? | 16:40 |
| nacc | your /boot also looked bad, fwiw | 16:41 |
| nacc | madLyfe: have you run `sudo apt-get autoremove` recently ? | 16:41 |
| madLyfe | ya. its using amd drivers to rung GPUs but i dont remember how i set them up. someone helped me and it was kind of set it and forget it. months ago. update && upgrade never posed a space issue till now i guess. | 16:42 |
| madLyfe | nacc: nah | 16:42 |
| nacc | madLyfe: becuse you were ignoring the message from apt the whole time | 16:42 |
| nacc | it tells you to run autoremove every time, iirc | 16:42 |
| madLyfe | https://paste.fedoraproject.org/paste/-RIiGhNJboQs7-Edk7lH4Q/raw | 16:43 |
| madLyfe | i dont remember that msg | 16:43 |
| nacc | madLyfe: well, that would free up 3G of space if you run it | 16:44 |
| nacc | 4G of space! | 16:44 |
| madLyfe | yay! ty | 16:45 |
| madLyfe | what is that removing? are they old debs from previous udpates? | 16:46 |
| teward | old debs, old kernels, packages which were installed as dependencies but are no longer needed since the dependent package or packages are not on the system or the deps changed, etc. | 16:47 |
| madLyfe | ah k. tyvm | 16:47 |
| nacc | to be clear, what it's removing is *only* the third thing teward listed | 16:48 |
| nacc | that happens to include the first two | 16:49 |
| madLyfe | i see. | 16:49 |
| madLyfe | when should this be ran? i dont recall seeing that msg anywhere. | 16:50 |
| nacc | madLyfe: probably worth a perusal of `man apt-get` if you don't know what it does | 16:50 |
| nacc | madLyfe: on my system(s) every time I run apt or apt-get it emits a message about X unused packages that can be removed | 16:50 |
| madLyfe | man is manual? | 16:50 |
| madLyfe | as in RTFM? | 16:51 |
| nacc | madLyfe: yeah :) | 16:52 |
| madLyfe | ok. i want to thank you guys for the hand hodling today! | 16:53 |
| madLyfe | blackflow: i agree about running commands from ppl on the internet but nacc has helped me plenty in the past so i trust the nacc. | 16:54 |
| blackflow | madLyfe: it's not just about trust. it's about learning by yourself ;) | 16:54 |
| blackflow | blindly running commands, you won't learn a thing. | 16:54 |
| madLyfe | unless hes been taken over by a malicious actor. very possible these days. :P | 16:55 |
| madLyfe | true. | 16:55 |
| nacc | madLyfe: heh | 16:55 |
| nacc | madLyfe: unattended-upgrades can be configured to run the autoremove periodically, iirc | 16:56 |
| teward | nacc: yes, but it needs you to uncomment some things | 16:56 |
| teward | in the default config *otherwise* it doesn't. | 16:56 |
| teward | it also can barf if you try and let it upgrade kernels automatically, but that's a different issue. | 16:56 |
| teward | (related to not having run autoremove in the past before enabling the option) | 16:57 |
| madLyfe | also, the updates that require restarts wont be applied because these servers are rarely restarted. | 16:57 |
| teward | you can also have unattended-upgrades do that as well but if uptime is your concern then you shouldn't. And you should probably reboot every quarter or such because kernel security patches | 16:58 |
| madLyfe | ya thats kind of what i was doing today because power went out. good time to run the updates. | 17:00 |
| nacc | teward: good points | 17:04 |
| madLyfe | well crap it looks like it was removing amdgpu stuffs, hopefully it doesnt take out anything important. | 17:05 |
| madLyfe | getting amd drivers working wasnt fun. | 17:05 |
| madLyfe | also, if youve run updates and log into a fresh ssh session, it says **a restart is required*, why is the command 'reboot' and not 'restart'? or why does the statement not read '**a reboot is required**'? | 17:13 |
| madLyfe | sorry, its '*** System restart required ***' | 17:14 |
| jelly | technically you do not require a reboot; shutdown and power on will achieve the same restart goal | 18:03 |
| madLyfe | what should i do if one of my servers has just been hanging here for a long time during and update & upgrade? https://paste.fedoraproject.org/paste/J4gA9Z8HRYn5sBG9G0KGQw/raw | 18:18 |
| teward | define "a long time" :P | 18:19 |
| madLyfe | close to an hour | 18:20 |
| madLyfe | previously it got stuck here: https://paste.fedoraproject.org/paste/QcOKX178hCZwEbDNr9MG2Q/raw at fetching. | 18:21 |
| madLyfe | had to pull the power on it and power it up again. | 18:22 |
| madLyfe | ok it just now lost the ssh connection. | 18:22 |
| madLyfe | im guessing it froze up or something | 18:22 |
| madLyfe | https://usercontent.irccloud-cdn.com/file/f8el4LyY/image.png | 18:23 |
| tomreyn | madLyfe: if you install updates remotely over a non LAN connection, be sure to always run apt in a GNU screen / tmux on the server. | 18:41 |
| madLyfe | well i am remote, but on the lan. | 18:42 |
| madLyfe | remote as in from another puter on the lan. | 18:43 |
| tomreyn | the L in LAN stands for local | 18:43 |
| tomreyn | maybe you need a KVM | 18:44 |
| tomreyn | or serial console | 18:44 |
| madLyfe | well these are SM servers w/o the IPMI. i only have one server with IPMI | 18:44 |
| madLyfe | regardless, i should just go pull the power again and try once more? | 18:45 |
| tomreyn | i was thinking they all had *some* kind of ipmi | 18:45 |
| madLyfe | not these old ones from what ive gathered. | 18:46 |
| madLyfe | still. if it killed the server, wouldnt it kill the server in a tmux session as well? | 18:46 |
| tomreyn | if it's frozen, you probably want to power cycle it, then probably take measures to ensure the hardware is in good shape, when continue working on it. | 18:46 |
| teward | and if it keeps freezing then I'd start to assume something hardware-based may be at play, since it sounds like this is not hte first freeze you've had on this system. | 18:47 |
| tomreyn | sure, GNU screen + tmux dont survive power cycling or reboots in general. | 18:47 |
| madLyfe | well this is the second time on this server today after the power outage. other than that ive never had this issue before on any of the servers. | 18:47 |
| teward | power outages cause all sorts of problems (this is why UPSes exist heh) | 18:48 |
| madLyfe | tomreyn: no i mean it freezing during update & upgrade. | 18:48 |
| madLyfe | if it freezes the server then being in a tmux session wouldnt really matter? | 18:49 |
| tomreyn | madLyfe: if the system freezes, the session is frozen, too, and it wont magically recover after power cycling. | 18:49 |
| madLyfe | what is your definition of a 'power cycle'? im just asking because on the IPMI there is actually a power cycle button.. | 18:50 |
| tomreyn | you need to make the hardwar enot freeze | 18:50 |
| tomreyn | didnt oyu say you had no ipmi there? | 18:50 |
| tomreyn | my definition of 'power cycling' is probably the same as everyone elses: cutting power for a short time, then restoring it. | 18:51 |
| Tea | Does the name I choose for a DKIM selector matter? | 18:52 |
| madLyfe | I have one ipmi. just not on this server. | 18:53 |
| madLyfe | holding the power button doesn't even shut it down lol | 18:54 |
| teward | sounds like a hardware problem then | 18:54 |
| madLyfe | probably the flash drive | 19:00 |
| nacc | rbasak: anything you need from me re:git-ubuntu? | 19:00 |
| madLyfe | seems to be updating fine now: https://usercontent.irccloud-cdn.com/file/n5wlnmu8/image.png | 19:06 |
| madLyfe | welp looks like it froze again | 19:44 |
| madLyfe | now it gets stuck during boot. im guessing its because it froze during this upgrade process: https://usercontent.irccloud-cdn.com/file/PAlNMjhv/image.png | 19:59 |
| madLyfe | ill try and install the OS again later and see how that goes. | 19:59 |
| sarnold | o_O | 20:01 |
| sarnold | that's a funny place for it to hang | 20:01 |
| sarnold | can you check dmesg output? journalctl output? | 20:02 |
| sarnold | filesystem freespace? free inodes? | 20:02 |
| madLyfe | nah its a total freeze of the system | 20:02 |
| madLyfe | power down button didnt even work when it froze at that point in the apt update progress | 20:03 |
| madLyfe | ill just have to restart from scratch on that server. shouldnt take very long once i get rolling. | 20:04 |
| sarnold | madLyfe: oh. ew. maybe try the .. | 20:09 |
| sarnold | madLyfe: .. fwts package? | 20:10 |
| madLyfe | I'll come back if I can't get it situated after a fresh go. I'm not losing any important data so it's ok. | 20:13 |
| jamespage | coreycb: around still? | 21:14 |
| coreycb | jamespage: yep | 21:14 |
| jamespage | coreycb: just wanted to check with you before I upload a whole load of neutron bits with py3 enabled | 21:15 |
| coreycb | jamespage: that's fine with me | 21:15 |
| jamespage | coreycb: I've been using a <module>-common package for configuration files etc... | 21:15 |
| jamespage | coreycb: which the python{3}-<module> packages depend on - does that sound ok? | 21:15 |
| jamespage | prior to that the python-<module> package included the etc files | 21:15 |
| coreycb | jamespage: that sounds different from python-<module> depending on <module>-common? | 21:16 |
| coreycb | sorry. other way around. | 21:17 |
| jamespage | yeah it is | 21:17 |
| coreycb | jamespage: currently we have <module>-common depending on python-<module> | 21:17 |
| jamespage | coreycb: I could not think of a different way todo it - we have to preserve etc files for upgraders who will only have install the python-module | 21:18 |
| coreycb | jamespage: that's a good point | 21:18 |
| jamespage | coreycb: these are generally python modules with config files, rather than more complex agent based packages like neutron for example | 21:19 |
| coreycb | jamespage: that sounds like it makes sense. i missed that. i'm going to have to make a pass on the merged debian packages to do that. | 21:19 |
| jamespage | coreycb: did you do it the other way around? | 21:20 |
| coreycb | jamespage: yeah | 21:20 |
| jamespage | coreycb: well I think its appropriate still for packages with agent binary packages - like neutron, nova cinder etc... | 21:20 |
| jamespage | coreycb: http://launchpadlibrarian.net/375048921/python-oslo.versionedobjects_1.33.1-0ubuntu1_1.33.1-0ubuntu2.diff.gz is the diff for the generalized autopkgtest for oslo packages I'm doing | 21:21 |
| coreycb | jamespage: i think autopkgtest-pkg-python should cause imports to be tested but maybe not dual stack? | 21:23 |
| jamespage | coreycb: tbh it does not deal with the name of the module being oslo.* and the actual module import being oslo_* | 21:23 |
| coreycb | jamespage: ok, i see | 21:24 |
| jamespage | which is why most of them fail all of the time - I tried to get a fix into autopkgtest but it was rejected in favour of work not done todo better autodiscovery | 21:24 |
| jamespage | which I don't have cycles for | 21:24 |
| jamespage | the import test is basically autopkgtest-pkg-python and the -ostestr one is extra | 21:24 |
| jamespage | basically we'll run unit tests on all reverse-depends changes which is nicer | 21:25 |
| coreycb | jamespage: ok makes sense, cool | 21:25 |
| coreycb | jamespage: thanks for the update. i'll check out your neutron uploads. | 21:27 |
| === rev_strangehope_ is now known as revstrangehope | ||
| nacc | powersj: did you see the failure just now? (git-ubuntu-ci): error: flag provided but not defined: -t | 22:53 |
| nacc | failed: sudo lxd waitready -t 60 | 22:53 |
| powersj | nacc: I merged in rbasak's changes let me double check 'em in a bit | 22:53 |
| powersj | did that this morning | 22:53 |
| nacc | thanks | 22:53 |
| nacc | it seems weird, as lxd says it is an option (on my 18.04 using the deb) | 22:54 |
| rbasak | I had thought I'd tested the deb version locally | 22:57 |
| rbasak | I had to add waitready as on the snap version it takes a while for the socket to become ready (the snap doesn't appear to use systemd to manage the daemon, so no socket activation) | 22:57 |
| rbasak | I will check again tomorrow. If necessary I can make the waitready only happen on the snap option. | 22:58 |
| rbasak | Then the next change to land is the change of the Jenkins job itself to use --snapcraft-snap | 22:58 |
| nacc | sorry, i meant just that i've got the lxd deb locally and it says -t is an option and takes a parameter | 22:58 |
| nacc | whatver lxd the ci job is suing is complaing about the passing of -t | 22:58 |
| nacc | does it expect -t=60 ? | 22:58 |
| nacc | or -t60 | 22:59 |
| rbasak | I can reproduce now. | 23:01 |
| rbasak | Ah | 23:01 |
| rbasak | Scrolling back I had tested it but not noticed that it failed :( | 23:01 |
| rbasak | Looks like it doesn't have the -t shortcut and it's --timeout=60 | 23:02 |
| nacc | fun | 23:03 |
| rbasak | https://github.com/CanonicalLtd/server-test-scripts/pull/3 | 23:07 |
| rbasak | Tested and succeeds in both snap and deb modes now | 23:07 |
| rbasak | Sorry for my test failure on the last PR | 23:07 |
| * powersj pulls | 23:08 | |
| gordonjcp | hi there | 23:08 |
| gordonjcp | is there a way to find out why a package wants to install certain dependencies? | 23:08 |
| nacc | rbasak: cool, thanks! | 23:08 |
| nacc | gordonjcp: well, each package lists its dependencies | 23:08 |
| nacc | gordonjcp: do you mean what they use those dependencies for? | 23:08 |
| gordonjcp | nacc: essentially yes | 23:09 |
| nacc | gordonjcp: no, i don't think that really makes sense | 23:09 |
| * rbasak attempts a retest | 23:09 | |
| gordonjcp | nacc: I want to install collectd on a server, which is for collecting up various stats into (among other things) an .rrd file | 23:09 |
| gordonjcp | nacc: it wants to install a full X desktop | 23:09 |
| gordonjcp | like, about 1GB of deps, including X.org, Wayland, all manner of associated libs | 23:10 |
| gordonjcp | I know the Ubuntu packagers are generally a bit on crack with their dependencies, but that seems excessive | 23:10 |
| nacc | gordonjcp: ignoring the random snark, what version of ubuntu? | 23:10 |
| rbasak | gordonjcp: try using --no-install-recommends | 23:10 |
| gordonjcp | 16.04 | 23:11 |
| rbasak | That cuts it down quite a bit. | 23:11 |
| gordonjcp | a bit, but it's still trying to install libmir and friends, wayloand and friends and X.org and friends | 23:11 |
| rbasak | I don't see that | 23:12 |
| rbasak | Can you be specific about a package that you think shouldn't be included? | 23:12 |
| nacc | gordonjcp: i don't think that's correct and i just tested in a lxd 16.04 | 23:12 |
| gordonjcp | libwayland-cursor0 libwayland-egl1-mesa libwayland-server0 libx11-xcb1 | 23:12 |
| gordonjcp | ^ one line from th edeps | 23:12 |
| nacc | gordonjcp: with --no-install-recommends? | 23:13 |
| rbasak | Do you have any third party repositories enabled? | 23:13 |
| nacc | https://paste.ubuntu.com/p/89dkDNG5Xs/ | 23:13 |
| nacc | is what you should get | 23:13 |
| gordonjcp | I've already got the pango and cairo stuff in, because rrdtool uses that | 23:14 |
| gordonjcp | although why collectd pulls it in when it doesn't draw graphs I don't know | 23:14 |
| gordonjcp | I don't have any third-party repositories enabled | 23:14 |
| rbasak | collectd links with librrd4 | 23:14 |
| rbasak | librrd4 can render using GDK | 23:14 |
| nacc | gordonjcp: please use a pastebin and show the exact output | 23:14 |
| gordonjcp | although now you mention it, this is a server on a VPS provider, maybe they've got some "funnies" in | 23:15 |
| rbasak | (presumably to the screen, on request) | 23:15 |
| rbasak | Try to reproduce on an official Ubuntu image please. | 23:15 |
| rbasak | If you can't, then you'll need to figure out how it's different from the OS your provider gives you. | 23:15 |
| gordonjcp | https://rangerovers.pub/static/out.txt | 23:16 |
| gordonjcp | rbasak: okay, I'll grab a copy of 16.04 server and try that | 23:17 |
| gordonjcp | rbasak: that won't be until probably wednesday though | 23:17 |
| gordonjcp | so you might miss the fun :-D | 23:18 |
| arooni | this line works great ... until the server reboots; do i need to do something to make it permanent ? | 23:32 |
| arooni | sudo iptables -t nat -A PREROUTING -p tcp --dport 500 -j REDIRECT --to-ports 5000 | 23:34 |
| rbasak | powersj: am I right in thinking that https://github.com/CanonicalLtd/server-test-scripts/blob/master/git-ubuntu/integration_test is dead now as https://github.com/CanonicalLtd/server-jenkins-jobs/blob/master/git-ubuntu/jobs-ci.yaml no longer references it? | 23:37 |
| powersj | rbasak: I believe so | 23:37 |
| rbasak | OK. I'll drop that file in my next PR then | 23:38 |
| powersj | ok | 23:38 |
| sdeziel1 | arooni: you can install the iptables-persistent package | 23:58 |
| sdeziel1 | arooni: it will take care of loading your iptables and ip6tables rulesets on boot | 23:58 |
Generated by irclog2html.py 2.7 by Marius Gedminas - find it at mg.pov.lt!
