| Deihmos | is it normal for ubuntu server to have a new update every single day? | 00:48 |
|---|---|---|
| sarnold | all the updates are mailed to eg bionic-changes or xenial-changes: https://lists.ubuntu.com/archives/bionic-changes/ | 00:49 |
| sarnold | you could take a look at them to figure out how often it happens | 00:49 |
| sarnold | security updates are almost always monday through thursday, but we may release emergency updates outside of that window; and your server might receive them some time after we release them, based on your settings... | 00:50 |
| sarnold | SRU updates may come out any day | 00:50 |
| RoyK | Deihmos: frequent updates are good - just install unattended-upgrades and let it do its business | 01:02 |
| === cpaelzer__ is now known as cpaelzer | ||
| lordievader | Good morning | 06:27 |
| ygk_12345 | hi all good morning | 07:08 |
| ygk_12345 | can someone look into this please | 07:08 |
| ygk_12345 | https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1823146 | 07:08 |
| ubottu | Launchpad bug 1823146 in linux (Ubuntu) "network namespaces error" [Undecided,Incomplete] | 07:08 |
| ygk_12345 | is anyone familiar with network namespaces ? | 07:18 |
| ygk_12345 | is anyone here familiar with network namespaces | 07:27 |
| bhuddah | ygk_12345: can you replicate the bug in a lab setting? | 07:28 |
| ygk_12345 | bhuddah: i have 4 ubuntu 18 servers which is having this problem but each time a reboot seems to fix it | 07:30 |
| bhuddah | ygk_12345: so at the moment all you know is that it happens "randomly"? | 07:32 |
| ygk_12345 | bhuddah: yes and the command seems to never complete and hangs | 07:32 |
| bhuddah | ygk_12345: and there are several different commands that hang by that time? | 07:33 |
| ygk_12345 | bhuddah: no only this command | 07:33 |
| bhuddah | ygk_12345: you wrote apport hangs too... | 07:33 |
| ygk_12345 | bhuddah: apport doesn't hang but goes on forever . it is not related to the original issue | 07:34 |
| bhuddah | ygk_12345: is that the first namespace you're trying to add? | 07:47 |
| ygk_12345 | bhuddah: no. I already have some created earlier | 07:47 |
| bhuddah | ygk_12345: no idea how to debug that sort of intermittent error there. sorry :/ | 07:49 |
| ygk_12345 | can anyone help me with my issue please | 07:50 |
| bhuddah | ygk_12345: maybe try in #netfilter maybe they have an idea. | 07:55 |
| tomreyn | what i recommended yesterday was to clone one of those untouchable production systems which run with outdated kernel versions and update the kernel on the cloned system, to see whether it still happens then. | 07:56 |
| tomreyn | the kernel version ygk_12345 is using on those production systems is not just not the latest but also specifically known to break userspace. | 07:57 |
| lotuspsychje | same here, i reccomended you also to get your system up to date first ygk_12345 | 07:58 |
| ygk_12345 | lotuspsychje: but elsewhere we have another system with the same kernel and it is working fine | 07:58 |
| bhuddah | ygk_12345: what is your primary goal? fixing the machine or understanding the bug? | 07:59 |
| ygk_12345 | bhuddah: both so that it doesn't occur again in the future | 07:59 |
| bhuddah | ygk_12345: OR | 08:00 |
| lotuspsychje | i really dont get why admins keep back important updates/kernels in production.. | 08:01 |
| bhuddah | in my experience it comes from bad development practices mostly. arcane software with outdated software requirements. | 08:02 |
| blackflow | lotuspsychje: one day when you'll have a fleet of servers and any minute of downtime means clients and management yelling at you and money lost, you'll know :) though, honestly, you keep them back only temporarily until the breakage is solved or it's fully tested. | 08:15 |
| blackflow | also, I read kernel changelogs and do NOT upgrade the kernel every time there's one. there's no need to reboot for fixes and patches that don't concern the system it's running on and there's a lot of those in Ubuntu. | 08:15 |
| ygk_12345 | lotuspsychje: bhuddah tomreyn actually when you have a running critical system, you cant risk to upgrade everytime there is one as it may break the software application. And thats a pain. | 08:18 |
| bhuddah | ygk_12345: you have a development system, a test system, an integration system and then a live system. of course you can risk doing upgrades. | 08:21 |
| tomreyn | ygk_12345: i'm aware. i'm also aware that you don't run critical systems on a known bad foundation. | 08:21 |
| tomreyn | and as bhuddah says, surely you can and hsould be able to upgrade your dev / test and maybe integration systems | 08:22 |
| blackflow | what's "integration systems".... some new marketing/devops speak? like "serverless"? | 08:23 |
| tomreyn | staging | 08:23 |
| blackflow | how is it different from testing? | 08:23 |
| bhuddah | it's the development system for the integrators... sort of. | 08:24 |
| blackflow | what "integrators"? | 08:24 |
| bhuddah | the people who operate the deployment and so on? | 08:25 |
| blackflow | and again how's that different from testing. | 08:25 |
| blackflow | you have prod, and you clone prod for testing purposes (you test changes on systems that resemble production as much as possible, but aren't prod themselves). so what else do yo do? | 08:25 |
| blackflow | I guess I get triggered by devops circlejerk. "serverless integration devops meta-testing bull"....... GAH | 08:27 |
| ygk_12345 | any kernel gurus here ? | 08:27 |
| ygk_12345 | atleast they can help me I believe | 08:28 |
| bhuddah | blackflow: i think that's even way older than devops. we've been doing that for ages. separation of different systems for different purposes... | 08:28 |
| bhuddah | ygk_12345: reproduce the error on a test system, please. don't make all the people nervous. | 08:29 |
| blackflow | bhuddah: I think you're confusing unit testing terminology (levels of unit -> integration -> functional) with deployment strategies. | 08:30 |
| bhuddah | blackflow: i bet it's a translation problem. i'm not english native and sometimes we call stuff by weird names. | 08:32 |
| ygk_12345 | bhuddah: :) | 08:32 |
| ygk_12345 | bhuddah: the thing here is that it worked earlier on these systems but suddenly we started seeing these issues. so testing on a similar system wont expose the real root cause | 08:33 |
| ygk_12345 | bhuddah: it will go well with the test system also. it is related to these particular defective systems that is causing this issue | 08:34 |
| bhuddah | ygk_12345: then clone one of the system. i don't know. that's all part of bug reproduction... | 08:34 |
| ygk_12345 | bhuddah: so we cant really replicate the issue | 08:34 |
| bhuddah | hunting unicorns... | 08:35 |
| ygk_12345 | bhuddah: tomreyn how to clone a running system ? | 08:37 |
| bhuddah | tell me you have backups... | 08:37 |
| bhuddah | please. | 08:37 |
| ygk_12345 | bhuddah: we dont have any backups as of now | 08:39 |
| bhuddah | aah. the "important" "production" machine without backups. | 08:40 |
| bhuddah | i think you can just rsync the whole machine btw. or rsync an lvm snapshot. | 08:42 |
| blackflow | lol! | 08:42 |
| blackflow | ygk_12345: "testing on a similar system wont expose" -- then they're not similar enough, that's the whole point here, and hinted at by bhuddah with the suggestion to clone production. | 08:43 |
| blackflow | but see, you should have a clone of prod to begin with, and if you don't, then make one asap and always run changes through that _first_. and indeed, your prod is only as valuable as the effort you put into backups. | 08:44 |
| lotuspsychje | blackflow: i fully understand yeling customers isnt pleasing, but keep running older kernels/security updates can result in even more yelling customers right? | 08:58 |
| lotuspsychje | and as tomreyn suggested, i would stay long on his current kernel | 08:59 |
| lotuspsychje | wouldnt | 08:59 |
| blackflow | lotuspsychje: depends on what it is. sometimes a bug fixed can nuke your prod. happened to us twice. once when PHP fixed a long standing bug in class constructors (param precedence), we had to change the apps before applying the PHP update (which was also a security fix) because the apps were written for the working, though bugged PHP API. | 09:00 |
| blackflow | the other time ubuntu kernel had an apparmor security fix that broke boot, two years or so ago. held that off until next patch level that fixed it again. | 09:00 |
| blackflow | so yeah there's no clear answer but one: test on as similar systems as possible, preferably down to every screw and nut in the chassis :) | 09:02 |
| lotuspsychje | yeah test systems are wise | 09:02 |
| blackflow | right. just saying why there are quite valid reasons to hold off important updates. but yeah those are temporary situations one should aim to resolve asap. | 09:03 |
| lotuspsychje | asap i understand blackflow | 09:04 |
| lotuspsychje | blackflow: but ygk_12345 said yesterday he was still on that kernel because he was running..something | 09:05 |
| lotuspsychje | ygk_12345lotuspsychje: we have installed openstack on it. so cant take that risk | 09:08 |
| blackflow | lotuspsychje: yeah. I was just replying to "why admins keep back important updates/kernels in production". there be reasons. but also yes, one should look into fixing that asap. | 09:30 |
| lotuspsychje | allrighty :p | 09:30 |
| === cpaelzer__ is now known as cpaelzer | ||
| coreycb | sahid: cinder rc2 uploaded, thanks | 12:16 |
| sahid | coreycb: ack | 12:19 |
| J_Darnley | Why does ubuntu 18.04 not have an ip address despite me leaving dhcp when I installed it. | 13:21 |
| J_Darnley | The interface today has no ip despite it getting one yesterday | 13:21 |
| J_Darnley | Okay, what is this cloud-init thing? | 13:29 |
| tomreyn | it's a utility to make cloud deployments easier, i think. you can uninstall it if you dont think you need it. | 13:36 |
| J_Darnley | Is there a fallback for network configuration if I choose to do that? Or do I need to install one? | 13:39 |
| J_Darnley | Hm, I'd better search the installed packages | 13:39 |
| tomreyn | ubuntu server 18.04 uses systemd-networkd for network configuration by default. there is also netplan, which can generate configurations for both systemd-networkd and network-manager (which is used on desktops by default). | 13:40 |
| J_Darnley | systemd is fine, I can work it, just about | 13:42 |
| Holiday | tomreyn: here | 14:00 |
| tomreyn | Holiday: hi, please repost your questions here | 14:02 |
| Holiday | Has anyone else notice any issues with networking when apt updates systemd, systemd-sysv, libsystemd0? We have 18.04 deployed but have to use ifupdown because of our vCenter version, and things seem to work fine until apt updates those libraries. When it does, networking is lost (nic doesn't reflect the IPs) | 14:02 |
| Holiday | restart networking and things appear. On one system as a test, I removed netplan.io and it appears it's the only one this morning that DIDN'T lose networking.. so I'm wondering if there's an issue with systemd updates and netplan | 14:02 |
| Holiday | this is a VM, ubuntu server 18.04 LTS, and should be using whatever the default is for the network management iirc | 14:02 |
| Holiday | (minus the fact I had to put ifupdown on it) | 14:02 |
| tomreyn | do you see the connectivity loss reflecte din logs? | 14:04 |
| Holiday | I do see the loss reflected and maybe it is a "who controls" issue although it only seems to show the one interface being mentioned (lo) | 14:11 |
| tomreyn | can you share the relevant logs of such an event on a pastebin? | 14:12 |
| Holiday | systemd stops the networking, starts it again, stops the name resolution, and then barfs with systemd-networkd lo link is not managed by us and that's the last of it | 14:12 |
| Holiday | sure | 14:12 |
| tomreyn | lo would be the loopback device. | 14:12 |
| Holiday | yeah, never mentions the actual ether | 14:12 |
| tomreyn | the loopback device becoiming unavailable would impact name resolution via the systemd-resolved dns cache listening on 127.0.0.53:53 | 14:14 |
| tomreyn | my (limited?) understanding is that netplan.io is never run automatically, so whether it is installed or not should not impact connectivity. | 14:16 |
| Holiday | tomreyn: https://pastebin.com/ekQwFzGQ | 14:16 |
| Holiday | always seems to be when systemd is updated | 14:17 |
| Holiday | now maybe it's because I had to do the ifupdown and renamed the interface to make the whole "vCenter version configure it's IP" deal to work, which does.. and otherwise everything seems happy | 14:17 |
| Holiday | its literally just after the apt update runs | 14:17 |
| cyphermox | Holiday: tomreyn: netplan does run automatically at very early boot to take the yaml and convert into the appropriate config for NetworkMaanger or systemd-networkd | 14:21 |
| cyphermox | that said, if there is no YAML, it does nothing | 14:21 |
| Holiday | @cyphermox hrm.. wonder if it's because I left the dhcp4: yes in the 01-netplan.yml config file.. basically it's the default with just ifupdown installed. Although, like I said, works on boot, ifdown && ifup etc all resolve the issue.. it's just when apt updates the systemd | 14:24 |
| cyphermox | if you have dhcp4: yes for an interface in netplan yaml, then yes it could possibly do something if systemd is restarted | 14:25 |
| cyphermox | you shouldn't have the same interface configured in both ifupdown and netplan, that will cause conflicts | 14:25 |
| Holiday | I'll have to remove that then. I just left it as the Ubuntu netplan.io docs just say if you need/want to use ifupdown, just installing the package is enough | 14:26 |
| tomreyn | thanks c-mox, i wasn't aware of that. | 14:36 |
| cyphermox | to be clear, at boot, what happens is the same as running 'netplan generate'; it does nothing but write files, no restarting of services | 14:38 |
| Ool | cyphermox: just to know, when you reboot it do a netplan apply ? | 15:06 |
| cyphermox | no | 15:07 |
| cyphermox | at boot time, it's always just running the generator; only writing files to /run | 15:08 |
| cyphermox | unless I misunderstood your question | 15:08 |
| cyphermox | 'netplan apply' and 'netplan try' are just for users to run when you change the config and want to see the result immediately | 15:09 |
| Ool | but if you change the config and reboot without do netplan apply, the new config is apply at the next boot ? | 15:10 |
| Ool | cyphermox: dsl, I do my best in english :) | 15:11 |
| cyphermox | yes, if you reboot the config is applied | 15:24 |
| maxel | Hi all, I'm not sure how to begin troubleshooting this situation, but I've been attempting to upgrade an ubuntu server vm from 16.04 to 18.04 lts. every time I've done it, the biggest things I've noticed that break are the ssh server, and my mounting of an xfs file system | 15:48 |
| maxel | am I doing something wrong with my upgrade, or is this sort of thing expected? | 15:48 |
| rbasak | maxel: local customisations may break if the surrounding infrastructure has changed between releases. There's no avoiding that unfortunately. | 15:51 |
| rbasak | maxel: but for default or simple cases to break is a bug. | 15:51 |
| maxel | I had the weirdest problem with the ssh servere, where ufw seemed to introduce a rule to disable ssh, once I allowed it again I had a problem where I would log in, and it would immediately kill the session | 15:53 |
| rbasak | Can you reproduce it on a fresh 16.04 installation thatn is then upgraded to 18.04? | 15:54 |
| maxel | I have not gone to those lengths yet, no | 15:55 |
| blackflow | maxel: are your ssh keys DSA or RSA? iirc between xenial and bionic OpenSSH has stopped supporting DSA | 15:57 |
| maxel | I wasn't even using ssh keys, just user:pass login | 15:58 |
| maxel | although, when upgrading I opted to use the maintainers configuration, I'm not sure if I should be opting to keep my existing config | 15:59 |
| blackflow | maxel: yes if you changed it from default | 15:59 |
| blackflow | (which would be the case if you used keys) | 15:59 |
| blackflow | also, if that's some hosted server, companies usually modify the ssh config to allow root, they don't pre-suppose your non-root admin account name. mostly. if that's teh case, you'll have to re-enable root login | 16:00 |
| blackflow | --- if you're using root, thatis. | 16:00 |
| maxel | I was not using root to login, but a user that is in wheel | 16:01 |
| maxel | and this is a self hosted server, I have an esxi server running vms, this is my ubuntu server vm | 16:01 |
| blackflow | maxel: well, did you change the /etc/ssh/sshd_config in any way? | 16:01 |
| maxel | so nothing is enforced as far as policy | 16:01 |
| maxel | AFAIK I only changed the default port, and I checked the new port after upgrading | 16:02 |
| blackflow | maxel: are you sure sshd is listening on that port? if you changed it back to your custom port, did you restart ssh.service? | 16:02 |
| maxel | I should try an upgrade again, I keep going back to my snapshot of this vm after running into so many problems after an upgrade | 16:02 |
| blackflow | or reload, that should suffice | 16:02 |
| maxel | yeah, I checked with a netstat -tlpn to see it's listening on the correct port | 16:02 |
| blackflow | maxel: journalctl -eu ssh.service might have clues | 16:03 |
| maxel | so I'm going back to my pre-upgrade state, I'm going to opt to keep my custom configs for any service. there are a couple prompts when upgrading asking which I want | 16:03 |
| blackflow | ideally you should always keep local config modifications and then inspect them later to see if anything needs to change. | 16:05 |
| maxel | ok, and I also noticed when ugprading it has an issue because I have postgresql running and it does not seem to like that | 16:06 |
| maxel | should I just shut that service down before upgrading? | 16:06 |
| tomreyn | no, you should just keep services running. | 16:10 |
| tomreyn | but "it has an issue because I have postgresql running" is really not very easy to respond to because it's so unspecific. | 16:11 |
| maxel | I'll post on here exactly what the prompt during upgrade is | 16:11 |
| tomreyn | good plan | 16:11 |
| maxel | it's not an issue, it just asks because of I think some libraries that are running when upgrading | 16:11 |
| maxel | so I'm getting back to a clean, upgraded state in my 16.04 install | 16:12 |
| tomreyn | so it's just a prompt, not an issue? | 16:12 |
| maxel | right, it just made me paranoid because the description sounded like it couldn't perform an upgrade on those libraries since they were being used by the service | 16:12 |
| tomreyn | so then there's no need to post it here unless something actually goes wrong. | 16:13 |
| blackflow | maxel: over the years I found it best to boot into single user + network mode and upgrade with every service shut down. | 16:13 |
| blackflow | ¬. | 16:13 |
| blackflow | oops | 16:13 |
| blackflow | ¬. | 16:13 |
| maxel | ok, so I am on 16.04.6, 0 packages can be updated, 0 security updates. 18.04.2 LTS is available | 16:15 |
| maxel | I'll make a snapshot here before I do the release upgrade again | 16:15 |
| tomreyn | run ubuntu-support-status --show-unsupported | 16:16 |
| maxel | http://paste.ubuntu.com/p/HbZXqWvmYJ/ | 16:17 |
| tomreyn | those "unsupported" ones are not supported by canonical (but may or may not be supported by the community or 3rd parties). they have an upgrade path with your current apt sources. | 16:19 |
| tomreyn | "no longer downloadable" ones are not 'backed by' an apt source, apt does not know how they got there, or what to do with them, really. they are 'foreign'. | 16:20 |
| maxel | I don't think I'm using them anymore. sounds like you think I should remove those | 16:21 |
| tomreyn | those are not supported by canonical, and they have no upgrade path (security patches), unless you maintain those differently. | 16:21 |
| tomreyn | i would not want software installed which has no way to get security patches. | 16:22 |
| maxel | were you referring to the no longer supported only, or also the unsupported packagfes | 16:23 |
| maxel | and also, is there an easy way to clean up/remove the unsupported packages? | 16:23 |
| tomreyn | anything i wrote after i meantiuoned "no longer downloadable" referred to packages which are in the "no longer downloadable" section of the output you posted. | 16:24 |
| tomreyn | *mentioned | 16:25 |
| tomreyn | https://github.com/tomreyn/scripts#foreign_packages outputs the "no longer installable" ones in a form that is easier to parse. it also lists packages which are in versions not available in the currently active apt repositories. | 16:26 |
| tomreyn | it does not discuss "unsupported" packages, though. | 16:27 |
| maxel | alright, hopefully this helps, removing some packages before upgrading | 16:30 |
| tomreyn | you could temporarily disable all apt sources which are not supported by Canonical, then run the foreign_packes script, this would cause those packages ubuntu-support-status considers "unsupported" to be listed as "no longer downloadable" | 16:30 |
| tomreyn | i don'T expect the packages ubuntu-support-status lists under "Unsupported" to cause problems during the release upgrade. | 16:31 |
| tomreyn | at least not those that come from ubuntu apt sources | 16:31 |
| maxel | ok, cleaned everything up and am trying a release upgrade again | 16:37 |
| maxel | first thing coming up is "some third party entries in your sources.list were disabled. you can re-enable them after the upgrade with the 'software-properties' tool or your package manger." | 16:38 |
| maxel | I just don't know what that is referring to or what could be affected | 16:39 |
| maxel | 3 packages no longer supported by canonical | 16:39 |
| tomreyn | some of your packages are clearly from 3rd party repositories (apt-cache policy lists actiive apt repositories). personally, i would remove any packages from 3rd party repositories before upgrading. | 16:39 |
| tomreyn | (or test that they don't cause problems during the upgrade) | 16:41 |
| maxel | ok, so this was the postgres issue I was talking about. I've got a prompt about libc6. it says "running services and programs that are usign NSS need to be restarted, otherwise they might not be able to do lookup or authentication any more. The installation process is able to restart some services (such as ssh or telnetd), but other programs cannot be restarted automatically. One such program that needs manual stopping and restart | 16:42 |
| maxel | after the glibc upgrade by yourself is xdm - because automatic restart might disconnect your active X11 sessions." | 16:42 |
| maxel | and it found postgresql as the service | 16:42 |
| maxel | I should be fine to upgrade glibc now? | 16:43 |
| maxel | also, I don't know how I would test packages don't cause problems during the upgrade | 16:43 |
| tomreyn | you run a graphical login manager on a server? | 16:43 |
| maxel | not that I'm aware of | 16:43 |
| maxel | if something is, I don't actively use it | 16:44 |
| maxel | I only use ssh to connect to this server | 16:44 |
| tomreyn | xdm is that | 16:45 |
| tomreyn | this message basically tells you that if you proceed, you will need to ensure all services are restarted in the end. but since you will be rebooting anyways, this is nothing to worry about. | 16:46 |
| maxel | ok | 16:46 |
| tomreyn | but you really should review the packages you have installed if this is a (clone of a) production system | 16:47 |
| maxel | it is a production system only for myself. it's just a samba share host and it runs miscellaneous services I use | 16:49 |
| tomreyn | i see | 16:49 |
| maxel | so I appreciate the help! | 16:50 |
| tomreyn | you're welcome. | 16:51 |
| maxel | this server had turned into something I'm terrified to touch because I do not want to mess up that shared drive, but I really need to get more comfortable with ubuntu | 16:52 |
| maxel | alright so now I'm to choosing which config to use. sshd_config is up first, and I believe you said I should be able to use the existing config | 16:53 |
| tomreyn | i don't think i said so, no. but if changes are needed, it should be brought up during this phase. | 16:56 |
| maxel | ok, I'm giving a shot retaining existing configs for sshd and nginx. now I'm removing packagges | 16:57 |
| blackflow | question... are service configs comin from Debian preferred, or is Ubuntu willing to change them for the better? Eg. munin-node is confiugred to run in the background, as Type=forking. I presume because Debian wants to keep the illusion of init freedom. Ubuntu doesn't. Would a patch be accepted to make it Type=simple? | 17:37 |
| JanC | did you ask the Debian maintainer? | 17:40 |
| blackflow | nope. But I know what their answer will be. "we support sysv which requires type=forking etc..." | 17:40 |
| blackflow | since Ubuntu does not need to support sysv (I assume?!), then I suppose it could have proper service unit files, properly confined and run in most optimal fashion, but that deviates from Debian. | 17:41 |
| blackflow | I'm running munin that way, and am testing confinement, and I believe those settings are generally useful in Ubuntu's Munin. | 17:41 |
| sarnold | my guess is such a change would be more likely to be accepted if we already have a delta from debian for a given package | 17:42 |
| JanC | Debian might be open for a solution that adapts to the init system if that's possible | 17:42 |
| blackflow | I even have some apparmor profiles but those are extremely hard to write generally useful because of so man plugin options | 17:42 |
| blackflow | so *many | 17:42 |
| blackflow | sarnold: I've got maintainership experience with other systems, but .deb is outright scary and I wouldn't know where to begin checking that. | 17:44 |
| sarnold | blackflow: yes | 17:44 |
| blackflow | JanC: I don't think they are. I alredy asked for some other services that suffer from teh same problem: Type=forking even though they support being run as simple or even, in some cases, as notify | 17:44 |
| sarnold | blackflow: you can see the delta that ubuntu carries from debian on https://patches.ubuntu.com/ | 17:45 |
| JanC | I assume it probably depends on who the Debian maintainer is... | 17:45 |
| blackflow | sarnold: yea there's a tiny patch fixing permissions | 17:47 |
| blackflow | JanC: so, start by asking the Debian maintainer, and if they refuse, try to fix it downstream in Ubuntu? | 17:48 |
| blackflow | I mean... am I correct in assuming that Ubuntu does not _need_ to support other inits? systemd is the only officially supported? | 17:48 |
| sarnold | yes | 17:49 |
| blackflow | because these changes I have in mind would not be possible otherwise. | 17:49 |
| JanC | if it can be fixed in Debian, I'm sure the Ubuntu developers prefer that | 17:49 |
| blackflow | I can try that, see what happens. | 17:49 |
| blackflow | sarnold: oh btw, that ZFS wiki... was I talking to you about it? can't remember... neway, folks from #zfsonlinux said they had the wiki access and would fix it, but I don't think they did... | 17:51 |
| maxel | so I just finished my upgrade to 18.04 lts. my ssh service is no longer running and I'm not sure why the upgrade stopped the service | 17:51 |
| blackflow | sarnold: oh YOU removed those..... thanks! :) | 17:52 |
| sarnold | blackflow: indeed, someone said they'd get to it, but a few days alter I noticed it wasn't done, so I just removed the ones that were outright ridiculous | 17:53 |
| sarnold | blackflow: I think ptx0 preferred them there so he had something to laugh at :) | 17:53 |
| maxel | ok, I've recreated the situation I do not understand. I stared up ssh service, and I am able to connect with putty, but as soon as I type in my password it kills the connection | 17:53 |
| blackflow | sarnold: of course we would :) | 17:54 |
| blackflow | *he | 17:54 |
| blackflow | sarnold: it was me who asked you about it, but we were having a discussion in #zfsonlinux about that, and then later someone said they had the access (as I myself don't), and they would fix it, so I left it at that. | 17:55 |
| sarnold | blackflow: aha :) it was long enough ago that I've forgotten the details.. and long since out of scrollback :) | 17:56 |
| blackflow | then ptx0 kicked me out of the chan because I said Oracle vs Google was about Dalvik, and I never bothered to return in that cesspool of a chan, and forgot about the wiki. | 17:56 |
| sarnold | rofl | 17:56 |
| blackflow | banned me for 24 hours actually :) | 17:56 |
| sarnold | that sounds like him. yeah. | 17:56 |
| blackflow | yeah :) | 17:56 |
| blackflow | never actually managed to get any meaningful help from #zfsonlinux, so there's no loss. :) | 17:57 |
| blackflow | maxel: do you have server side access? the service journal entries might hold a hint | 17:58 |
| blackflow | journalctl -eu ssh.service | 17:59 |
| maxel | yeah, I found the journalctl error, got a PAM access denied, tweaked the sshd_config to not use PAM and it's working | 17:59 |
| maxel | I'm not sure why it changed through the upgrade though | 17:59 |
| Mead | hello, I just installed 18.04-server on a system, strange thing is that it gets a ipv6 address from my gateway but not a ipv4. | 17:59 |
| blackflow | maxel: UsePAM is default, so if you didn't have it before, it must've been your change | 17:59 |
| maxel | and also it is not automatically starting the ssh service | 18:00 |
| blackflow | maxel: is it enabled? systemctl status ssh.service | 18:00 |
| maxel | loaded, but inactive after a restart | 18:00 |
| maxel | I started it up manually last boot | 18:00 |
| blackflow | oh wait, I think it's now socket activated? | 18:01 |
| blackflow | as in, won't start until first attempt on the port | 18:01 |
| maxel | it is denying my attempt to connect, well not denying | 18:01 |
| maxel | but my connection doesn't work | 18:01 |
| blackflow | until you manually start ssh.service? | 18:02 |
| maxel | right | 18:02 |
| maxel | it might be because my server is starting in emergency mode | 18:03 |
| maxel | I've never had it in that state before though | 18:03 |
| blackflow | just checked, there's ssh.socket but it's not require'd by ssh.service, so that's not being socket activated, I guess. look through the logs, if it's enabled, but doesnt' start, is something preventing it? | 18:04 |
| maxel | hmm, ok. and the other problem I'm wrestling with is the mounting of a filesystem failing. the log just says "wrong fs type, bad option, bad superblock" to a raid disk that is formatted in XFS that worked in my 16.04 install | 18:06 |
| maxel | I see the drive listed in both fdisk and parted | 18:12 |
| blackflow | maxel: what/how are you trying to mount exactly? | 18:18 |
| blackflow | can you pastebin the commands you run, and their outputs, and/or fstab itself? | 18:19 |
| maxel | here is my fstab: https://pastebin.com/TjLV499M | 18:22 |
| maxel | trying to remember what I can look at to see the actual command being run when mounting | 18:23 |
| maxel | internet is being spotty | 18:24 |
| blackflow | maxel: is that hardware raid, or software/mdadm? if the latter, you must mount the appropriate /dev/mdX | 18:24 |
| maxel | it is raid, and mounted through a raid card in the server and then shared through esxi to the vm | 18:24 |
| blackflow | so the VM has no idea it's raid? | 18:25 |
| maxel | I don't believe so | 18:25 |
| maxel | I'm not sure how to validate that | 18:25 |
| blackflow | btw what do you mean by "mounted through a raid card?" fstab needs a block device | 18:26 |
| maxel | I just mean to set up the raid I had to install a raid controller in the copmuter, and connect my 4 drives to that | 18:26 |
| blackflow | I don't know what "shared through esxi" means but if it's anything like "sharing an external directory" into the VM, like host-guest dir sharing, that's not gonna work | 18:26 |
| maxel | and then I created the raid volume through an interface to that raid card | 18:26 |
| blackflow | and then you formatted the resulting RAID _device_ with xfs? | 18:27 |
| maxel | I'm trying to describe that esxi (the hypervisor OS) sees the volume fine, and that I had to basically share the resources with this vm | 18:27 |
| maxel | I'm sorry I am so unhelpful here, I can't even remember how I set up the raid volume anymore. I believe the raid card formatted the volume and it happened to use xfs | 18:28 |
| blackflow | "share" how, I have no idea what vmware does there. for that fstab line, /dev/sda needs to be a block device. is it? does `blkid` run in the VM show it? | 18:28 |
| blackflow | maxel: that's the thing with hardware raid. it's proprietary who-knows-what-and-good-luck-if-you-have-issues kind of nonsense. | 18:28 |
| maxel | ah, and I needed hardware raid because I wanted raid 5, and that can't be done with my mobo | 18:29 |
| maxel | blkid seems to see the volume yes | 18:29 |
| blackflow | also, is it just the fact that you're maybe missing a partition on that device? it's a bit unusal the entire drive is being mounted like that. usually there's a GPT or MBR-based partition table on them | 18:29 |
| blackflow | what does `parted /dev/sda unit mib print` (run as root) show? | 18:30 |
| maxel | certainly possible, every memory of how I initially set this up has escaped me, so I'm struggling with some of these questions | 18:30 |
| blackflow | maxel: mdadm doesn't need motherboard support. "fakeraid" does, but that's not good either. | 18:31 |
| maxel | 1 0.00MiB 7680000MiB 7680000MiB xfs | 18:31 |
| mdeslaur | kstenerud: how far did you get with php 7.2.16? looks like 7.2.17 is out with some security fixes: bug 1823386 | 18:31 |
| ubottu | bug 1823386 in php7.2 (Ubuntu) "[MRE] Please update to latest upstream release 7.2.17 & 7.3.4" [Undecided,New] https://launchpad.net/bugs/1823386 | 18:31 |
| blackflow | maxel: ah so you need /dev/sda1 in that fstab line | 18:31 |
| blackflow | and you mount /dev/sda1 and not /dev/sda | 18:31 |
| maxel | is that a normal thing to change in a release upgrade? | 18:32 |
| blackflow | change what exactly? | 18:32 |
| maxel | the sda volume name or whatever that is | 18:32 |
| blackflow | I don't think anything changed there. | 18:32 |
| maxel | I'm not sure if you saw what I was doing before, but this volume was working 2 hours ago in ubuntu 16.04 before I ran the release upgrade | 18:33 |
| blackflow | unless.... can you actually please pastebin the full output of that parted command? I'm interested in the partition table shown | 18:33 |
| maxel | pastebinit is nor working for some reason | 18:34 |
| blackflow | maxel: `parted /dev/sda unit mib print | nc termbin.com 9999` | 18:34 |
| maxel | https://pastebin.com/5SG2vqsd | 18:34 |
| blackflow | sudo parted if you're not root | 18:34 |
| friendlyguy | hi there! i just upgraded a server from ubuntu 1604 to 1804, but there were some errors during the upgrade | 18:35 |
| maxel | friendlyguy, I'm going through the same process right now | 18:35 |
| friendlyguy | now dhcp isnt working any more, had to configure the interface manually | 18:36 |
| friendlyguy | there is an error with systemd-shim | 18:36 |
| friendlyguy | i tries to override a file and comes back with an error that its not allowed | 18:37 |
| blackflow | maxel: I see, so that's "loop", no partition table there. that's a bit weird setup. dunno if supported by xfs actually. could be that's what changed. | 18:37 |
| maxel | hmmm, well I set something up that is over my my head apparently | 18:38 |
| blackflow | maxel: or you know what, this could all be red herring. if you didn't format it as xfs and are expecting the raid system to have done so, just reformat it yourself | 18:38 |
| friendlyguy | /usr/share/dbus-1/system-services/org.freedesktop.systemd1.service with /usr/share/dbus-1/system-services/org.freedesktop.systemd1.service.systemd | 18:38 |
| blackflow | maxel: but if you DID have data there, then DON'T | 18:38 |
| maxel | I do have data that I want to keep | 18:38 |
| maxel | this is where I'm confused because however it is set up, it works pre-upgrade | 18:39 |
| maxel | I can go back to a vm snapshot and see everything fine | 18:39 |
| blackflow | maxel: try to mount it without specifying the fs type. `mount /dev/sda /mnt` and see if that works | 18:39 |
| maxel | but when I come back to the 18.04 ugpraded snapshot there are problems | 18:39 |
| maxel | how would I even validate that worked? | 18:40 |
| blackflow | maxel: newer kernel, it's possible xfs changed some expectations, I wouldn't know sorry. | 18:40 |
| blackflow | maxel: if it didn't complain and there be files on /mnt, then it worked. | 18:40 |
| maxel | huh, yeah, looks like it worked | 18:40 |
| maxel | how do I unmount? I want to change it back to the original name I had | 18:41 |
| friendlyguy | ah... looks like renaming that file did the trick. now apt-get install -f continues | 18:41 |
| friendlyguy | lets see how that works out :) | 18:41 |
| blackflow | maxel: umount /dev/sda | 18:41 |
| blackflow | maxel: maybe it's something about those options, lemme see | 18:41 |
| friendlyguy | yay | 18:42 |
| maxel | hmm, umount says target is busy | 18:42 |
| teward | are you currently `cd`'d into wherever it's mounted? | 18:43 |
| maxel | I was not | 18:43 |
| maxel | just restarting the server now | 18:43 |
| blackflow | sudo umount it | 18:44 |
| blackflow | weirdly, it's very difficult to find xfs manpage :) | 18:44 |
| maxel | and ssh still isn't starting up automatically, but one problem at a time | 18:45 |
| maxel | here's the process and options it is attempting: Process: 1042 ExecMount=/bin/mount /dev/sda /media/BMFD -t xfs -o defaults,users,barrier=0 (code=exited, status=32) | 18:47 |
| maxel | so when I run mount with no arguments it seems to work | 18:47 |
| blackflow | maxel: well, I don't use xfs, but that barrier=0 options doesn't seem right. the manpage says its a boolean, so either barrier or nobarier should be specified | 18:47 |
| marz | When Debian goes stable there are no version bumps to the software. Is it the same with Ubuntu ? | 18:47 |
| blackflow | maxel: what process is that, can you pastebin the whole unit? | 18:48 |
| maxel | I can't even remember where those arguments are defined | 18:48 |
| blackflow | marz: yes with exceptions like SRU, and some desktopy things like FireFox being latest | 18:48 |
| blackflow | marz: in the xfs(5) manpage | 18:48 |
| maxel | you may be on to something, if the options changed then this could explain all my problems | 18:48 |
| sarnold | marz: it depends. firefox, chromium-browser, mysql, mariadb, etc get bumped.. the browsers to new versions as they are released, databases to the new minor releases.. | 18:49 |
| bipul | I am stuck while giving a chroot jail configuration at /etc/schroot/schroot.conf for more then one chroot jail. | 18:49 |
| blackflow | (which needed xfsprogs to be installed, and funnily google didn't return a "xfs manpage" but insisted if I meant ZFS) | 18:49 |
| bipul | Does anyone know? | 18:49 |
| blackflow | maxel: that wouldn't be unusual across kernel versions | 18:49 |
| sarnold | marz: we've had to do new version bumps for samba, in the past, but that was painful all aronud and we really hope to never do that again. | 18:49 |
| sarnold | marz: but yes we do *vastly* prefer to apply specific security patches | 18:49 |
| marz | I’ve seen updates to packages on the server that aren’t security updates | 18:50 |
| maxel | blackflow, yeah, that has to be it, I just ran the mount command manually and it worked | 18:50 |
| blackflow | maxel: but try the command with all the options you specified in that unit | 18:51 |
| maxel | right, I just changed barrier=0 to just barrier and it mounted | 18:51 |
| blackflow | maxel: eh, barrier=0 sounds more like nobarier | 18:52 |
| maxel | now I need to tweak the service so it works correctly | 18:52 |
| maxel | good point | 18:52 |
| blackflow | not sure it'd be wise to change that tho' | 18:52 |
| blackflow | I mean, barriers = good, on systems with no data checksums. | 18:52 |
| blackflow | slower, but more integrity there. | 18:53 |
| maxel | I can try both, I had some issues mounting that drive as a samba share and I don't remember what I had to do to get it to work | 18:53 |
| blackflow | I'd leave defaults (ie. don't specify it at all) if you didn't know exactly which one you need | 18:53 |
| sarnold | marz: yes, there's standard bug updates too; those should also be smaller fixes, but sometimes new minor versions are accepted there too | 18:55 |
| sarnold | hah | 18:55 |
| friendlyguy | humm, dhcp is working though it didnt update the resolv.conf file | 19:02 |
| friendlyguy | there is no name resolution | 19:02 |
| friendlyguy | any idea where to start searching? | 19:02 |
| maxel | blackflow, well, looks like you figured it out. it's all because of that barrier option. I removed that option and everything is working. drive is mounted, all the issues with services not working were because of the emerggency mode | 19:04 |
| maxel | so thanks a ton for the help! | 19:04 |
| blackflow | maxel: awesome! | 19:07 |
| maxel | I wish I could remember why I added that option to begin with | 19:07 |
| blackflow | maxel: usually to speed things up. barriers force flushes of writeback caches periodically so it's slightly slower in some cases, but better for integrity. | 19:08 |
| blackflow | I'd leave defaults, if you didn't know exactly what and why you need of it. | 19:08 |
| ahasenack | rbasak: another question is, there is a directory with a file in version 1 of the package, and in version 2 that directory isn't declared in d/dirs anymore, nor is that file used | 19:18 |
| ahasenack | while upgrading, dpkg warns that it can't remove a non-empty directory | 19:18 |
| ahasenack | I'm adding a preinst snippet for that, it checks the deb version and does an rm -f of the file, and the directory is taken care of by dpkg itself after the upgrade | 19:19 |
| ahasenack | sounds right? | 19:19 |
| ahasenack | it's in /var | 19:19 |
| ahasenack | /var/cache/<dirname>/file | 19:19 |
| RoyK | ahasenack: stop the process, remove the chache dir or move it somewhere else and restart the process | 20:16 |
| ahasenack | there is no process (no daemon, if that's what you mean) | 20:17 |
| RoyK | then just move the dir somewhere else and remove it once the problem is solved | 20:18 |
| ahasenack | the new version doesn't use that anymore | 20:23 |
| ahasenack | what I did is working, seems clean, I just wanted to double check | 20:23 |
| ahasenack | preinst: rm -f /dir/file | 20:23 |
| ahasenack | new package: debian/dirs doesn't mention /dir anymore | 20:23 |
| ahasenack | upgrade: dpkg takes care of removing /dir as long as it's empty (which preinst took care of) | 20:23 |
| === PonzoSporrigan is now known as ghost64 | ||
| === DalekSec_ is now known as DalekSec | ||
| === Wryhder is now known as Lucas_Gray | ||
| === ChibaPet is now known as mason | ||
| === boshhead_ is now known as boshhead | ||
Generated by irclog2html.py 2.7 by Marius Gedminas - find it at mg.pov.lt!