[01:57] <patz0r> hey guys, i've been trying to work around an issue i've been having installing some new servers lately, think i have found the issue but i'm not sure if it's a big in the live installer
[01:57] <patz0r> i'm trying to do something really really simple, create an mdadm raid1 to install my OS on, but I can't get it to work at all with the live installer
[01:58] <patz0r> seems to work ok with the old style alternative iso
[01:58] <patz0r> but then i can't create bonds during the setup with that iso
[01:58] <patz0r> i wouldn't think it would be this hard tbh...
[01:59] <sarnold> are you using 18.04 or 18.04.1?
[01:59] <patz0r> 18.04.1
[01:59] <patz0r> i think the main issue is that it's booting in uefi
[01:59] <mwhudson> you can't have /boot on your raiid
[02:00] <mwhudson> which is something we need to fix
[02:00] <patz0r> why does it work with the non live installer?
[02:00] <patz0r> i would think it's a very common thing to do on bare metal machines
[02:02] <sarnold> mwhudson: is that limitation still around? I thought that'd been fixed by the .1 release? :(
[02:03] <patz0r> sorry if this is a common issue but i'm new to 18.04/18.04.1 and finding it doesn't work quite how i'd expect from past experience
[02:03] <mwhudson> sarnold: i don't think 18.04(.0) supported raid at all did it?
[02:04] <sarnold> mwhudson: yeah I thought .0 was pretty limited and .1 a lot better, but not yet perfect... I guess it's got a lot further to go than I realized
[02:05] <patz0r> do you guys have any suggestions for the easiest way to work around this? i've got 5 bare metal machines to install and I only have HBAs so I can't do any hardware raid
[02:06] <sarnold> patz0r: maybe try FAI?
[02:06] <sarnold> patz0r: the server is fai-server
[02:06] <patz0r> i have successfully installed my root partition to an md0 raid1 array using the alternative installer, and it did allow me to create 512mb efi partitions at the start of both disks
[02:07] <patz0r> do you think that would have installed the /boot partition to both disks successfully?
[02:07] <sarnold> patz0r: how terrible is it to try scripting the networking config after the fact?
[02:07] <patz0r> not hard will only take me a few mins to do the networking, it's a simple setup
[02:08] <mwhudson> the thing about /boot on raid is that it works (grub can read from raid just fine) but grub can't write to it so recordfail doesn't work
[02:08] <patz0r> i don't use any of the cloud init features from the live installer, i turn all of that off
[02:08] <mwhudson> so if you install a bad kernel, you need to interrupt the bootloader to unwedge yourself which isn't all that easy for servers in general
[02:08] <sarnold> mwhudson: wait what?
[02:08] <mwhudson> but that said d-i lets you shoot yourself in the foot this way
[02:09] <sarnold> mwhudson: I thoguht we just failed to properly configure grub to boot from a degraded array
[02:09] <mwhudson> (d-i will also let you try to put /boot/efi on raid5 which definitely doesn't work but well)
[02:09] <mwhudson> sarnold: this was my understanding anyway, i am not an expert
[02:09] <sarnold> me neither, my world starts with getty or sshd :D
[02:10] <mwhudson> the issue of booting from a degraded array is sort of orthogonal to what i am talking about
[02:10] <patz0r> i don't think there is anything in 18.04 that i actually need, do you guys think it's safer for me to use 16.04 at this stage?
[02:10] <patz0r> this will just be a storage box
[02:11] <mwhudson> patz0r: the 16.04 installer will work no better than the 18.04 alternate installer
[02:11] <patz0r> same bug in both?
[02:12] <mwhudson> what do you mean?
[02:12] <mwhudson> you said the 18.04 alternate installer didn't work because you couldn't configure a bond
[02:12] <mwhudson> if that's the case i don't expect the 16.04 d-i based installer would either
[02:13] <patz0r> that was just a minor inconvenience, i can configure the bond after the fact, i thought you guys were saying that the alternative installer would let me do this but i might still have issues if a disk failed
[02:14] <patz0r> i'd go pull a disk out and test it if i wasn't 4000km from my servers right now
[02:14] <patz0r> hence my caution in how i set this up now
[02:15] <sarnold> it'd be worth testing with a VM locally :)
[02:16] <sarnold> I've been meaning to do that for .. uh .. a very long time.
[02:16] <patz0r> that's probably a better idea sarnold
[02:16] <patz0r> are you guys ubuntu dev's ?
[02:16] <mwhudson> yes
[02:18] <patz0r> cool, thank you for your help with this, been getting frustrated with it as i thought it should be easy
[02:18] <mwhudson> this whole area seems to be spectacularly badly documented in genera;
[02:18] <mwhudson> (bonds too!)
[02:19] <sarnold> yeah... I apparently don't *know* what ought to be epxected to work and in which releases ..
[02:19] <patz0r> i like the bonds networking feature in the live installer, but overall i still prefer the old installer/old style
[02:20] <patz0r> i'm not the kind who's getting my hands dirty in linux daily though, i got too many other machines to worry about
[02:21] <patz0r> i'm going to test the alternative installer raid1 on a vm now, will report back on that shortly if it would be beneficial
[03:20] <patz0r> so i've tested it, doesn't work if the first disk in the raid suddenly disappears
[03:22] <patz0r> that's a little concerning :(
[03:58] <patz0r> anyone have any ideas on how to maybe leave /boot off my raid1 but periodically update it via crontab or something so it may not be too painful to fix if i have major issues down the track?
[14:22] <plm> Hi all =D
[14:22] <TheHonorableKit-> good morning
[14:41] <smoser> rbasak: do you have ideas on https://jenkins.ubuntu.com/server/job/git-ubuntu-ci/80/console ?
[14:44] <rbasak> smoser: looks like a regression in the nightlies too: https://jenkins.ubuntu.com/server/view/git-ubuntu/job/git-ubuntu-ci-nightly/
[14:44] <rbasak> smoser: probably some change to a module in PyPI
[14:45] <smoser> i didnt know we used pypi
[14:45] <smoser> we should not
[14:45] <smoser> where does pypi get used ?
[14:46] <rbasak> To create the snap
[14:46] <smoser> well, yes. but i thought we were installing ubuntu packages for python deps.
[14:46] <rbasak> We aren't.
[14:47] <rbasak> The problem is that we're stuck on Xenial, because snaps are stuck on Xenial.
[14:47] <rbasak> And we needed newer stuff.
[14:47] <smoser> well, then need to pin versions of stuff.
[14:47] <rbasak> So initially we used a snap based on a newer Ubuntu, but that broke horribly and we were told that it snaps based on !Xenial aren't supported.
[14:48] <rbasak> I believe Bionic based snaps are in testing right now, but I don't know to what extent that's useable right now.
[14:48] <smoser> i'm looking at the snapcraft.yaml, its not obvious to me how pypi gets used.
[14:49] <rbasak> To fulfil requirements of anything that says plugin: python
[14:49] <rbasak> https://git.launchpad.net/usd-importer/tree/setup.py
[14:49] <rbasak> But also for launchpadlib
[14:50] <rbasak> Pinning versions also caused problems for us in the past
[14:50] <smoser> how?
[14:50] <rbasak> Because stuff we directly specify doesn't pin themselves
[14:50] <TheHonorableKit-> any idea when Ubuntu Server will be updated to 18.10LTS?
[14:50] <rbasak> eg. we say argcomplete==1.8.1 but argcomplete 1.8.1 might not pin its dependencies
[14:51] <rbasak> And then they move and stuff explodes
[14:51] <smoser> yeah. there are solutions to that.
[14:51] <smoser> i thought
[14:51] <smoser> but yeah.
[14:54] <smoser> man. what a pain. :-(
[14:55] <sdeziel> TheHonorableKit-: 18.10 is not an LTS version
[14:56] <sdeziel> TheHonorableKit-: but 18.10 was released for the server edition already https://www.ubuntu.com/download/server
[14:59] <TheHonorableKit-> yes, I saw it was released. I'm on LTS though. guess I'll have to wait??
[14:59] <smoser> TheHonorableKit-: you can upgrade if you want
[14:59] <smoser> but that comes with maintenance
[14:59] <smoser> 18.04 is supported for 5 years.
[14:59] <smoser> 18.10 is supported for 9 months.
[14:59] <TheHonorableKit-> I did try, of course, but I'm somewhat new to linux administration. I had to revert to a backup because the upgrade failed miserably
[15:00] <smoser> so essentially, if you go to 18.10, in 6 months (or < 9m) you need to go to 19.04
[15:00] <TheHonorableKit-> if there's ever an LTS version of 18.10, I'll upgrade then
[15:00] <smoser> there will not be
[15:00] <smoser> the next lts will be 20.04
[15:00] <TheHonorableKit-> O.o well ****
[15:01] <smoser> if you just want to use some things from 18.10 or play with it to test things, i'd suggest looking at lxd
[15:01] <smoser> or multipass
[15:01] <smoser> lxd will provide you with containers to run whatever distro you want inside.
[15:01] <TheHonorableKit-> meh, I'm good with 18.04 for now
[15:01] <smoser> multipass will provide you with ubuntu in a kvm guest
[15:01] <smoser> lxd is really a wonderful tool, i really do suggest you try it out.
[15:01] <TheHonorableKit-> although I always did want to have a kali-linux distro run within an ubuntu distro, that would be fun
[16:19] <nacc> cpaelzer: rbasak: did you want to schedule some time to go over php?
[16:21] <rbasak> nacc: that'd be good, thanks. As a handover type thing? I'm not familiar with it at the moment so would need you to drive that meeting I think.
[16:22] <wr> need to setup dhcp, dns, ntp on ubuntu server 16.04lts, what docs link should follow?
[16:22] <cpaelzer> nacc: I was working with kstenerud for basic merge training and long term we hope he takes that over
[16:22] <cpaelzer> I asked him to get in contact with you next week
[16:23] <cpaelzer> having one experienced one like rbasak on it as well surely will be good this cycle
[16:24] <nacc> rbasak: cpaelzer: kstenerud: sounds good. Mostly about the process needed since it's such a maze of interdependencies
[16:26] <rbasak> nacc, kstenerud: so some time next week then? nacc: when works well for you?
[16:28] <kstenerud> I'm on European time, so hopefully not too late for me
[16:30] <sdeziel> wr: each of those services come with various implementations, have you picked the one you want already?
[16:31] <wr> sdeziel, need all
[16:32] <wr> sdeziel, isc dhcp
[16:32] <sdeziel> wr: right but each one of those have different implementations, hence the need to pick the ones you'd like to use
[16:33] <sdeziel> wr: https://help.ubuntu.com/community/isc-dhcp-server
[16:34] <sdeziel> wr: for the DNS, there is bind9 doc here: https://help.ubuntu.com/lts/serverguide/dns.html.en-CA
[16:34] <wr> sdeziel, yes bind9 thats the one gonna use
[16:34] <sdeziel> wr: for the isc DHCP server, this one might be more current https://help.ubuntu.com/lts/serverguide/dhcp.html.en-CA
[16:35] <sdeziel> wr: and finally for NTP: https://help.ubuntu.com/lts/serverguide/NTP.html.en-CA
[16:35] <sdeziel> wr: I basically just skimmed https://help.ubuntu.com/lts/serverguide/index.html.en-CA for you ;)
[16:37] <wr> sdeziel, awesome now i won't screw this up
[16:37] <sdeziel> wr: well, I did screw up pointing you to doc for 18.04. Please look at https://help.ubuntu.com/16.04/serverguide/index.html instead ;)
[16:47] <ghone> Hi all, our nginx on ubuntu (16.04) servers is being killed (restarted ?) randomly for a last couple of days. Any clue if any change has been applied recently to ubuntu ?
[16:48] <lordcirth> ghone, any errors in nginx log?  syslog? dmesg?
[16:52] <ghone> Nov 14 06:21:23 server systemd[1]: Reloading. Nov 14 06:21:23 server systemd[1]: Started ACPI event daemon. Nov 14 06:21:23 server systemd[1]: Starting A high performance web server and a reverse proxy server...
[16:52] <ghone> why systemd restarts nginx ? because is is being reloaded ? if so why it is being reloaded ?
[16:58] <rawco> so if my /boot is full and i can’t autoremove kernels, i have to manually delete them, right?
[17:02] <rawco> ok, ended up using dpkg to remove the old linux images
[17:03] <lotuspsychje> rawco: wich ubuntu release is this
[17:03] <rawco> 18
[17:04] <rawco> it looks like its downloading them again
[17:05] <rawco> i did apt-get autoremove -f
[17:11] <rawco> should i expand /boot ?
[17:14] <lotuspsychje> rawco: 18.04 should have unattended upgradesnow and auto cleanup kernels
[17:16] <sdeziel> ghone: could you pastebin the journalctl output of nginx?
[17:17] <joop_> hi people. I have a problem with ubuntu server 18.04.1 LTS on Dell Wyse thin clients (Z90DW).
[17:18] <joop_> Whenever I issue `shutdown`, the OS shuts down but the system does not perform a poweroff.
[17:18] <joop_> Radeon module is disabled in the kernel, nomodeset flag and acpi=on/off/etc flags have been tried.
[17:19] <ghone> sdeziel: https://pastebin.com/FjsheCaz
[17:20] <sdeziel> ghone: nginx[73195]: nginx: [emerg] open() "/etc/nginx/ssl/privkeypassword.txt" failed (2: No such file or directory) in /etc/nginx/sites-enabled/ji
[17:20] <joop_> /etc/nginx/ssl/privkeypassword.txt
[17:20] <joop_> yeah.
[17:20] <ghone> Yes I know
[17:20] <ghone> but why it is being restarted by systemd ?
[17:21] <ghone> it happens at least once a day ...
[17:21] <sdeziel> ghone: maybe it's the logrotate job that signals it?
[17:22] <sdeziel> ghone: it usually runs around 6:20 in the morning
[17:24] <sdeziel> joop_: have you tried "halt -p" ?
[17:25] <joop_> sdeziel: Yes, that's been tried.
[17:25] <ghone> sdeziel: logrotate sends USR1 which triggers nginx to start new log files but not restart
[17:25] <joop_> I have tried every combination / permutation of the shutdown and halt commands and their parameters.
[17:25] <joop_> I have tried some kernel flags in /etc/default/grub
[17:25] <sdeziel> joop_: sorry, that's not an area I know well
[17:26] <leftyfb> joop_: have you tried this using a live cd/usb?
[17:27] <joop_> leftyfb: yep, I have.
[17:27] <joop_> Lubuntu live - didn't work either.
[17:27] <leftyfb> what type of hardware is this? Maybe it doesn't support acpi commands?
[17:27] <joop_> Ubuntu server after installation - 'Remove intallation media and press enter' -> done that, doesn't power off either
[17:28] <joop_> leftyfb: AMD G-T56N processor with Radeon HD6310 graphics on a Dell Wyse Z90DW thin client./
[17:29] <leftyfb> joop_: tried a different linux distribution to see if it's actually an ubuntu problem as opposed to a linux kernel problem?
[17:29] <joop_> leftyfb: https://paste.ubuntu.com/p/YXQXDBZr4W/
[17:29] <leftyfb> or hardware
[17:29] <sdeziel> ghone: "/sbin/start-stop-daemon: warning: failed to kill 20255: No such process" is suspicious, as if the PIDfile contained the wrong pid
[17:30] <sdeziel> ghone: I'd recommend fixing the config error, stopping nginx, starting it and confirming the right pid ends up in the pidfile
[17:30] <joop_> Well the weird thing is, a colleage of mine tried to install FreeBSD and had nothing but issues with the GPU but he reported that OpenBSD works fine - including system poweroff.
[17:30] <leftyfb> OpenBSD != linux
[17:31] <ghone> sdeziel: it happens on a few servers ... no just one ...
[17:31] <ghone> sdeziel: it happens on a few servers ... not just one ...
[17:31] <joop_> No, it's BSD, i know, but still, it is known to work.
[17:31] <joop_> wondering...
[17:32] <ghone> sdeziel: we restart it manually and it is running for about a day and then being shut down ...
[17:32] <sdeziel> ghone: have you checked cron jobs and systemd timers looking for stuff touching nginx? automated configuration system?
[17:41] <joop_> I love the Wyse thin client but this is bugging me.
[17:42] <joop_> Now trying with acpi=strict.
[17:42] <joop_> Didn't change.
[19:34] <nacc> rbasak: yes, next week would be fine with me, i can try and work around kstenerud's schedule, i'm a bit further east now
[20:33] <sdeziel> I'd appreciate if someone with systemd experience could comment on the proposed fix in LP: #1803601, please
[20:33] <ubottu> Launchpad bug 1803601 in base-files (Ubuntu) "motd-news.service scheduled even when /etc/update-motd.d/50-motd-news is not executable" [Undecided,New] https://launchpad.net/bugs/1803601