| coreycb | jamespage: looks like tempest commit 8d94885df02ea0a4826a1f271f011dfefd2c2ca9 is where the get_domain failure started | 01:07 |
|---|---|---|
| jonfatin- | Ubuntu 16.04 netinstalls are broken with archive mirror. here is my kickstart (hasn't changed in a year or so) https://pastebin.com/raw/3k7STH8M | 04:17 |
| jonfatin- | https://imgur.com/a/azXDjn9 | 04:19 |
| jonfatin- | Confirmed existing bug yay... https://bugs.launchpad.net/ubuntu/+source/linux-base/+bug/1820755 | 04:31 |
| ubottu | Launchpad bug 1820755 in linux-base (Ubuntu) "Netboot install fails due to linux-image-4.4.0-143-generic.postinst linux-update-symlinks not found" [Undecided,Confirmed] | 04:31 |
| jamespage | coreycb: bah - UCA qemu does not have the patch I thought it had - that's my bad | 06:07 |
| jamespage | it was in ca-patches but that had not refreshed when I ran the backport I think | 06:07 |
| lordievader | Good morning | 07:28 |
| === dpawlik_ is now known as dpawlik | ||
| blackflow | any suggestions on how to shut up the php-fpm warnings about "server reached max_children setting"? google isn't helpful. Yes, I know what the error means, no I'm not gonna raise the number of PHP procs. The requests should queue up, as is expected, by nginx. Yes, I'm monitoring it and will decide when to upgrade based on too many accepts waiting in the queue. So... there apparently isn't a | 08:43 |
| blackflow | config to shut that warning up, or I'm missing it from docs, google. | 08:43 |
| blackflow | other than completely dropping warnings for the pool, with ~E_WARN in error_reporting, which I don't really want to | 08:45 |
| blackflow | nvm, asking in ##php, thanks. | 09:00 |
| kstenerud | @cpaelzer if you have time, I'd like your comments on whether I'm on the right track wrt https://bugs.launchpad.net/ubuntu/+source/keepalived/+bug/1810583 | 09:32 |
| ubottu | Launchpad bug 1810583 in keepalived (Ubuntu) "Daily cron restarts network on unattended updates but keepalived .service is not restarted as a dependency" [High,Triaged] | 09:32 |
| cpaelzer | kstenerud: the bug has no recent updates, where would an update be to think about it? | 09:41 |
| kstenerud | I'm trying to tackle it, since I'm doing ha stuff now. There's a post here https://chr4.org/blog/2019/01/21/make-keepalived-play-nicely-with-netplan-slash-systemd-network/ which talks about an upstream fix, which is talked about https://github.com/acassen/keepalived/issues/836#issuecomment-382011552 | 09:43 |
| kstenerud | but it's a huge merge: https://github.com/acassen/keepalived/commit/8ecbb591994a567375d78239d075ed032d9f9b07 | 09:44 |
| kstenerud | I can tease out the individual commits, but it will take awhile, so I'm wondering if this is the best way to deal with this issue? The problem is that keepalived doesn't play nice with systemd-networkd. It drops virtual IPs, which renders the entire HA stack useless | 09:45 |
| cpaelzer | kstenerud: a lot to read ... still reading | 10:05 |
| kstenerud | yeah, sorry. | 10:06 |
| kstenerud | From what I've read, the fixes were put in place a year ago, but it's a huge merge, so there's a ton of little commits, most not directly related to the problem. I'm thinking that maybe I can find one or two commits that deal specifically with this problem, or at the least get a list of anyhing that touches vrrp_whatever.c | 10:07 |
| kstenerud | The only other mitigations people have found so far were to force reload of multiple daemons on a trigger, which can cause other problems... | 10:08 |
| cpaelzer | kstenerud: lets sort out what these things do please | 10:10 |
| cpaelzer | kstenerud: the referenced git entry does a lot of things as it is a whole fixes branch | 10:10 |
| kstenerud | yes, that's what I'm saying. I'm in the process of sifting through all the little commits to find the ones that are actually relevant | 10:11 |
| cpaelzer | kstenerud: and eventually all it provides in regard to the issue is that keepalived will realize that it "lost" its VIP to then action on it | 10:11 |
| kstenerud | I just want to make sure I'm going about this the right way | 10:11 |
| cpaelzer | can't promise to know the right way, but I can try to help :-) | 10:11 |
| cpaelzer | kstenerud: that feature we will get with keepalived 2.0 | 10:11 |
| kstenerud | Yeah that's cool. I just wanted your gut reaction to this, as in "oh god no you're doing this all wrong" :P | 10:12 |
| cpaelzer | but since we care on this issue on >=Bionic we need a solution for pre keepalived-2.0 anyway | 10:12 |
| cpaelzer | my gut reaction seems to make what we found in the blog a best practise - if it is seconded by a bunch of people to apply to all their problems | 10:12 |
| kstenerud | ok, then I'm going to dive deeper into the PR and find the commits that touch the vrrp stuff | 10:13 |
| cpaelzer | then I'd put it into the Readme file and maybe even a hint to the issue (bug) and recommended solution (ref to the Readme) in a NEWS update | 10:13 |
| cpaelzer | kstenerud: consider this from the blog (if that is true) "I came to the conclusion that this actually wouldn’t fit the actual problem, as keepalived would just note the removed VIP and failover to another machine. But I wanted to fix the underlying problem itself, instead of just coping with the symptoms" | 10:14 |
| cpaelzer | considering that, would identifying and backporting the changes (if even possible) help a lot? | 10:15 |
| kstenerud | It's still not a complete solution, but it does prevent the vrrp from going down and never coming back | 10:15 |
| cpaelzer | instead - if confirmed really good - we should make the config approach that the blog had best practise | 10:15 |
| kstenerud | true | 10:16 |
| kstenerud | we'll probably need both | 10:16 |
| cpaelzer | kstenerud: I think the two things are not mutually exclusive | 10:16 |
| cpaelzer | yeah | 10:16 |
| cpaelzer | identify the changes and evaluate if they are backportable at all | 10:16 |
| cpaelzer | get in touch with people reporting the bugs if the proposed configuration would work for them | 10:16 |
| cpaelzer | if backportable it seems a win to do so | 10:16 |
| kstenerud | yeah | 10:17 |
| cpaelzer | If not then it is to some extend your call - if keepalived without the fixes it totally broken then we might consider backporting keepalived-2.0 at some point | 10:17 |
| cpaelzer | but that is a despareate measure that has to be considered only later if needed | 10:17 |
| cpaelzer | as e.g. upgraders still have ifupdown and are probably happy | 10:17 |
| cpaelzer | kstenerud: if the config style mentioned in the blog really turns out to be superior that is mostly communication - as I said readme/news also whatever docs that we ahve on the net about it, maybe a blog post on your own | 10:18 |
| cpaelzer | kstenerud: TL;DR to me it seem the right way is going both ways and decide then | 10:19 |
| kstenerud | ok sounds good | 10:19 |
| cpaelzer | and there semes to be a lot of communication involved, so make sure you are in touch with people affected by the issue and also others running keepalived but not being affected | 10:19 |
| cpaelzer | we don't want to break one of these groups in favor for the other | 10:19 |
| cpaelzer | kstenerud: there is the sad truth that sometimes a fix has so much regression risk that even thou it would be nice to have we are not doing it and instead have to kindly ask to move to a newer version | 10:20 |
| cpaelzer | I love to fix issues for our users, but not breaking all other users sometimes wins - so consider "making 19.10/20.04 a great working solution" and then recommending an upgrade an alternative fallback | 10:21 |
| cpaelzer | and the new recommended config style that avoids the issue could be the documented and recommended solution for 18.04-19.04 then | 10:21 |
| cpaelzer | kstenerud: ^^ | 10:21 |
| kstenerud | ok | 10:22 |
| coreycb | jamespage: ohh do you think that explains tobias-urdin's issue? | 11:54 |
| ahasenack | good morning | 12:06 |
| rbasak | cpaelzer: in https://code.launchpad.net/~paelzer/ubuntu/+source/qemu/+git/qemu/+merge/364760 | 12:16 |
| rbasak | rmdir "$NEWCONFFILE" || echo "failed to remove $NEWCONFFILE" | 12:16 |
| rbasak | If that fails, the preinst will succeed. Is that intentional? | 12:16 |
| cpaelzer | rbasak: yes | 12:33 |
| cpaelzer | intentional | 12:33 |
| cpaelzer | I listed the output in the testcase | 12:33 |
| cpaelzer | that is when people have made customizations to the extend that we can't cover them all (e.g. placed other things in that dir) | 12:33 |
| cpaelzer | the intention is there to have the errors shown but NOT fail the package upgrade itself | 12:34 |
| cpaelzer | rbasak: feel free to convince me otherwise, we could as well make it a "please clean DIR foo to make this upgrade smoothly" + exit 1 | 12:34 |
| cpaelzer | I'd be fine with that as well | 12:34 |
| rbasak | Ah, I missed your bug comment, sorry. I see it now. | 12:35 |
| rbasak | There doesn't seem to be an obviously correct answer to what the behaviour should be. | 12:35 |
| cpaelzer | yeah but are imperfect, but we could come up with "more edgy cases" of something already being a very rare use case all day | 12:37 |
| cpaelzer | therefore this was the line I decided to draw | 12:37 |
| rbasak | Yeah | 12:37 |
| rbasak | I think what you've done is fine. | 12:37 |
| rbasak | I just wanted to check that the || echo failed was intentional. From your bug comment that I missed it's clear that it was and that you've thought this through. | 12:38 |
| rbasak | I think what you've done is fine> in intended behaviour I mean. I still need to check again that the code does as intended :) | 12:38 |
| rbasak | cpaelzer: so is that the last time you'll accidentally put a filename in to the target of a dh_install file? :-) | 12:39 |
| brektyme | Hi, I'm getting a segfault when pxebooting the installer on 16.04, at the dectect disks step, I'm not really sure what the issue is and I've not seen it before it started yesterday. Does anyone have an idea of what I sould look at or what the issue could be? | 12:43 |
| cpaelzer | rbasak: no promises | 13:09 |
| cpaelzer | but it certainly reduces the chances that I'll do so | 13:10 |
| brektyme | I found a workaround on https://bugs.launchpad.net/ubuntu/+source/debian-installer/+bug/1817358, that got me past the segfault but the installer is failing while installing the kernel, looks like a dependancy problem with linux-modules-extra | 13:14 |
| ubottu | Launchpad bug 1816846 in glibc (Ubuntu) "duplicate for #1817358 segfault in libc-2.23.so netinstall installation pxe" [Undecided,Confirmed] | 13:14 |
| === cryptodan_d is now known as cryptodan | ||
| jamespage | coreycb: might do | 13:36 |
| jamespage | coreycb: I need to finish off some ceph SRU Testing and I'll take another test run at stein/proposed | 13:37 |
| coreycb | jamespage: ok. i'm still trying to figure out that tempest domains issue. i narrowed down on the commit. it seems we need to account for the switch of the test class inheritance from base.BaseIdentityV3AdminTest to base.BaseIdentityV3Test. | 13:39 |
| coreycb | not sure how yet though | 13:39 |
| jamespage | coreycb: oh I think I know what that is | 13:39 |
| jamespage | coreycb: obs if you revert the commit its all good | 13:39 |
| jamespage | I think the problem is the test uses on of the pre-created users todo the get domain call | 13:40 |
| jamespage | the users are in admin_domain, but the get domain call is for the default domain | 13:40 |
| coreycb | jamespage: ok, that's along the lines I was thinking as well. switching default_domain_id to admin_domain didn't seem to fix it. | 13:41 |
| jamespage | coreycb: hmm | 13:44 |
| kstenerud | cpaelzer: All dependent packages of php7.2 now succeed, except rss-bridge, which failed due to being unable to connect to the deb archive: https://objectstorage.prodstack4-5.canonical.com/v1/AUTH_77e2ada1e7a84929a74ba3b87153c0ac/autopkgtest-disco/disco/armhf/r/rss-bridge/20190320_112057_9cf57@/log.gz | 13:56 |
| kstenerud | I tried restarting the build but I don't have permission | 13:56 |
| jamespage | coreycb: its working ok for me - using the ID of the admin_domain | 13:59 |
| coreycb | jamespage: oh great! maybe i've got something residual hanging around in my deployment. | 14:00 |
| cpaelzer | kstenerud: rsyslog was built just this morning | 14:02 |
| cpaelzer | kstenerud: maybe there was a problem publishign it in time for armhf | 14:03 |
| cpaelzer | kstenerud: give it an hour and then run it again | 14:03 |
| cpaelzer | kstenerud: I'm almost sure it will resolve then | 14:03 |
| kstenerud | I don't have permission to run it | 14:03 |
| cpaelzer | ok I can do that for you | 14:03 |
| kstenerud | ok thanks! | 14:03 |
| cpaelzer | kstenerud: to me it looks like the metadata in the repo was updated but the ftp not yet fully synced (or something like ti) | 14:05 |
| cpaelzer | kstenerud: running again http://autopkgtest.ubuntu.com/running#pkg-rss-bridge | 14:05 |
| cpaelzer | lets see in ~1h | 14:05 |
| cpaelzer | kstenerud: it passed that step already | 14:06 |
| jamespage | coreycb: it does indeed work | 14:07 |
| kstenerud | cool! | 14:07 |
| jamespage | coreycb: https://github.com/openstack-charmers/openstack-charm-testing/pulls/javacruft if you have cycles | 14:08 |
| coreycb | jamespage: merged, thanks | 14:14 |
| coreycb | jamespage: are you ok with releasing keystone LDAP backend fixes early to rocky-updates? they've only been there since last night but testing successful and it's critical. | 14:21 |
| coreycb | jamespage: fyi bug 1798184 bug 1820333 | 14:21 |
| ubottu | bug 1798184 in python-ldappool (Ubuntu Cosmic) "[SRU] PY3: python3-ldap does not allow bytes for DN/RDN/field names" [High,Fix committed] https://launchpad.net/bugs/1798184 | 14:21 |
| ubottu | bug 1820333 in keystone (Ubuntu Cosmic) "[SRU] ldap search should not encode attributes" [Critical,Fix committed] https://launchpad.net/bugs/1820333 | 14:21 |
| jamespage | coreycb: +1 | 14:21 |
| jamespage | its basically completely broken right now so a fast-track is appropriate | 14:22 |
| coreycb | jamespage: alright, i'll go ahead and release that then | 14:23 |
| jamespage | coreycb: stein-proposed passes our tempest tests! | 15:02 |
| jamespage | promoting all the things... | 15:02 |
| === dpawlik is now known as dpawlik_ | ||
| coreycb | jamespage: \o/ | 15:33 |
| genewitch | is there any reason to use --user with "pip" command, i'm having issues with certain libraries (libgpuarray specifically) where if the main application is pip installed with --user it can't see a hand compiled libgpuarray | 17:57 |
| tomreyn | maybe you need to change PATH or LD_LIBRARY_PATH | 18:17 |
| bin_bash | hello, I want to change the values in /etc/apt/apt.conf.d/20auto-upgrades to be "0" using "echo" how is this possible? | 18:30 |
| sarnold | it depends what's in that file. best is to just change whatever writes it to write whatever settings you need | 18:30 |
| bin_bash | well usually i just manually change it, but i'm trying to put it in a quick bash script | 18:31 |
| sarnold | sed -i is probably the better tool for the job | 18:31 |
| bin_bash | APT::Periodic::Update-Package-Lists "1"; | 18:31 |
| bin_bash | APT::Periodic::Unattended-Upgrade "1"; | 18:31 |
| bin_bash | this is what the file has by default | 18:31 |
| bin_bash | i want 1 to become 0 | 18:31 |
| bin_bash | hm sed is a good idea, i didnt consider that | 18:32 |
| bin_bash | sarnold: im looking at the manpage, not sure i'm seeing how to do this | 18:34 |
| tomreyn | here's a simple example: sed -i 's/^APT::Periodic::Update-Package-Lists "1";$/APT::Periodic::Update-Package-Lists "0";/' | 18:41 |
| tomreyn | here's a simple example: sed -i 's/^APT::Periodic::Update-Package-Lists "1";$/APT::Periodic::Update-Package-Lists "0";/' /etc/apt/apt.conf.d/20auto-upgrades | 18:41 |
| tomreyn | ^ I forgot the file to edit initially | 18:42 |
| tomreyn | but you'll really need to understand regular expressions, or you won't know what you need to escape. | 18:42 |
| tomreyn | try #bash | 18:42 |
| bin_bash | thanks | 18:43 |
| bin_bash | ideally i'd just replace 1 with 0 in that file | 18:43 |
| bin_bash | rather than the whole line | 18:43 |
| tomreyn | if you just want to replace any 1 by one character, that's also possible, but i think that's most likel ynot what you actually want. | 18:44 |
| tomreyn | *any 1 character by a 0 character | 18:45 |
| sarnold | you definitely want to make sure you know which exact config settings you're changing | 18:45 |
| bin_bash | it'd be weird if there was another 1 in the file | 18:45 |
| sarnold | apt has something like nine thousand config choices | 18:46 |
| bin_bash | not in this one file | 18:46 |
| sarnold | and there's no saying what other admins or tools have put in which files | 18:46 |
| tomreyn | or removed from them, or whether edited to be in a different format, or commented in or out | 18:47 |
| tomreyn | even regular expressions mey be unsuitable to help you there. | 18:48 |
| bin_bash | there shouldn't be anything else in this file except these two lines... | 18:48 |
| sarnold | yeah if the settings are set via the APT { Periodic { Update-Package-Lists } } } style, it'll be harder to edit | 18:48 |
| sarnold | bin_bash: maybe today .. who knows what'll be there in six months | 18:49 |
| tomreyn | people like you probably wrote software for boeing 737 max 8 | 18:50 |
| bin_bash | ... | 18:50 |
| bin_bash | wow | 18:50 |
| tomreyn | i'm not meaning to blame you there, just pointing out such things can really matter | 18:51 |
| bin_bash | that's gotta be one of the worst false equivalency attempts i've ever had lobbed against me. | 18:51 |
| bin_bash | I can't imagine how a single personal-use script is even comparable in your thought process | 18:52 |
| sarnold | man if you're upgrading your planes in flight .. something is way wrong :) | 18:52 |
| lordcirth | bin_bash, that was excessive. But if a thing is worth doing, it's worth doing right. | 18:52 |
| bin_bash | if you're running ubuntu on a plane, there's some serious fucked up shit going on | 18:52 |
| lordcirth | bin_bash, point is, don't hack something together poorly, do it right the first time | 18:53 |
| bin_bash | lordcirth: i'm really just trying to reduce having to manually change this in a deployment script for dev usage. i never suggested hacking it together, i'm simply looking for the easiest implementation for the specific use-case | 18:54 |
| tomreyn | yes, that was excessive, sorry bin_bash. but you should try to get into this kind of thinking - how risky is it if i do it wrong, and can i actually measure that reliably, or should i just try to make it reliable no matter what. | 18:55 |
| bin_bash | i'm only ever modifying THIS file. i'm only changing the values from 1 to 0, and was therefore askined the best way to implement this. even if more lines were added i would /still/ want them to change from 1 to 0 because i want to disable any automatic upgrades | 18:55 |
| sarnold | bin_bash: I still think you'd be better served by figuring out what's putting that file in place and using the correct values from the start | 18:55 |
| tomreyn | bin_bash: you mention deployment. many deployment frameworks provide a way to edit configuration files in a better way. | 18:56 |
| sarnold | if it's copying /usr/share/unattended-upgrades/20auto-upgrades into place, it might as easy as asking those scripts to copy /usr/share/unattended-upgrades/20auto-upgrades-disabled instead | 18:57 |
| bin_bash | sarnold: what? | 18:57 |
| sarnold | bin_bash: I don't have /etc/apt/apt.conf.d/20auto-upgrades on my systems; there *is* a /usr/share/unattended-upgrades/20auto-upgrades though | 18:58 |
| sarnold | bin_bash: so I'm curious if your deployment script is copying that file over.. | 18:58 |
| sarnold | bin_bash: in which case it'd be as easy as asking it to copy a *different* file .. | 18:58 |
| bin_bash | sarnold: which version? | 18:58 |
| sarnold | bin_bash: which would be way more reliable than copying the wrong file, and then editing it :) | 18:58 |
| sarnold | ii unattended-upgrades 1.1ubuntu1.18.04.9 all automatic installation of security upgrades | 18:59 |
| bin_bash | sarnold: your setup sounds weird... | 19:00 |
| tomreyn | i see /etc/apt/apt.conf.d/20auto-upgrades on 14.04, not on anything newer | 19:00 |
| bin_bash | the unattended upgrades file has always been in etc | 19:00 |
| bin_bash | this is 18.04 | 19:00 |
| bin_bash | and i found this stack overflow for 16.04, which has a similar sed command to what was suggested above: https://askubuntu.com/questions/1059971/disable-updates-from-command-line-in-ubuntu-16-04 | 19:01 |
| tomreyn | dpkg -S /etc/apt/apt.conf.d/20auto-upgrades | 19:01 |
| tomreyn | ok, ignore this, that's maybe misleading | 19:01 |
| bin_bash | this even says the same file too: | 19:02 |
| bin_bash | https://linuxconfig.org/disable-automatic-updates-on-ubuntu-18-04-bionic-beaver-linux | 19:02 |
| tomreyn | yes the file is created via postconf apparently | 19:04 |
| bin_bash | hm interesting | 19:05 |
| === englishm_ is now known as englishm | ||
| tomreyn | s/postconf/postinstall/ oops | 22:11 |
Generated by irclog2html.py 2.7 by Marius Gedminas - find it at mg.pov.lt!