[00:09] is there an ubuntu terminal that supports the OSC 52 ;; would allow set-clipborad in terminal [00:21] probably xterm does; maybe urxvt .. [00:27] rbasak: not sure if you want to just land that fixlet, or try and build it locally to sanity check it is ok? I suppose in the edge channel it's ok to just land it [00:28] ahasenack: as to the error messages, remember when running from the snap it is only looking in the snap's env, so your system packages are irrelevant [00:28] ahasenack: it's just a generic message, which implies that it wasn't able to load pygit2 successfully [00:28] s/load/import/ [01:00] Just joined to grab topic links and say cheers it fixed my issue~ === giraffe is now known as Guest29863 [06:01] Good morning [09:12] <^kiokoman^> wellcome2 [09:12] <^kiokoman^> wrong chat sorry [10:10] cpaelzer, ahasenack: git-ubuntu edge fixed I believe [10:10] rbasak: rev 429 or newer? [10:12] rbasak: yep working on 429 now [10:12] thank [10:12] s [11:39] hello, im trying to setup a hostroute on ubuntu 18.04 [11:39] http://pastebin.centos.org/782911/ [11:40] is the relevant snippet, however when trying to access the 10th subnet im getting no route to host [11:40] it works perfectly fine when i set a 192.168.0.0/24 ip on the server, however i need the ip to be 192.168.2.0/24 [11:40] any ideas? [11:42] DK2: you need to have a gateway in the network [11:42] it is on-link [11:42] both subnet are in the same vlan [11:43] same vlan perhaps but not in the same network [11:43] but shouldnt hostroutes work for exactly cases like this? [11:43] how can you can to the 192.168.0.0/24 where is your gateway from 192.168.2.0/24 ? [11:43] how can you go * [11:44] there is no gateway, it should use 192.168.0.254 [11:48] for me, or the dest is on the same network => no gateway, or is another network => gateway [11:52] https://netplan.io/examples#directly-connected-gateway [12:05] but doesnt seem to work [12:17] thedac: coreycb: any update on https://bugs.launchpad.net/bugs/1750121 ? my suspicion is that this is somehow dependent on your CI/charms setup [12:17] Launchpad bug 1750121 in neutron-dynamic-routing (Ubuntu Bionic) "Dynamic routing: adding speaker to agent fails" [High,Fix committed] [13:12] so, http://pastebin.centos.org/782951/ [13:12] how can i add a second route here? [13:12] i also want 10.82.27.0/24 to go trough .254 [13:59] How strong is the recommendation to keep ubuntu-server on a server? I'd like to remove some of the packages this packages depends on a smaller server. [14:07] rbasak: I thought there was a way to pass extra arguments when using git-ubuntu build-source, like genchange's -v, but I don't see that in the --help output. Is there a way? [14:10] ah, I think it's anything at the end that is not recognized as a git-ubuntu flag? [14:10] if len(args.rem_args) != 0: [14:10] logging.warning( [14:10] "Appending specified flags (%s) to `dpkg-buildpackage %s`. " [14:11] * ahasenack tries that [14:37] I'm running 16.04.4. Whenever I reboot after a kernel upgrade, I run apt autoremove, then it makes me reboot again :| [14:37] any way to get around that? [14:45] Run the autoremove after the kernel upgrade but before rebooting the first time [15:18] good point [15:50] runelind_q: seems like an apt bug if you ask me [15:50] I agree, it shouldn't need to happen. [15:50] there were bugs filed about it in 2015 [15:52] runelind_q: well, ubuntu doesn't really have a medal in bug fixing [15:53] why fix bugs when you can add more features? ;p [16:09] runelind_q: exactly [16:09] runelind_q: I don't use ubuntu much on servers anymore - I went back to debian [16:10] * RoyK knows he's swearing in church [16:14] ahasenack: -- ... [16:14] ahasenack: (cf. `git ubuntu build --help`) [16:15] I do feel like there are much more kernel security updates in Ubuntu compared to CentOS/RH, maybe they're just not fixing them ;p [16:16] ahasenack: i'm not sure if i've tested buildsource -- , tbh [16:16] ahasenack: but i think it does work, and needs a MP to add that to the help epilogue [16:17] nacc: -- ... means "MS" ;) [16:18] RoyK: I don't know what you're talking about. [16:18] RoyK: you mean morse code? [16:19] :) [16:38] rbasak: ok, looks like git-ubuntu.self-test now passes in edge? [16:38] ahasenack: --^ fyi [16:52] nacc: it worked (-- ...) [16:57] frickler: We still see that problem. By chance are you at the OpenStack Summit? I'd love to show you what we are seeing. [16:58] We are not doning anything special in our CI setup. And again the intial peering setup works. It is only after restarting the neutron-bgp-dragent that we see the error. [17:00] ahasenack: cool [17:11] thedac: yes, that's because the original bug also doesn't appear in the default initial setup. I only discovered it when setting up a second network node with another dr-agent [17:11] thedac: and no, I'm not at the summit [17:12] frickler: what info can I provide to move forward? I can tar up neutron.conf etc [17:15] thedac: a description how to reproduce would be good. or would there be a way to hold a failed CI node and let me debug it? like we could do on openstack CI? ;) [17:17] frickler: oK, I'll do a completely manual setup based on the testing documentation (what our CI automation is built on) and see there is any difference. [17:17] The CI env is inaccessible, unfortunately. [17:18] thedac: is the testing doc public? [17:19] frickler: yes, https://docs.openstack.org/neutron-dynamic-routing/latest/contributor/testing.html [17:20] Our CI scripting is almost exactly that ^^^ [17:20] Only point to highlight is the neutron-bgp-dragent is on its own node. [17:23] thedac: ah, that's one of the things I asked on LP. you need to install the same pkg version of the bgp agent on the neutron server, because it serves the RPC call from there [17:23] my patch changes both sides and if they are not in sync, you get that error [17:24] frickler: they both have the same version of python-neutron-dynamic-routing [17:29] ahasenack: cool, can you file a bug that build-source's help doesn't mention it, and fix it? :) [17:45] thedac: hmm, o.k., so I guess it's best if you try to reproduce it manually. for me, the only attempt where I saw your error was when I upgraded from current pkgs to proposed and didn't restart neutron-server. maybe you could try that restart anyway, just to confirm that you don't have the same situation [17:46] frickler: ok, I'll do a few more tests and report back on the bug [17:51] Hi guys, I am struggling with converting my network config to netplan, anyone around that could assist? Would appreciate it, thanks :) Old Config: https://pastebin.com/YjMRhhAa New Netplan Config that does not work: https://pastebin.com/2msq7hYV [17:54] rbasak: would you be interested in this mysql 18.04 bug https://bugs.launchpad.net/ubuntu/+source/mysql-5.7/+bug/1772337 that has apparmor DENIED messages in dmesg? (https://launchpadlibrarian.net/371151468/Dmesg.txt) [17:54] Launchpad bug 1772337 in mysql-5.7 (Ubuntu) "package mysql-server-5.7 5.7.22-0ubuntu18.04.1 failed to install/upgrade: installed mysql-server-5.7 package post-installation script subprocess returned error exit status 1" [Undecided,New] [18:16] name="/etc/my.cnf" [18:16] first, that's a silly path mysql, and you should feel bad for using it.. second, that has the look of something that ought ot have been included in profiles ages ago.. [18:17] wait, that path looks wrong... [18:17] shouldn't it be pulling from the mysql conf roots? [18:17] (mysql i mean) [18:18] ahasenack: rbasak: ^ [18:18] apyes, it looks wrong, I commented as such in the bug [18:18] that last line is a bit funny too, mysqld running with fsuid 1001?? [18:18] it looks like a heavily modified install, even the initscript was changed [18:19] sounds less like a package bug and more of a "PEBKAC User Error" bug [18:19] wow [18:21] is "This system is so heavily modified from the defaults it's not supportable by us" a viable response? (Just saying) [18:22] either of you know if it's possible to get details out of an autopkgtest run? [18:22] beyond the logs that it gives normally [18:22] teward: a local run? [18:23] ahasenack: no, autopkgtest.u.c [18:23] but if i have to do a local run of this package I will be displeased [18:23] teward: there are artifacts you can download, but that's it [18:23] searx breaks with the nginx upload, but the error is uwsgi-level [18:23] ahasenack: where? [18:24] found it nevermind [18:24] grrr yeah the artifacts won't help me here... [18:25] ahasenack: if I were to do a local run of this, I have two questions: [18:25] (1) how would i do that, and [18:25] (2) is there a way to get into the environment if it fails to actually get debugging info? [18:25] teward: http://autopkgtest.ubuntu.com/ and go find your package there [18:25] on a 18.04 server install. When the ISO is booting some services take forever to start and hangs. Snap server, kernel message of scsi_eh blocked for 120 seconds [18:25] anyone seen that [18:25] (it's not loading here, btw) [18:25] swein: the first time you boot it? And it's the "live server" install? [18:25] ahasenack: yes liv eserver install and on first ISO boot [18:25] I haven't seen that, sorry [18:25] but scsi_eh blocked could be a hardware problem, everything ok with your disks? [18:25] yay net lag >.< [18:25] teward: (2) not that I know of [18:26] teward: (1) is doable, is that in cosmic? [18:26] nothing is connected to the megaraid sas controller, it's just plugged in [18:26] ahasenack: yes, though my system here is Xenial [18:26] ahasenack: TL;DR I need systemctl status -l output [18:26] teward: do you know if the test needs a vm, or if it can run in a container? What does d/t/control say? [18:26] ahasenack: E: Unknown, not my package [18:26] still digging [18:26] ahasenack: if you want to help I'd gladly welcome it [18:26] which package is it? [18:27] ahasenack: searx, its autopkgtests were triggered by NGINX uploads. [18:27] somehow [18:27] sure, sounds more exciting than a pile of 10 apache bugs from 200x [18:27] ahasenack: the nginx direct autopkgtests all succeed [18:27] so it's not an nginx-level issue [18:27] digging into the failed tests the error is in the uwsgi package when it tries to load [18:27] and because of systemd its error message is suppressed saying "Run systemctl status uwsgi for details" [18:28] it's usual, I half lost my mind chasing down triggered tests when I uploaded snmp once [18:28] got all the way down to bind9 even [18:28] heh [18:28] well I value a second set of eyes [18:28] actually oddly enough [18:28] ok, so isolation-container, looks like a lxd is fine [18:28] it looks like the test has always-failed since 2018-05-05 [18:28] I'll run the tests, but just so you know, [18:29] you would first run autopkgtest-build-lxd to prepare a lxd image for autopkgtest to use [18:29] and then autopkgtest -s -U /path/to/package (or package name) -- lxd [18:29] more or less [18:29] going from memory [18:29] -s is for it to stop when it fails and give you a shell [18:30] -U is to run apt-upgrade, since the image you prepared earlier could have been prepared weeks ago [18:30] ahasenack: indeed. Though Xenial won't work for it. [18:30] i'll need to use my bionic box [18:30] it should work on xenial [18:30] those scripts don't exist in Xenial [18:30] it probably has the old name, that's all [18:30] adt I think [18:30] there it is [18:30] don't ask [18:30] I wouldn't know why [18:30] not going to :p [18:31] heh lol [18:31] ahasenack: doesn't work with snapped LXD 3.x [18:31] apparently [18:31] the build-lxd one? [18:31] mhm [18:31] at least, not on Xenial [18:31] do you have a cosmic image already? [18:32] you need to give it the name of an image [18:32] ahasenack: ah, well, there's no Cosmic images for me to pull [18:32] it will boot it, make modifications, and give you a new one [18:32] i already tried [18:32] so [18:32] lxc image copy ubuntu-daily:cosmic local: [18:32] try that [18:32] *now* it works [18:32] when was taht turned on? [18:32] it didn't work last week o.O [18:32] yesterday I think [18:32] or fri [18:33] correct [18:33] ah that explains it [18:33] *waits for the image to copy down* [18:34] (autopkgtests annoy me a little but meh) [18:38] hmm i think my system derped, gonna reboot [18:54] ahasenack: any idea how long it takes for autopkgtests to build the LXD? Seems to be taking a while, even though I already copied down the image :/ [18:55] and as soon as I say that it completed :/ [18:55] *kicks self* [18:56] what memory stick tool you use to format for dos boot to do bios upgrade ? [19:00] last time I needed a dos boot disk I think I found a freedos image somewhere that worked [19:01] hmm,, what format tool ? :P [19:02] I just used dd to write the thing to a usb stick [19:02] if you actually have to *format* something, maybe the mtools package can help [19:02] I hope we still package it, any way ;) [19:06] teward: I reproduced the failure locally: https://pastebin.ubuntu.com/p/BZDCmwgcX4/ [19:06] ahasenack: it looks like it's a uwsgi failure [19:06] not an nginx one [19:07] is libapache2-uwsgi involved? [19:07] it shouldn't be [19:07] * ahasenack checks [19:07] ahasenack: can you pull `systemctl -l status uwsgi`? [19:07] right, not installed [19:07] teward: https://pastebin.ubuntu.com/p/D3fk24tVwJ/ [19:07] super helpful [19:08] ahasenack: no, but that points at the issue not being an NGINX related issue [19:08] and probably a package-level issue [19:08] what's odd is it only fails on three archs [19:09] ahasenack: it's interesting to note, too, that searx's autopkgtest has been failing for a while [19:09] this is the first time it's on my radar though, the last time it worked was prior to may 5th [19:11] I have no idea how these things work [19:16] teward: I get a core dump, actually [19:16] ERROR: apport (pid 10684) Tue May 22 16:16:15 2018: apport: report /var/crash/_usr_bin_uwsgi-core.0.crash already exists and unseen, doing nothing to avoid disk usage DoS [19:18] interesting. [19:18] ahasenack: maybe this needs to be opened as a critical level bug against searx [19:19] searx or uwsgi? [19:19] since the searx tests have failed for everything since may 5th [19:19] ahasenack: searx [19:19] unless uwsgi's autopkgtests also fail [19:20] though i don't see any [19:20] ahasenack: nginx wouldn't have any effect on uwsgi [19:20] well, what crashed was uwsgi [19:20] I'm checking with debug packages to see if the backtrace shows another library [19:20] ack [19:20] ahasenack: if we can rule out nginx as the cause, then I can rest easy, but I'll have to ask the release team to force ignore those tests for this run [19:21] hm, no debug packages for searx [19:21] ahasenack: are there any for uwsgi? [19:21] yes, I installed them, but the backtrace is still invisible in the critical area [19:21] let me paste [19:21] hmm [19:21] teward: https://pastebin.ubuntu.com/p/8DYQSxg93t/ [19:23] ahasenack: it sounds to me then like this is a searx bug [19:23] let me check something [19:23] i have an example uwsgi somewhere, and a container that i can test with heh [19:24] ahasenack: if this simple 'hello world' uwsgi app works, then we can blame the searx package [19:24] if it fails as well, then we can blame uwsgi [19:24] either way, nginx is not an issue [19:25] teward: still, that was never enough of an excuse to not chase dep8 failures :) [19:26] it's all part of making ubuntu better :) [19:26] indeed [19:28] ahasenack: happen to have the old commands for adt to run autopkgtests in LXD? Or should I get a Bionic environment sooner than later :P [19:28] try adt [19:28] I think the arguments did not change [19:29] dumps me this: https://paste.ubuntu.com/p/64TkTfWDC8/ [19:29] so adt isn't an app, but there's adt-sub apps so IDK [19:32] I'm almost thinking it's a permission problem [19:32] teward: right, so first adt-build-lxd [19:32] ahasenack: i did [19:32] it completed [19:32] I think it's just adt-build-lxd [19:32] then do lxc image list, see if you have an autopkgtest specific image now [19:33] adt/ubuntu/cosmic/amd64 is there, yes [19:33] cool [19:33] now we want adt-run [19:33] I use: [19:34] -U -s -o ../dep8-output --apt-pocket=proposed source-package-name -- lxd adt/ubuntu/cosmic/amd64 [19:34] try that [19:34] -U: upgrade [19:34] -s: stops on fail [19:34] -o: output report [19:34] --apt-pocket: tell it to use proposed, like ubuntu does [19:34] -- lxd: virt type [19:34] and the last is the lxd image to use [19:34] [19:34] I was typing, not copying and pasting :) [19:35] loophole! [19:35] * genii eye-rolls [19:35] haha [19:37] ahasenack: worked with adt-run -U -s -u somefilepath --apt-pocket=proposed source-pkg-name --- lxd adt/ubuntu/cosmic/amd64 [19:37] triple dash o.O [19:37] oh, right [19:37] why not [19:38] "-u somefilepath" seems wrong, -u is for user [19:38] typo [19:38] -o [19:38] unless that changed between adt and what we have now [19:38] ok [19:41] ahasenack: confirmed it's not an uwsgi segfault, a simple uwsgi app works [19:41] so the issue is searx [19:49] ahasenack: it looks like this was synced in from Debian directly [19:49] what would you suggest for filing the bug against this? [19:50] did your test finish? [19:50] the adt one [19:50] I'm trying to start it manually in a terminal, to see when it segfaults, but am having a hard time with that [19:50] actually looks like LXD exploded [19:50] hang on [19:51] it's not a simple initscript [19:51] traced it to this [19:51] May 22 16:44:30 autopkgtest uwsgi[6741]: + start-stop-daemon --start --quiet --pidfile /run/uwsgi/app/searx/pid --exec /usr/bin/uwsgi -- --ini /usr/share/uwsgi/conf/default.ini --ini /etc/uwsgi/apps-enabled/searx.ini --daemonize /var/log/uwsgi/app/searx.log [19:51] but that incurs in other errors [19:57] ahasenack: running local test now, lxd decided to fubar initially [19:58] ok [19:58] also troubleshooting some major chaos with a VPN tunnel at work so, split-attention [19:59] * ahasenack fetches a snack [20:05] ahasenack: i think we have to contact the Debian people on that [20:05] because why they do taht in the test [20:06] instead of install uwsgi, drop it in the app area, and then call the uwsgi command which runs this as a daemon, (not sure if it can?) it is failing on that command [20:06] which is unique to the test [20:09] let me poke this a bit more [20:10] ahhhhh interesting [20:19] back [20:37] ahasenack: hmm i found something interesting [20:37] i need to do some debugging thoguh [20:38] ahasenack: i think i found the breakage [20:38] sounds good [20:38] do tell [20:38] possibly. [20:39] ahasenack: the problem is I can't find the explicit init.d to call [20:39] but when I try and run uwsgi directly things're not being placed in there proper [20:39] so it might not be the proper 'test' [20:40] is there a way to see the CLI call made to execute uwsgi by systemd or start-stop-daemon? [20:40] I tried [20:40] closest I got was to insert a "set -x" in /etc/init.d/uwsgi just before the do_* call for start [20:40] and then follow in /var/log/syslog [20:41] I got to the start-stop-daemon line I pasted above, at 19:51:15 utc [20:41] but that line in itself didn't segfault [20:42] hmm [20:43] ahhh i think i know what's going on... [20:43] sounds even better [20:45] ahasenack: i manually tried the same command to start uwsgi [20:45] and found this in syslog: May 22 20:44:35 adt-virt-lxd-dxnxve systemd[1]: uwsgi.service: Failed to reset devices.list: Operation not permitted [20:45] I see that all the time in lxds [20:45] and when it tries to start apport crash handler it fails too [20:45] ahasenack: the problem is, that's the only 'error' i can see [20:45] it doesn't actually start the process [20:45] now, when I call the start-stop-daemon line directly? [20:45] it errors with missing file/folder in the python app [20:46] ahasenack: can we call the package broken and blacklist it? [20:46] i don't think there's any rdeps [20:46] and this is a 'new' package [20:46] new as of Bionic [20:46] teward: I get this in a vm: https://pastebin.ubuntu.com/p/r7D9thcbwK/ [20:46] ahasenack: i got that too [20:46] check /var/log/uwsgi/app/searx.log [20:46] what do yo usee? [20:46] I have no clue how uwsgi is supposed to work [20:47] ahasenack: neither do I, but if any of the python underneath fails to launch uwsgi itself explodes [20:47] and that could trigger a segfault with a core dump [20:47] teward: I get this: https://pastebin.ubuntu.com/p/P34n2BVckK/ [20:47] but note this is running as root, a normal user cannot create that log file [20:47] I was thinking that had something to do with it [20:48] it could [20:48] whatever happened, those vars didn't get expanded looks like [20:48] yeah it didn't [20:48] which it wouldn't from a shell [20:48] I have /run/uwsgi/app/searx/ but it's empty [20:49] well from a shell it drops you into it might not [20:49] deb-confnamespace comes from /usr/share/uwsgi/conf/default.ini [20:50] and that's not being expanded, it seems, by uwsgi properly. [20:54] aha [20:54] ahasenack: ran the uwsgi command that should start uwsgi properly [20:54] getting a permission denied on a bind [20:55] underneath gdb [20:55] teward: it seems to be failing in debian too: https://ci.debian.net/packages/s/searx/ [20:55] ahasenack: should we raise a 'critical' or equivalent bug in Debian? [20:55] because it's not fit to exist [20:55] if autopkgtests are dying [20:55] and it's going to block other things here in Ubuntu from migrating from proposed [20:55] nginx included [20:55] I think a bug is ok. debian doesn't yet gate on dep8 tests like we do [20:55] ahasenack: bug on it here? [20:56] (Ubuntu) [20:56] I'd suggest a bug over there, and you can use that bug to ask an AA to let it pass for now [20:56] a bug here too also works, having both linked [20:58] ahasenack: so, serious-level bug on Debian about autopkgtest failures which suggest the application doesn't work? [20:58] 'cause that's where i'm at now [20:59] I don't know enough to say if the app doesn't work or not, but a segfault is bad [20:59] but I didn't check debian to see if it segfaults there too [20:59] ahasenack: i'm inclined to open it here in Ubuntu and say it segfaults horribly [20:59] the fact that the test fails in the same place is suggestive, though [20:59] and the autopkgtests make it fail horrible. [20:59] ok [20:59] do you have the crash file too? [20:59] in /var/crash? [21:00] ahasenack: no but I have gdb trace when you have the application run under uwsgi directly [21:00] so see if this works for you... [21:01] how did you run it? [21:01] gdp --args "uwsgi -s 4000 --ini /usr/share/uwsgi/conf/default.ini --ini /etc/uwsgi/apps-enabled/searx.ini" [21:01] gdb* [21:01] which has uwsgi directly run the application as configured on port 4000 [21:02] that gave a permission denied in gdb [21:02] after dropping root privs to uid/gid 33 [21:02] I start that as root and it exits after failing to write to that /run directory [21:03] let me replicate again [21:04] dose anyone have a solution how to fix this problem? https://askubuntu.com/questions/1002933/vmware-dhcp-no-internet-access, same problem in here as well https://bugs.launchpad.net/ubuntu/+source/systemd/+bug/1624320 [21:05] Launchpad bug 1624320 in systemd (Ubuntu) "systemd-resolved appends 127.0.0.53 to resolv.conf alongside existing entries" [Low,Confirmed] [21:05] ahasenack: i'm getting the 'no such file or directory' error myself... except when I set the env vars manually [21:05] so... [21:05] which vars? [21:06] UWSGI_DEB_CONFNAME and UWSGI_DEB_CONFNAMESPACE [21:06] UWSGI_DEB_CONFNAME='searx', UWSGI_DEB_CONFNAMESPACE='app' [21:06] which uwsgi conf says it uses to populate the 'fake' options [21:07] that's all way over my head :) [21:07] the workers all segv [21:07] *but* if I run the same command under gdb... [21:07] oooh actually [21:08] i don't get any usable traceback this time [21:08] ahasenack: interesting to note though that i get all the segfault calls, and it constantly respawns [21:08]