=== jamesh_ is now known as jamesh
=== Foxhoundz is now known as BenderRodriguez
=== Foxhoundz is now known as BenderRodriguez
=== ledeni_ is now known as ledeni
coreycbsil2100: hey! any chance you could take a look at neutron in the bionic unapproved queue? it has some critical bug fixes.12:13
=== mhcerri is now known as mhcerri_
=== mhcerri_ is now known as mhcerri
seb128bdmurray, hey. Are those "[bionic/nautilus] Possible Regression" emails supposed to be sent daily? Also I think Trevinho responded to it, was it good enough or was there still concerns?13:16
bdmurrayseb128: Hi! The emaisl are supposed to be sent whenever a new regression is found which could be daily. One of the emails I received talked about the crashes in general as a bunch rather than each one individually. As an example I'd prefer to know that crash 69d1bc is a memory error, while crash b9dfd02 will be fixed by the next SRU, etc....14:00
seb128Trevinho, ^14:00
seb128bdmurray, k, let's see if we can get what you need ... is there a place that shows all the report ids that are of a concern atm?14:01
seb128I deleted some of the emails, I though the content was the same14:01
Trevinhoseb128: I've looked at all of them,14:01
Trevinhopersonally the only fix I'm concerned about is https://code.launchpad.net/~3v1n0/ubuntu/+source/nautilus/+git/nautilus/+ref/ubuntu/bionic-fix-file-remote-type-search-crash14:02
Trevinhoand covered by that branch14:02
seb128Trevinho, can you take those ~20 ids and do a list of "id: <reason>" to help bdmurray?14:02
bdmurrayTrevinho, seb128: or "id, id, id: <reason>" if there are some that are the same type14:02
Trevinhomost of them are like: "I don't know", actually, a part from the one mentioned above which might cause some of them which I've marked as duplicated alaready14:03
seb128Trevinho, I think we need a full summary in one email to help bdmurray to be confident if the update is fine or not14:03
bdmurray"I don't know" doesn't sound like the phasing should be increased, unless its an old crash.14:03
Trevinhoas for the most of them I'm quite sure they happen as per other upstream changes and just changed the trace compared to what we had before, but nothing really concerning14:03
seb128Trevinho, well "I don't know" doesn't give confidence it's not a regression?14:04
seb128well if you write that it's fine14:04
seb128"not a new issue, different signature but the problem was already existing"14:04
Trevinhothese unknowns, are more of them related to different trace I think, but most of them seems like memory errorrs unrelated to an actual change, but I can see if I can resume it14:05
seb128Trevinho, well, we just need a list and show that we looked at all the reports and are confident the SRU is still fine14:07
seb128seems you are confident14:07
seb128but your reply didn't convince bdmurray14:07
Trevinhoseb128: yeah, but if we want to be better, would be nice to reupload to fix to #179502814:08
seb128so please provide it in a format that he can review/understand enough to be fine with it as we are14:08
seb128Trevinho, ok, I can look at doing that ... do you think we should block the SRU until that one lands?14:08
Trevinhoas I think there might be crashes related to memory issues that might be caused by that too...14:08
Trevinhomaybe it's better.14:08
seb128sounds like we do14:08
seb128k, let me put on my list to sponsor this week14:08
seb128bdmurray, ^ we are going to do a follow-up SRU with a fix and then we let you know the status :)14:08
Trevinhothat branch is based on `XubuntuCancel` one though, so let me know if you want me to rebase it on top of ubuntu/bionic instead14:09
rbasaksil2100: I'm going to finish up SRU reviewing packagekit that I started yesterday now, if that doesn't clash with you?14:09
Trevinhoseb128: ^14:10
bdmurrayseb128: got it, let me know if you want the SRU reviewed.14:10
seb128Trevinho, yes please, let's not interlock those14:10
seb128bdmurray, will do, thx14:10
sil2100rbasak: no problem, thanks for the heads-up14:40
seb128bdmurray, oh, other topic, I mentioned it at the sprint, but it would be nice to remove 15.04/15.10 from the error tracker legend ... is there a bug tracker/place for such requests?14:48
bdmurrayseb128: https://bugs.launchpad.net/errors/14:53
seb128bdmurray, thx14:53
seb128bdmurray, https://bugs.launchpad.net/errors/+bug/179610714:56
ubottuLaunchpad bug 1796107 in Errors "Remove 15.04 and 15.10 from the graph/legend" [Undecided,New]14:56
=== lool- is now known as lool
bdmurrayseb128: got it, thanks14:59
dokooSoMoN, tdaitx: the libreoffice tests fail with OpenJDK 11. does it need just a rebuild?15:12
dokoare these related at all?15:12
oSoMoNdoko, not sure, I need to take a closer look at the error, but I'm stepping out now, can do later in the evening15:30
rbasaksil2100: did you manage to look at bug 1782031 please?16:55
ubottubug 1782031 in openscap (Ubuntu Xenial) "[SRU][xenial] Enable SCE option and systemd probe in libopenscap8" [Undecided,In progress] https://launchpad.net/bugs/178203116:55
rbasakFrom history:16:55
rbasaksil2100_: around? May I have a second SRU opinion on bug 1782031 please? Seems to me there may be a functional (surprising) change to users there if unknown/notchecked ends up going to fail.16:56
ubottubug 1782031 in openscap (Ubuntu Xenial) "[SRU][xenial] Enable SCE option and systemd probe in libopenscap8" [Undecided,In progress] https://launchpad.net/bugs/178203116:56
rbasak13:53 <ubottu> bug 1782031 in openscap (Ubuntu Xenial) "[SRU][xenial] Enable SCE option and systemd probe in libopenscap8" [Undecided,In progress] https://launchpad.net/bugs/178203116:57
tdaitxoSoMoN: btw, hsqldb1.8.0 also breaks with openjdk-11 (System::runFinalizersOnExit() was removed) and libreoffice depends on it (it is the only dependency), do you know why? hsqldb 2.4 fixes this and we do have it in the archive, but it is not clear if it is a sane replacement for LO17:41
sil2100rbasak: looking at it now17:49
ahasenackinfinity: hey, for when you are around, do you have squid's apparmor profile enabled by any chance?18:03
infinityahasenack: How would I know?18:04
ahasenackinfinity: aa-status on the box, or ps faxwZ and check if the squid process is listed as "confined"18:05
ahasenackinfinity: I got logs from jibel that show apparmor denied messages right around the crash, so I'm thinking he did enable it18:05
infinity ps faxwZ | grep squid | awk '{print $2}'18:06
ahasenackyep, enforce, sorry18:06
ahasenackso it's enabled18:06
infinityI didn't actively enable it.  Surely, it's a default thing?18:06
ahasenackI didn't think it was18:06
ahasenackI had to explicitly enable it in my test boxes/vms18:06
ahasenacksomething to investigate, but it gives a hint about this bug18:07
infinityI mean, this laptop install dates back to vivid, so I can never remember exactly what all I may have done to it, but I'm pretty sure my squid setup is just a side-effect of "apt-get install squid-deb-proxy" and editing the whitelists.18:07
ahasenacknow let's see what kind of DENIED messages you have related to squid18:07
infinitySo, maybe the AA profile isn't enabled by default *anymore*, but it was in the past, and the maintainer scripts (correctly) don't change the current setup on upgrade?18:07
ahasenacklots of things18:08
ahasenackcould have happened. I will check that, but now I want to see if I can correlate the profile with the crash18:08
infinityahasenack: http://paste.ubuntu.com/p/PM8D6WdQD8/18:08
sil2100rbasak: hmmm, indeed an interesting case this is18:09
infinityahasenack: And I woke up to apport telling me of a new crash.  Yay.18:09
ahasenack[95825.651047] audit: type=1400 audit(1537596453.254:301): apparmor="DENIED" operation="connect" info="Failed name lookup - disconnected path" error=-13 profile="/usr/sbin/squid" name="run/dbus/system_bus_socket" pid=24740 comm="squid" requested_mask="wr" denied_mask="wr" fsuid=0 ouid=018:10
ahasenackthere are two messages18:10
ahasenackone about the net_admin capability18:10
ahasenackand the above18:10
ahasenackI've seen other fixes in apparmor profiles about this disconnected path issue18:10
infinitySIGABRT is comm_openex().  That's the same one we were looking at before, right?18:10
ahasenackyes, the result of the failed assert18:10
infinityThat does smell of something an AA denial could cause.18:10
ahasenacklet me get you a diff18:11
ahasenackfor the profile18:11
ahasenackI'll also post it to the bug18:11
jdstrand_ahasenack: make sure the profile uses attach_disconnected18:12
=== jdstrand_ is now known as jdstrand
ahasenackjdstrand_: yeah, it's not using it18:12
ahasenackthat will be my diff :)18:12
ahasenackI'm also wondering about the net_admin capability18:13
ahasenackbut that has been in use since squid3 as far as I can tell18:13
ahasenackearly squid318:13
jdstrand* bind to any address for transparent proxying18:13
jdstrandfrom man capabilities18:13
infinityYeah, squid seems like a solid net_admin consumer.18:14
* jdstrand nods18:14
sil2100rbasak: let me think about it and leave a comment tomorrow, I think I have a split opinion about this SRU18:20
ahasenackinfinity: apply this to /etc/apparmor.d/usr.sbin.squid: https://pastebin.ubuntu.com/p/R6Z84ZdsfP/18:22
ahasenackthen issue sudo apparmor_parser -r -T -W /etc/apparmor.d/usr.sbin.squid18:22
ahasenackjdstrand: looks ok? ^18:22
rbasaksil2100: thanks18:23
rbasakjmbl: ^18:23
infinityahasenack: Applied.18:24
ahasenackinfinity: does dmesg show a profile_replace going on for squid and squidguard?18:25
infinityahasenack: Applied.Oct  4 12:24:10 nosferatu kernel: [1057755.099944] audit: type=1400 audit(1538677450.686:496): apparmor="STATUS" operation="profile_replace" profile="unconfined" name="/usr/sbin/squid" pid=18164 comm="apparmor_parser"18:26
infinityOct  4 12:24:10 nosferatu kernel: [1057755.109369] audit: type=1400 audit(1538677450.695:497): apparmor="STATUS" operation="profile_replace" info="same as current profile, skipping" profile="unconfined" name="/usr/sbin/squid//squidguard" pid=18164 comm="apparmor_parser"18:26
infinityErm, that.18:26
ahasenacknow we wait I guess18:26
infinityI love waiting.18:26
ahasenackyou have been getting one crash per day basically?18:27
ahasenackI wonder if logrotate triggers it18:27
ahasenackI tried here, no dice18:27
jdstrandahasenack: lgtm18:27
ahasenackjdstrand: thx18:27
infinityOct 04 00:00:14 nosferatu squid[32112]: assertion failed: comm.cc:428: "!isOpen(conn->fd)"18:29
infinity Oct 04 00:00:15 nosferatu squid[10409]: Starting Squid Cache version 4.1 for x86_64-pc-linux-gnu...18:29
infinity Oct 04 00:00:15 nosferatu squid[10409]: Service Name: squid18:29
infinityHrm, unless I logrotate at midnight now, that's not the trigger.18:29
ahasenackis the crash always around that timE?18:29
infinityahasenack: Not sure.18:30
infinityahasenack: Huh.  Yep.  Midnight every day.18:30
infinityahasenack: So maybe cron.daily moved?  Isn't it meant to be 6am or something?18:31
infinityYeah, cron.daily and, thus, logrotate, is 6:25...18:31
ahasenack25 6    * * *   root    test -x /usr/sbin/anacron || ( cd / && run-parts --report /etc/cron.daily )18:31
ahasenackmine is that18:31
infinityExcept there's also a systemd timer for logrotate now?18:32
jibelahasenack, this machine has been upgraded since utopic, so it might be something that was set then removed18:32
ahasenackdo you see logrotate messages around that time?18:32
infinityNot sure where I'd see them.18:32
infinityAhh, syslog apparently.18:33
infinityAnd yes, logrotate is at midnight here.18:33
infinityThanks, systemd new world order, for changing everything.  Love you.18:33
infinityahasenack: So yeah, I think it's fair to assume that the service restart/reload from logrotate is the trigger.18:34
ahasenackcan you run it manually with logroate -f /etc/logrotate.conf? It will rotate your logs earlier than expected, if you care about it deeply18:34
infinityahasenack: I can get sciency about that in a few minutes, sure.  Should probably unapply the apparmor patch first, confirm a few times that it's reproducible, then apply pach.18:35
infinitypatch, too.18:35
ahasenackthat's fine, whenever you can18:35
ahasenackI'll post on the bug with the patch18:36
ahasenackjibel: can you check if you have the squid apparmor profile enabled? Check with "ps faxwZ | grep squid", see if it's listed as enforced18:36
ahasenackjibel: ah, sorry, just saw your bug update18:37
infinityvorlon: My Technical Architect, is that a thing we should care about?  logrotate moved to systemd timers (apparently) and now runs at midnight instead of the previously-expected cron.daily time.18:40
infinityDidn't mean to imply you were a Fisher Price My First Architect.18:40
jmblrbasak, sorry was away from desk. ok thanks. I am happy to help answer any questions. Unfortunately, we need that functionality to ship a product. sighhhh :-)18:45
infinityahasenack: Oh, changing apparmor profiles doesn't apply to running processes, does it?18:47
* infinity suspects not.18:48
sarnoldyou need to apparmor_parser --reload /path/to/profile18:49
jdstrandinfinity: if the process is already runningg under the parser, it does with apparmor_parser -r (or --replace, not --reload though)18:51
jdstrandinfinity: if the process started outside of confinement, it needs to be restarted (see 'sudo aa-status')18:52
infinityKay, I did replace.18:52
infinitySo, this fix isn't a large enough hammer.18:52
infinityhttps://paste.ubuntu.com/p/nN5HqbS7ZB/ <-- ahasenack18:52
infinityI also tried a hard restart for kicks.18:52
infinityLooks like it's happer about net_admin, but still whiney about dbus, and still asserting.18:52
infinityhappier, too.18:52
ahasenackI didn't handle dbus in that patch18:53
ahasenackthat log is a bit different though18:53
ahasenackinfinity: let's zero in on apparmor first, though. If you disable apparmor profile for squid, does the crash disappear?18:53
infinityahasenack: Maybe! (how?  sorry, I'm apparmor stupid)18:54
ahasenackaa-complain /usr/sbin/squid I *think* is enough18:54
ahasenackit should still log, but not actually deny18:54
ahasenacknot sure if restarts are needed, just give it a try18:54
sarnoldargh. sorry about --reload. :( been dicking with systemd lately :(18:54
infinityahasenack: No dice: https://paste.ubuntu.com/p/4zm9dDKgk2/18:56
infinityI should probably fix the config file it's complaining about too, but I can't imagine that being the issue. :P18:56
ahasenackit's not, I get that too18:56
ahasenackjust no crash18:57
ahasenackinfinity: so is this enough to crash it? /usr/sbin/squid -k rotate18:57
infinityahasenack: Yup.18:58
infinityahasenack: So, I guess apparmor was a red herring (but clearing out all those sketchy DENYs still seems like a solid plan)18:58
ahasenacknow why can't I get it to crash18:59
infinityThat's a mystery I'm not sure I can solve.18:59
ahasenackinfinity: is that a host, or a container/vm? And amd64 I assume?18:59
infinityahasenack: amd64 bare metal.18:59
ahasenackinfinity: is this squid the version from the ppa, or 4.1 from cosmic?19:01
infinityahasenack: cosmic.19:01
infinityahasenack: Not against trying the PPA now that we have a consistent reproducer.19:01
ahasenackinfinity: can you try the ppa one, and then run /usr/sbin/squid -k rotate command?19:01
infinityahasenack: URL to the PPA again?19:01
infinity(or short name for apt-add...)19:01
ahasenackinfinity: add-apt-repository ppa:ci-train-ppa-service/345019:02
infinityWhy does squid take so friggin' long to shut down?19:03
infinity(longterm complaint, this isn't new)19:03
ahasenackI know19:03
ahasenackit waits 30s19:03
infinityErr, wat?19:03
infinityThere's a sleep in there, it's not DOING anything?19:03
ahasenackit's like a graceful shutdown, but it always does that, regardless if there are open connections or not19:03
infinityThat should be fixed..19:04
infinityOh, that's fun.  Upgrading squid doesn't restart squid-deb-proxy.19:04
infinityProbably also a longstanding bug, but ew.19:04
* infinity restarts manually.19:05
ahasenack /o\ surrounded by bugs19:05
infinityI mean, squid can't be expected to know about *all* its potential rdeps, but the ones it does know about (cf: the apparmor profile knows of some), it should probably try to detect and restart.19:06
ahasenackthere are ways to link systemd units, maybe it could be done19:06
infinityOr that.19:06
infinityOct  4 13:06:43 nosferatu squid[17502]: assertion failed: comm.cc:428: "!isOpen(conn->fd)"19:06
infinity(with the new squid)19:06
infinitySo, no dice.19:06
ahasenackthanks for your help19:07
infinityI wonder if it's just a stupid assertion that needs to not? :P19:07
infinityLike, exiting a loop there might be just as sane as DYING HORRIBLY.19:07
infinity(note: I've not looked at the code at all, maybe it's the 1 in 100 times when assert() is used correctly)19:08
infinityahasenack: The other possibility is that this is really expected behaviour, and upstream just didn't think about people like us who trap all unclean exits as errors and whine about them.19:12
infinityahasenack: Note that this is the child process that's dying and respawning, not the master, AFAICT.19:12
infinityahasenack: So maybe that assert just needs to be an exit, and we can wash our hands of it.19:13
infinityahasenack: But an understanding of WTF is going on would be helpful to determine that.19:13
infinityahasenack: I make that assertion based on my master processes running since I started them, and the children having new start times after rotate.19:14
infinityAlso, this seems to not have anything to do with squid-deb-proxy, it's the non-deb version that we're testing and killing with squid -k rotate, it looks like.19:15
ahasenackfound an old bug, looking at it: https://bugs.squid-cache.org/show_bug.cgi?id=479619:15
ahasenackmany comments19:15
sarnoldew. I have to admit I would have expected socket interaction with logfile rotating to have been sorted out twenty years ago.19:17
infinitySo many wheels to reinvent.19:19
infinityahasenack: Seems stalled on another round of review/commit, but the last patch looks plausibly not the worst?19:19
infinityOh, and the discussion moved to github... Somewhere.19:20
ahasenackyeah, tracking19:21
ahasenackinfinity: about the long shutdown: https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=89846919:26
ubottuDebian bug 898469 in squid "Squid waits on shutdown even though there are no active clients" [Normal,Open]19:26
ahasenackbeen there since 3.5.x19:27
infinityAlso, misreading "FreeBSD" as "Fedora" and then seeing libc.so.7 in a backtrace was mildly terrifying for a split second.19:27
ahasenacksaw someone with a plan B of using restart for logrotate instead of "squid -k rotate"19:27
infinityahasenack: Which then runs into the long shutdown issue. :)19:28
infinityahasenack: I wonder if it would be out of line to suggest that the Debian/Ubuntu packages should drop the default for that to 5s or something.19:29
infinitySince it can be jacked back up by the config file for people who actually want that.19:29
infinityI mean, even 5s is too long.  The real bug should be fixed upstream, but whee.19:30
infinity(The part where it doesn't differentiate between active clients and *any open socket*, and thus always waits the max time)19:30
infinityI assume this actually, hilariously, means that the log sockets we're currently asserting on are also responsible for the shutdown taking 30s.19:31
infinityNot that fixing A will fix B.19:31
infinityJust related code with two stupid bugs that should probably date and make hundreds of little bugs.19:31
ahasenackdebian should also be affected by the crash19:32
infinityOne would assume, yes.19:32
* ahasenack reads through the bug one more time19:32
infinityBut they don't have a crash handler installed by default, so less likely to notice.19:32
infinityAs I pointed out, it's the *child* that dies and respawns, so there's not DoS or anything here.19:32
infinitySo without a crash handler, you'd have to be scouring logs to even notice it happened.19:33
* infinity grabs his crossbow and goes to hunt wild tacos.19:33
sarnoldmm tacos19:34
ahasenackgot it in debian20:17
ahasenackOct  4 20:17:00 sid-squid4 squid[582]: assertion failed: comm.cc:428: "!isOpen(conn->fd)"20:17
ahasenacknot sure how yet, I ran -k rotate multiple times, sometimes with an open connection20:17
ahasenackbut getting there20:17
ahasenackinfinity: got a reproducer, and a one-line config change that explains why squid-deb-proxy doesn't crash by default, but squid does20:52
ahasenacksquid-deb-proxy has cache_dir specified, as does my home proxy. With that, it doesn't crash on squid -k rotate20:52
ahasenackI'll update the upstream bug, and might file a debian one as well now that I have a simple reproducer20:52
infinityahasenack: Shiny.  Well-sleuthed.21:07
infinityahasenack: So, the TLDR for me is that if I mask out the main service (as I wanted to do anyway), the bug goes away for me? :P21:08
infinityahasenack: (not at all a valid excuse to not fix it, obviously)21:08
ahasenackinfinity: yes21:08
ahasenackor add cache_dir to squid.conf, for reasons21:08
infinityShould it not have a baked-in default for that anyway, pointing to a dpkg-owned directory?21:09
ahasenackif you want to confirm it21:09
ahasenacksquid-deb-proxy does the right thing21:09
infinitySeems like a packaging bug tickling the upstream bug.21:09
ahasenacksquid.conf, I don't know why there is no default, probably because you have to specify a size21:09
ahasenackthe cache is only in ram then, without cache_dir, afaik21:09
infinityA default config with no cache dir seems vaguely useless.21:09
ahasenacklet me see if there is an option where you don't have to specify the max size21:10
infinityBut maybe I'm not imagning some weird use of squid where you would be happy with a tiny RAM cache.21:10
ahasenackwell, it's a proxy and a cache21:11
ahasenacktwo things in one21:11
ahasenackthe proxy bit can be used for access control21:11
infinityThere are other proxies that are better at that if you don't also want the caching, IMO.21:12
ahasenackall cache_dir types require a size parameter, something hard to guess a good default for21:12
infinityBut fair point.21:12
ahasenack# No disk cache. Store cache ojects only in memory.21:12
infinityI still think preconfiguring a cache dir (even if tiny) makes sense.  But, again, still not an excuse for not fixing the upstream bug.21:12
ahasenackafter all, squid-deb-proxy does have a default cache dir21:12
infinityI'd rather waste some arbitrary value (500M?) of everyone's disk than eat their RAM.21:13
infinityIf I had to pick a "sane" unpacked-but-not-user-configured state.21:13
ahasenack# use a different dir than stock squid and default to 40G21:13
ahasenackcache_dir aufs /var/cache/squid-deb-proxy 40000 16 25621:13
ahasenackthat's from squid-deb-proxy21:13
ahasenacksomeone made the call21:13
naccinfinity: yeah, esp. as a default configuration setting these days21:13
vorloninfinity: ah, poor merger I that I didn't notice the change to timers also changed the time it ran.  I think that yes, we ought to move it to again run in the 6am window.23:38
jbichavorlon: if you're around, there are some proposed hints: https://code.launchpad.net/%7Eubuntu-release/britney/hints-ubuntu/+activereviews23:51

Generated by irclog2html.py 2.7 by Marius Gedminas - find it at mg.pov.lt!