[00:14] <RikMills> someone reported the epiphany crash https://bugs.launchpad.net/ubuntu/+source/epiphany-browser/+bug/2046844
[00:14] -ubottu:#lubuntu-devel- Launchpad bug 2046844 in epiphany-browser (Ubuntu) "Epiphany browser does not launch on Ubuntu 24.04: core dumped" [Undecided, New]
[00:14] <RikMills> but time for sleep here
[00:14] <arraybolt3> ok so I'm getting a SIGILL even with host CPU passthrough, so it's not a problem with CPU features.
[00:46] <arraybolt3> did some Wayland experiments with labwc, which more-or-less failed miserably. I think Wayfire is likely going to be better.
[00:46] <arraybolt3> perhaps I'm just doing it wrong though, or maybe stefonarch's scripts are out of date
[01:30] <arraybolt3> nothing like discovering that your keyboard is stuck in German when trying to do a Wayland experiment XD
[01:42] <tsimonq2> Miriway?
[01:45] <arraybolt3> lxqt-panel requires window rules to work.
[01:45] <arraybolt3> but it can work
[01:46] <arraybolt3> I made some progress but for some reason the xdg-Lubuntu-Wayland folder isn't getting picked up
 According to stefonarch it doesn't work in wayland, that's why he uses another bar
[01:47] <arraybolt3> oh really? On his repo he commends about lxqt-panel, "Window rules are needed until Qt 6.5 is fully implemented." And I found https://labwc.github.io/integration.html#:~:text=In%20order%20to%20integrate%20componenents,always%20support%20per%2Doutput%20configuration.
[01:47] <arraybolt3> (excuse the Chrome-mangled link)
 Ouch
 I see
[01:52] <arraybolt3> hey, making progress!
[01:52] <arraybolt3> I finally got theming right
[01:52] <arraybolt3> https://i.imgur.com/hRqjsQz.png
[01:52] <arraybolt3> bar is horribly mangled but that's on the to-do list :P
[01:54] <arraybolt3> The main issue is that these scripts are horribly horribly horribly *focused* - they're made as if they're a proof-of-concept to get one dev's machine working and that's it.
[01:54] <arraybolt3> I guess they kind of are...
[01:55] <arraybolt3> but they need a lot of work to become Lubuntu-ready, which is what I'm trying to do now. I may just end up with another dev homebrew hacky mess, but we can hope not...
[01:57] <arraybolt3> One issue here is that it doesn't look like this stuff has very good support for XDG at all yet, either that or else I'm doing something wrong
[01:59] <arraybolt3> hmm, I may just be doing it wrong
[02:02] <tsimonq2> arraybolt3: Remember, XDG stuff comes from SDDM, which is a layer up :)
[02:02] <arraybolt3> hmm
[02:02] <arraybolt3> really.
[02:02] <tsimonq2> The .desktop file must exactly match Lubuntu-Wayland.desktop in this case.
[02:02] <arraybolt3> Well it does...
[02:02] <tsimonq2> yeah lemme grab the docs
[02:02] <arraybolt3> ...and it's not getting added to $XDG_CONFIG_DIRS
[02:03] <arraybolt3> anyway most of this is relatively low-priority compared to the fact that lxqt-session just flat out fails silently on almost everything.
[02:03] <arraybolt3> I have to launch every single component manually in a startup script which is waaaay less than ideal.
[02:03] <tsimonq2> Here's a step-by-step on what pulls from what:
[02:03] <tsimonq2> SDDM pulls the session name from /usr/share/xsessions when logging in. In our case, it's Lubuntu.desktop. Therefore, the DESKTOP_SESSION environment variable is set to Lubuntu, from this filename. Please note that we had to patch SDDM to do this properly; it used to make DESKTOP_SESSION an absolute path, which, when we continue in these steps, breaks things. TODO: send this upstream.
[02:03] <tsimonq2> xorg sets the XDG_CONFIG_DIRS environment variable to /etc/xdg/xdg-DESKTOP_SESSION and /etc/xdg and XDG_DATA_DIRS to /usr/share/DESKTOP-SESSION and usr/share at minimum. NOTE: this is an Ubuntu-specific patch in xorg.
[02:03] <tsimonq2> [startlxqt](https://github.com/lxqt/lxqt-session/blob/master/startlxqt.in) then grabs the values of XDG_CONFIG_DIRS and XDG_DATA_DIRS to use in the call to startlxqt, which then does a one-time copy of the settings from the first entry in both.
[02:04] <tsimonq2> For a while before we could figure it out, the Lubuntu Next 18.04 image had a black screen on the live CD and on bootup of a new Lubuntu Next system. The LXQt system had been logged in, but nothing was visible. This is why we had to patch SDDM, because the value of XDG_CONFIG_DIRS didn't have our valid XDG path; it was an absolute path put onto /etc/xdg/xdg-. So therefore, if the situation ever 
[02:04] <tsimonq2> comes up again, it means that XDG_CONFIG_DIRS can't find valid settings. This also caused Simon to confront upstream; they had moved the default settings to /usr/share from /etc/xdg because they believed that it would apparently make it easier for distributors. Not only did they not consider that /usr/share is not in the XDG spec, they did no verification in LXQt itself to ensure that /usr/share 
[02:04] <tsimonq2> at minimum is always there. When Simon submitted a patch, it was rejected on the grounds that it was too distro-specific, which was ironic because they didn't follow the XDG spec. Alf has since reconsidered and asked Simon to resubmit the patch.
[02:04] <tsimonq2> If you rename /usr/share/xsessions/Lubuntu.desktop to something else, please do change /etc/xdg/xdg-Lubuntu/ to the same name. Otherwise none of our settings will be applied.
[02:04] <arraybolt3> ah, xorg sets XDG_CONFIG_DIRS.
[02:04] <arraybolt3> Bingo.
[02:04] <arraybolt3> labwc doesn't :D
[02:05] <arraybolt3> so therefore startlxqtlabwc will have to
[02:05] <arraybolt3> which is very easy
[02:05] <tsimonq2> I wrote all of those notes *five* *years* ago. I knew we'd need them someday.
[02:05] <arraybolt3> nicely done
[02:05] <tsimonq2> And hey, if you wanna add anything to these notes, feel free :D
[02:05] <tsimonq2> thanks
[02:06] <arraybolt3> well at least it unravels the current mystery.
[02:06] <arraybolt3> Right now if I can just get lxqt-session to work, I'll be way closer to success.
[02:06] <tsimonq2> I remember spending like a full-time week banging my head against the wall on this, so I vowed that nobody shall have to deal with this pain again. :P
[02:07] <tsimonq2> (This was the one time where I had to touch X11 itself, SDDM, casper, ubuntu-cdimage, debian-cd, etc. all at once)
[02:07] <arraybolt3> grief
[02:07] <arraybolt3> sounds like "fun"
[02:07] <tsimonq2> When I say it gets worse, I mean it. :P
[02:08] <tsimonq2> (As in, things that may go wrong.)
[02:10] <tsimonq2> Anyway, let me know how far you get. I also appreciate the energy (and the efforts, this is cool!) but we likely have bigger fish to fry this time of cycle.
[02:11] <tsimonq2> I think the idea would be that we focus hard on this and bugfixing post-FF.
[02:11] <tsimonq2> That being said, please, continue your work and document your findings, so we at least have a starting point. Just don't sink days into it please :)
[02:12] <tsimonq2> Just trying to look at things from a bigger point of view, not to de-motivate by the way: that's my job as Release Manager, I think? :)
[02:15] <arraybolt3> I'm mainly working on this because it's something to do after some stress from earlier today (in multiple areas).
[02:15] <tsimonq2> Not a problem at all, thank you for what you do arraybolt3 :)
[02:16] <tsimonq2> arraybolt3: That's partly why I asked where the critical bug was, because I know we've both made a day of it... XD
[02:18] <arraybolt3> right?
[02:36] <tsimonq2> arraybolt3: I'm hitting the hay, I'll loop back bright and early for ya.
[02:36] <tsimonq2> Did LoB ever sponsor qtilities btw?
[02:36]  * tsimonq2 checks NEW heh
[02:36] <arraybolt3> I think he did.
[02:38] <tsimonq2> Ah, yep I see it :)
[02:38] <tsimonq2> Perfect!
[02:38] <tsimonq2> Anyway, o/ :)
[15:08] <arraybolt3> OK, vote on it. Do I dare attempt to build qtwebengine? >:-)
 Like a NCR?
[15:14] <arraybolt3> Not quite, I'm going to try dropping to FORTIFY_SOURCE=2.
[15:14] <arraybolt3> I suspect that's what's going wrong... maybe...
 Oh fun. It is only time right :P
[15:16] <arraybolt3> heh
[15:16] <arraybolt3> maybe epiphany-browser won't take so long to build, idk
[15:16] <arraybolt3> webkit is actually way faster. Not fast, but fast*er* at least.
 Valid point.
[15:19] <arraybolt3> Since they're both hitting the same SIGTRAP, one assumes it will work in both places, though granted Epiphany seems to *almost* not die if you try to `continue` it past the breakpoint, whereas Falkon dies horribly with an illegal instruction signal.
[15:19] <arraybolt3> so they might not be closely related enough
[15:20] <arraybolt3> at any rate both are a problem so I'll try and do Epiphany first.
 Your theory is sound to me.
 "If _FORTIFY_SOURCE is set to 1, with compiler optimization level 1 (gcc -O1) and above, checks that shouldn't change the behavior of conforming programs are performed."
 "With _FORTIFY_SOURCE set to 2, some more checking is added, but some conforming programs might fail."
[15:24] <arraybolt3> This could also be the fault of the new GCC come to think of it.
 I was diverting down that rabbit hole too.
[15:24] <arraybolt3> The only way I can even think that SIGILL could be hit on modern hardware is if a jump was made into the middle of an instruction, thereby shifting things all around.
[15:25] <arraybolt3> At least with Epiphany there appears to be a real breakpoint set (which is getting built for... some reason?), but with Falkon, I wonder if the SIGTRAP is because the processor hit a misaligned instruction that just so happened to be a software breakpoint.
[15:25]  * arraybolt3 does not understand assembly in general all that well and doesn't understand x86_64 instructions at all, so I might be misguided here
[15:26] <arraybolt3> I watched Ben Eater's "Hello World on a 6502" series and learned some 6502 assembly from that, that's about it :P
 I haven't touched assembly since college. That was... a while ago.
[15:27] <arraybolt3> https://fedoraproject.org/wiki/Changes/Add_FORTIFY_SOURCE%3D3_to_distribution_build_flags looks like Fedora's been using FORTIFY_SOURCE=3 all year without ill effects...
[15:28] <arraybolt3> I suppose I can check their spec file for epiphany and see what they're doing
 Yeah, I think they are O3 for everything too
 by default
[15:28] <arraybolt3> yuck
 I know Suse moved in that direction as well
[15:30] <arraybolt3> Fedora's Epiphany package seems surprisingly straightforward...
[15:30] <arraybolt3> ...so that leads me to suspect the compiler more than the FORTIFY_SOURCE flag
[15:31] <arraybolt3> though both changes were made at once so technically anything is possible
[15:33] <arraybolt3> well anyway, I'm going to start by dropping back to FORTIFY_SOURCE=2 since that's easier. If that fails, then I get to rebuild GCC
[15:33] <arraybolt3> which could be fun if the current compiler is b0rked
 Um yeah.
[15:35] <arraybolt3> interestingly Epiphany in Ubuntu has -O1 set
[15:35] <arraybolt3> in DEB_LDFLAGS_MAINT_APPEND
[15:48] <arraybolt3> alright, build started
[15:52] <arraybolt3> umm, Epiphany must depend on a webkit package because that was *way* too fast
[15:53] <arraybolt3> not sure why I didn't think of that
[16:06] <arraybolt3> Build is going. Of course it's only using like one core, but meh, good enough.
[16:06] <arraybolt3> maybe two cores, I hope so
[17:56] <arraybolt3> looks like my FORTIFY_SOURCE hack is working, though hilariously on at least one file, it gets set to 2, then redefined to 3, then back to 2 again :P
[18:00] <tsimonq2> -O1
[18:00] <tsimonq2> (Not a typo or a fat finger.)
[18:03] <tsimonq2> arraybolt3: Can we reproduce this on Debian Sid?
[18:03] <tsimonq2> If we can, it's most certainly something DeltaOne would like to know about.
[18:03] <arraybolt3> the weird failures?
[18:03] <tsimonq2> yeah
[18:03] <arraybolt3> Dunno yet, could you try it?
[18:03] <arraybolt3> My machine is fully engaged in building WebKit.
[18:04]  * tsimonq2 grabs some kind of Debian ISO...
[18:04] <arraybolt3> tsimonq2: meh, don't bother, use this:
[18:04] <arraybolt3> https://github.com/ArrayBolt3/debflex
[18:04] <arraybolt3> tsimonq2: ^ generates Debian VMs for you with a single command
[18:05] <arraybolt3> they're very minimal, you need to specify the list of packages you want in fairly gory detail (like you can accidentally leave out sudo if you're not careful), but it works well and I'm using it all the time.
[18:05] <tsimonq2> niceeeee
[18:05] <arraybolt3> (debflex is the eventual name of the project, right now the actual script is gen-debvm, but I hope to develop it into something more fully fledged.)
[18:05] <arraybolt3> oh btw it can generate Ubuntu VMs too :D
[18:06] <arraybolt3> just be careful with that if you use qemu-nbd, it assumes it can hork /dev/nbd0 for itself
 *sips on coffee as he watches his computer spin up 20 LXD-powered minimal system containers for build envs*
[18:07] <arraybolt3> @teward001: LXD is a bloated overrated mess, don't use it
[18:07]  * arraybolt3 ducks
 👀
[18:08]  * genii twitches
[18:11] <tsimonq2> arraybolt3: I don't wanna hear anything about "bloated" coming from the guy who literally runs bare qcow2 files :D
[18:13] <arraybolt3> lol, if it's any consolation I mostly use bare qcow2 files because of frustration with libvirt and VirtualBox.
[18:14] <arraybolt3> (every so often libvirt or a connection thereof randomly dies on me and needs a full host reboot to fix, VirtualCrud is a glitchy, crashy mess even on Linux, and while LXD is nice, it's just so horribly complicated I can't hardly make heads or tails of the command lines I'm reading unless they're like the absolute basics.)
[18:15] <arraybolt3> (QEMU is powerful and has much that I don't understand, but everything in it is relatively easy to piece together so I'm able to comfortably use and memorize the features I need often, and learn the ones I need seldom. And it seems to be more stable for me.)
[18:28] <tsimonq2> arraybolt3: You do realize the difference between LXD and QEMU is that LXD shares the system kernel and QEMU doesn't right?
[18:29] <tsimonq2> I'd argue that's WAY more lightweight heh
[18:29] <tsimonq2> And the shed for the bike should be blue...
[18:29] <tsimonq2> :P
[18:29] <arraybolt3> that's a valid point, but still, it's MENTAL BLOAT because it makes me remember too much!!!!!!1111!!!11
[18:29]  * tsimonq2 sends arraybolt3 some flash cards XD
[18:32] <arraybolt3> in all seriousness I just know my toolkit and like it already. I'll use LXD when the time comes that I really need it and can learn how to use it properly.
[18:33]  * arraybolt3 still remembers agonizing over the QEMU for Windows documentation back when I was still on Win8 and thought QEMU was a good alternative to VBox on Windows...
[18:41] <tsimonq2> omg I forgot how much customization we do in Lubuntu >_<
[18:41] <tsimonq2> Ctrl + Alt + T doesn't even work out of the box.... 
[18:42] <tsimonq2> starts just fine here?
[18:43] <tsimonq2> That being said, don't blindly take my word for it. If I can't reproduce it on a Noble VM, it's probably something deeper.
 I've been able to replicate it on different (L)Ubuntu machines.
[18:46] <arraybolt3> sigh, your point about LXD taking less resources is wearing on me...
 Don't let it
 I use both
[18:46] <arraybolt3> I'm going to have to recreate my whole build environment AGAIN just to get faster speeds XD
[18:47] <arraybolt3> (Currently all of my sbuilds are done inside a VM)
[18:47] <arraybolt3> (which runs headless and I SSH into)
 Don't chase waterfalls
[18:47] <arraybolt3> haha
[18:47] <arraybolt3> valid point
[18:47] <tsimonq2> aaaaHA I got it
[18:47] <tsimonq2> kay time to dig deep
 If you have something that makes you efficient carry on
[18:47] <arraybolt3> tsimonq2: got it to crash?
[18:48]  * arraybolt3 is on file 6837 of 6863 that needs compiled for WebKit
 >.< Almost thhere
[18:49] <tsimonq2> arraybolt3: yes
[18:50] <arraybolt3> oh no that was just one component of WebKit that finished
[18:50] <arraybolt3> sigh
[18:50]  * arraybolt3 needs to figure out why it's only using two of eight processors to do its job... eventually
[18:52] <tsimonq2> arraybolt3: WebKit or WebEngine?
[18:52] <arraybolt3> WenKit
[18:52] <arraybolt3> Epiphany browser
[18:52] <tsimonq2> ahhhhh okay
[18:53] <arraybolt3> supposedly was going to take 30 minutes or so to build, now hours later...
[18:53] <arraybolt3> but hey, better than a whole day
[19:04] <tsimonq2> https://paste.ubuntu.com/p/h3SPJMFRpP/
[19:09] <tsimonq2> ==5231==    by 0x69799EE: QStandardPaths::writableLocation(QStandardPaths::StandardLocation) (in /usr/lib/x86_64-linux-gnu/libQt5Core.so.5.15.10)
[19:09] <tsimonq2> Makes me think it's a Qt bug.
[19:09] <tsimonq2> We should eliminate PyQt5 as a variable in the process
[19:12] <tsimonq2> Very similar errors: https://paste.ubuntu.com/p/HGQmkRpbBM/
[19:12] <tsimonq2> These two lines are exactly the same:
[19:12] <tsimonq2> [5405:5405:1219/191120.152880:FATAL:credentials.cc(125)] Check failed: . : Permission denied (13)
[19:12] <tsimonq2> ==5405==    by 0x62E19D4: QWebEngineProfile::defaultProfile() (in /usr/lib/x86_64-linux-gnu/libQt5WebEngineWidgets.so.5.15.15)
[19:12] <tsimonq2> (Meaning, between the two pastebins.)
 i should just give you the five minute tutorial for LXD containers lol.  Its *really* not that complicated for basic use xD   *sips coffee, realizes he ran out* (re @lubuntu_bot: (irc) <arraybolt3> (every so often libvirt or a connection thereof randomly dies on me and needs a full host reboot to fix, VirtualCrud is a glitchy, crashy mess even on Linux, and while LXD is nice, it's just so horribly complicated I can't hardly make
 genii go fetch coffee.
[19:23]  * genii snarls ferally and lopes off to the coffeepot
[19:24] <tsimonq2> XD
[19:26] <arraybolt3> interestingly GNOME Web seems to be *working* once inside Valgrind over here...
[19:27] <arraybolt3> like I got a window with a new tab page and am now loading Google.com in it
[19:27] <arraybolt3> so that's beyond weird
[19:27] <arraybolt3> yep, Google seems to be loading right
[19:28] <tsimonq2> This incredibly basic script also fails: https://paste.ubuntu.com/p/C7VTnBqkdY/
[19:28] <arraybolt3> nice
[19:35] <arraybolt3> Wow. So if you ever need reminded how painfully slow GTK is, keep in mind that *all of KDE* can be run under Valgrind for testing, yet GNOME Web is just about brought to its knees by it.
[19:36] <tsimonq2> hahahahahahaha
[19:38] <arraybolt3> So yes indeed, GNOME Web is *almost* usable under Valgrind, and crashes with SIGTRAP when run directly.
[19:38] <arraybolt3> Whether that's because it's just so painfully slow I interrupted it too soon is TBD
[19:39] <arraybolt3> oh and also webkit is still building
[19:41] <arraybolt3> interesting, bwrap is involved when running GNOME Web without Valgrind
[19:41] <arraybolt3> but it doesn't get involved when running with Valgrind
[19:42] <arraybolt3> so I think this thing's faking me out
[19:42] <tsimonq2> arraybolt3: I found the issue.
[19:43] <tsimonq2> At least for QtWebEngine-based browsers.
[19:43] <arraybolt3> what is it?
[19:43] <tsimonq2> Wanna confirm that running export QTWEBENGINE_DISABLE_SANDBOX=1 works?
[19:43] <arraybolt3> sure
[19:43] <tsimonq2> Could potentially not work for WebKit, although that could be a different error entirely.
[19:43] <arraybolt3> Works over here, Falkon now loads.
[19:44] <arraybolt3> nice find
[19:44] <tsimonq2> \o/
[19:44] <arraybolt3> I bet that's why GNOME Web works under Valgrind, it's probably disabling sandboxing when it detects Valgrind.
[19:44] <tsimonq2> So, here's the deal on that.
[19:44] <tsimonq2> We're using an old QtWebEngine; we're a version behind.
[19:45] <tsimonq2> mitya57 is sick and lisandro is without power until the weekend, so, well, I become the de-facto Qt 5 maintainer in Ubuntu and Debian, just going off of co-maintainership. :P
[19:46] <tsimonq2> I'm willing to bet that updating to the latest QtWebEngine will do something. This is too big of an issue for them not to notice.
[19:46] <tsimonq2> So, the internal Chromium right now is 108.0.5359.124, the new one would be 119.0.6045.123 :P
[19:47] <arraybolt3> tsimonq2: well hold on now, I have an idea. If this is sandboxing related, I don't think it's going to help to upgrade QtWebEngine. It might, but I doubt it.
[19:47] <arraybolt3> Because there's a sandbox in both places, and disabling it seems to fix it in both places
[19:47] <arraybolt3> (both QtWebEngine and WebKit)
[19:47] <arraybolt3> If that's the case, we *may* be up against a kernel bug.
[19:48] <tsimonq2> NOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOo
[19:48] <arraybolt3> So are there any other sandboxed apps you can try to confirm?
[19:48]  * arraybolt3 installs Flatpak and tries some hackery
[19:48] <tsimonq2> WebKit doesn't have Chromium does it
[19:49] <arraybolt3> nope
[19:49] <arraybolt3> WebKit is a fully independent browser engine (well, almost fully independent, Blink/Chromium is based off of it or was long ago)
[19:49] <tsimonq2> hah so https://qt-kde-team.pages.debian.net/images/qt5_build_deps.png webview depends on webengine which depends on webchannel :P
[19:49] <tsimonq2> but both webkit and webengine depend on webchannel
[19:49] <tsimonq2> it might be in webchannel...
[19:50] <arraybolt3> mmm, doubt it from the GDB backtraces we got yesterday
[19:50] <arraybolt3> the failure was in a destructor within QtWebEngine itself for Falkon, and in something in glib called from WebKit in Epiphany
[19:50] <arraybolt3> IIRC anyway, the screenshots are up above
[19:51] <arraybolt3> I'm installing Firefox as a Flatpak currently since bwrap looked like it might have had something to do with the issues and I know Flatpak uses bwrap under the hood
[19:51]  * tsimonq2 sends a qtwebengine to experimental
[19:52] <arraybolt3> btw aren't Flat
[19:52] <arraybolt3> pak's inaccurate progress meters awesome? https://i.imgur.com/9m1MELl.png
[19:52] <arraybolt3> Like 146.5 MB vs 156.8 MB, ok I get it, but 17.8 KB / 333.4 MB O_O
[19:53] <arraybolt3> sigh, Flatpak'd Firefox works
[19:56] <arraybolt3> Confirmed, disabling the sandbox on Epiphany works even without Valgrind.
[19:57] <arraybolt3> had to use WEBKIT_DISABLE_SANDBOX_THIS_IS_DANGEROUS=1 to make it work 🤪
[19:58] <arraybolt3> interestingly a different WebKit-based browser, "surf", works just fine with sandboxing.
[20:01] <tsimonq2> I have no idea if this is related:
[20:01] <tsimonq2> 14:02 < ahasenack> beware, https://pastebin.ubuntu.com/p/98hftZCHBd/ crashes the kernel
[20:01] <tsimonq2> 14:02 < ahasenack> https://gitlab.com/apparmor/apparmor/-/issues/346
[20:01] -ubottu:#lubuntu-devel- Issue 346 in apparmor/apparmor "kernel null pointer dereference loading an invalid AppArmor profile, regression since 6.1" [Opened]
[20:01] <tsimonq2> 14:02 -ubottu:#ubuntu-security- Issue 346 in apparmor/apparmor "kernel null pointer dereference loading an invalid AppArmor profile, regression since 6.1" [Opened]
[20:01] <tsimonq2> jinx :P
[20:02] <arraybolt3> oh that might be similar
[20:03] <tsimonq2> Lemme try an install with the kernel in proposed real quick...
[20:04] <arraybolt3> I don't think it's related - Falkon launches just fine on 22.04 with kernel 6.2.0.
[20:04] <tsimonq2> BTW, I have about 10 minutes left for right now. My car is in the shop so I'll need to go get that picked up.
[20:04] <tsimonq2> hmm
[20:04] <arraybolt3> source - I just ran it
[20:05] <arraybolt3> I tried strace'ing both Falkon and Epiphany and didn't see a syscall in common that triggers the weirdnes
[20:05] <arraybolt3> *weirdness
[20:05] <arraybolt3> which I guess makes sense but...
 I was wondering about apparmor
[20:08] <tsimonq2> 2023-12-19T20:07:38.091817+00:00 lubuntu kernel: [  170.840872] audit: type=1400 audit(1703016458.083:45): apparmor="DENIED" operation="userns_create" class="namespace" info="User namespace creation restricted" error=-13 profile="unconfined" pid=2409 comm="qutebrowser" requested="userns_create" denied="userns_create"
[20:08] <tsimonq2> 2023-12-19T20:07:38.091818+00:00 lubuntu kernel: [  170.840934] traps: qutebrowser[2409] trap int3 ip:7f05a3c4cb13 sp:7ffed669c4d0 error:0 in libQt5WebEngineCore.so.5.15.15[7f05a1a29000+6931000]
[20:08] <tsimonq2> Yuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuup. It's apparmor
 Boom
[20:09] <tsimonq2> https://launchpad.net/ubuntu/+source/apparmor/4.0.0~alpha2-0ubuntu7
[20:09] <tsimonq2>   [Alex Murray]
[20:09] <tsimonq2>   * Enable user namespace restrictions by default (LP: #2046477)
[20:09] <tsimonq2>     - d/p/u/userns-runtime-disable.patch: add logic to disable user
[20:09] <tsimonq2>       namespace restrictions if kernel lacks support
[20:09] <tsimonq2>     - debian/usr/lib/sysctl.d/10-apparmor.conf: set sysctl value to 1 and
[20:09] -ubottu:#lubuntu-devel- Launchpad bug 2046477 in apparmor (Ubuntu) "Enable unprivileged user namespace restrictions by default" [Undecided, Fix Released] https://launchpad.net/bugs/2046477
[20:09] <tsimonq2>       update comment to match
[20:09] <tsimonq2>     - debian/apparmor.service: run After systemd-sysctl.service
[20:09] <arraybolt3> ahahahahaha
[20:10] <tsimonq2> https://discourse.ubuntu.com/t/spec-unprivileged-user-namespace-restrictions-via-apparmor-in-ubuntu-23-10/37626
[20:10] <arraybolt3> so it IS a kernel bug
[20:10] <tsimonq2> arraybolt3: apparmor =/ kernel tho :)
[20:10] <tsimonq2> A specific upload did this.
[20:10] <arraybolt3> so did this get implemented for 23.10?
[20:10] <arraybolt3> looks like not
[20:10] <tsimonq2> nah the upload says noble
[20:11] <tsimonq2> release on 2023-12-15
[20:11] <tsimonq2> So, it's been like this for about 4 days.
[20:11] <arraybolt3> but you're right, the trap always happens right after an audit failure for both Epiphany and Falkon.
[20:11] <arraybolt3> So I guess the appropriate solution is to make AppArmor profiles for those particular apps?
[20:12] <arraybolt3> anyway, I can file the bug
 That tracks. I noticed it over the weekend (re @lubuntu_bot: (irc) <tsimonq2> So, it's been like this for about 4 days.)
[20:12] <arraybolt3> unless kc2bez wants to, tsimonq2 I think you said you're busy
[20:12] <arraybolt3> (I ignored the apparmor thing thinking "yeah yeah you always are griping about things, AppArmor" :P)
 I think there was a bug you can pile on
[20:12] <arraybolt3> ah right, I remember that
[20:12]  * tsimonq2 reaches out to my security team contact
[20:17] <tsimonq2> See -members; tl;dr we need to start writing AppArmor profiles.
 If my brain is working correctly, there needs to be more profiles for all the browsers now. That seems tedious.
[20:18] <tsimonq2> Yeah Dan, I just confirmed as such
[20:18] <tsimonq2> AppArmor confinement is enabled by default anyway
 It would be much more than just browsers. (re @kc2bez: If my brain is working correctly, there needs to be more profiles for all the browsers now. That seems tedious.)
 Good point (re @RikMills: It would be much more than just browsers.)
 run # `reverse-depends src:qtwebengine-opensource-src`
 Ouch
 I would suggest people try some random apps from that to see if they are affected
 That is what I did yesterday
 digikam for example also crashes
[20:22] <arraybolt3> Evolution also has issues according to someone there.
[20:22] <arraybolt3> https://bugs.launchpad.net/ubuntu/+source/qutebrowser/+bug/2046844 (I overhauled the bug
[20:22] -ubottu:#lubuntu-devel- Launchpad bug 2046844 in qutebrowser (Ubuntu) "AppArmor user namespace creation restrictions cause many applications to crash with SIGTRAP" [High, Confirmed]
[20:23]  * RikMills subs
[20:23] <arraybolt3> basically since this was caused by AppArmor but intentionally, I set it to Critical/Won't Fix, then added known-affected apps and set them to Confirmed/High
[20:23] <arraybolt3> Thus matching with the requirement for Critical, "Severely affects applications beyond the package responsible for the root cause"
[20:25] <arraybolt3> I'm considering changing the apparmor one to Confirmed/Critical, since what if we need to roll the change back before release to have something usable?
[20:25] <arraybolt3> I think that's the right way to go here, surely the work they did to find packages that needed help missed a lot.
 https://bugs.launchpad.net/ubuntu/+source/falkon/+bug/2046796
[20:26] -ubottu:#lubuntu-devel- Launchpad bug 2046796 in falkon (Ubuntu) "falkon crashed with signal 5 in _nl_make_l10nflist()" [High, Confirmed]
 ^^ was private. now public
[20:29] <arraybolt3> and now marked as dupe since it's another Signal 5 SIGTRAP mess.
[20:36] <arraybolt3> Considering the widespread effects this has, I put messages in #ubuntu-flavors and #ubuntu-devel asking people to contribute to the bug report. The more packages we can find and add AppArmor profiles to, the better. That way we won't have to deal with a broken release or have to beg for the AppArmor change to be reverted.
 *does a chaos and spins a custom Lubuntu 22.04.3 ISO as a general repair system
[20:54] <arraybolt3> :q
[20:54] <arraybolt3> oops
 arraybolt3 waves goodbye secretly :P
 lol
 Yeah, especially given that some upstream apparmor people work for Canonical, I don't think they'd like it at all if we asked for a revery
 *revert
 Curveballs happen, everyone. I don't like how this breaks so many things either, but let's be honest, it came at a pretty darn convenient time to be able to do this kind of work. If you've been around for long enough, you know that not everyone respects Feature Freeze and that's caused issues before. Last cycle was the XDG stuff, I'd imagine this cycle will be full of apparmor and installer stuff
[21:09] <arraybolt3> Considering the level of disruption this may cause, it's still something to keep in the back of our heads (even if we do just start writing AppArmor profiles). People were furious when we chose to do something as simple as not ship Flatpak on the ISOs. If we break an entire class of highly useful AppImages, that could very well come back to bite
[21:09] <arraybolt3> Canonical in a way.
[21:10] <arraybolt3> I also wonder if this breaks Snaps :P
[21:10] <arraybolt3> *downloads Element Desktop via Snap to find out*
[21:12] <arraybolt3> ok, Element Desktop works, so at least there's that
 Flatpaks are too
[22:07] <arraybolt3> hmm, /usr/bin/flatpak has the needed AppArmor profile, doesn't that propagate to apps that are installed under it?
[22:07] <arraybolt3> Maybe not...
 not necessarily no
 consider snapped applications, even if snapd has the proper apparmor profile, the individual snaps need their own AppArmor profiles
 same for appimages
[22:07] <arraybolt3> oof
 LXD's one example, it has a whole AppArmor profile for itself
[22:08] <arraybolt3> This is going to be a fun mess.
 (many of the AA profiles are predefined by the snaps connectors at build time but still)
 enjoy the storm arraybolt3
[22:08] <arraybolt3> At least it sounds like something that can be worked through though. We're going to give developers a hard time, but with some documentation and SEO perhaps that can be worked around.
 you opened the can of worms :P
[22:08] <arraybolt3> @teward001: I love storms :D
 well this is why 'flatpaks' on their own are not defaults
[22:09] <arraybolt3> Nah, Canonical opened it, I just pointed at the worms crawling out
 sure 'flatpak' exists, but there aren't *default* flatpaks anywhere without profiles
[22:17] <arraybolt3> well Epiphany Flatpak works even without any AppArmor config
 🤨
 That makes me unhappy. (re @RikMills: digikam for example also crashes)
 Tip of the iceberg
 Yeah. When it goes from browser and gets into the photography realm, that's when I get involved and things get interesting.
 Will also crash plasma if you try to add some widgets that use qtwebengine
 Yep. At that point the entire DE is affected. Unfortunately, one could say "that's using something outside the repos, so idgaf", and, pedantically as we've experience in the past, "kde, so idgaf."
 Well, in this case we simply need to add an AppArmor profile per application.
 In that case, the entire Plasma Desktop would need to be added.
 We should upstream as much as we possibly can, but it's going to be fun trying to identify all the edge case packages
 LXQt works fine (re @Eickmeyer: In that case, the entire Plasma Desktop would need to be added.)
 I'm not sure where the line is drawn, to be completely honest with you
[23:01] <arraybolt3> He's saying, add a plasmoid that calls QtWebEngine, Plasmashell goes up in SIGTRAP most likely.
 Read what Rik just wrote. Adding a widget that uses qtwebengine would crash Plasma.
 It may be different for DEs than for individual applications
[23:01] <arraybolt3> Adding Plasmashell to the list of allowed applications means that anyone can circumvent the new security by just writing a KDE plasmoid.
 I really want to know what widgets these are :P
 Specifically, there's a picture frame one that I'm sure uses qtwebengine to make a slide show. That one comes with Plasma, and would be dead as a result of this.
 We also have to remember that KDE isn't GNOME... it doesn't run in a single thread, so I doubt that it would freeze Plasma entirely
 I dunno, let's find out XD
 If it does affect Plasma, this will also nuke Neon, so I'd escalate appropriately there too
 https://matterbridge.lubuntu.me/30a25b70/file_10272.jpg
 What happens when you click it :P
 Plasma dies
 Oh it does?
[23:05] <arraybolt3> woohoo
[23:05] <arraybolt3> ok this is even worse than some AppImages going belly-up.
 Niceeeeeeeeeeeeeeeeeeeeeeee
[23:06] <arraybolt3> Eickmeyer, RikMills: Care to share those findings at https://bugs.launchpad.net/ubuntu/+source/apparmor/+bug/2046844?
[23:06] -ubottu:#lubuntu-devel- Launchpad bug 2046844 in apparmor (Ubuntu) "AppArmor user namespace creation restrictions cause many applications to crash with SIGTRAP" [Critical, Confirmed]
 I think it crashes and restart
[23:06] <arraybolt3> Add Plasmashell to the list of affected packages
 No, because it's not a bug in plasmashell. (re @lubuntu_bot: (irc) <arraybolt3> Add Plasmashell to the list of affected packages)
 So no completely session fatal (re @RikMills: I think it crashes and restart)
[23:06] <arraybolt3> Eickmeyer: True, but it is a bug that can be *fixed* there. The solution that Canonical intended here is to add an AppArmor profile for everything that needs it.
[23:07] <arraybolt3> They tried to identify those things and fix them, and, well, they missed a bunch.
[23:07] <arraybolt3> They're willing to help fix anything that needs it though, so if things get listed they may fix it for us.
[23:07] <arraybolt3> "I will get a patch together to add these to the set of known applications that need unprivileged user namespaces that we are now shipping profiles for."
[23:07] <arraybolt3> (so says https://launchpad.net/~jjohansen)
 I tried Quassel and it worked fine but I am now wondering if I needed to hover over a link or something to invoke a crash.
 QUrl doesn't trigger it (re @kc2bez: I tried Quassel and it worked fine but I am now wondering if I needed to hover over a link or something to invoke a crash.)
 Interesting
 Honestly, I think this is more of a security shortcut than anything since writing an apparmor profile for plasmashell essentially circumvents the purpose. I don't think someone thought this issue through enough when patching apparmor.
[23:09] <arraybolt3> I mean... I wouldn't put it in those terms for the sake of avoiding conflict, but I tend to agree with you.
 Kay, nobody is allowed to use nukes on this one until January. Am I clear?
[23:09]  * arraybolt3 watches Eickmeyer label January 1st as the Day of Attack
 Nah, I'm just putting this on my list of to-dos.
 Lets just methodically go through all, apps that use qwebengine somewhere and see if they either degfault at startuo, or if trying a common task.
 URLs without a dependency on QtWebEngine are pretty ubiquitous (re @kc2bez: Interesting)
 +1 (re @RikMills: Lets just methodically go through all, apps that use qwebengine somewhere and see if they either degfault at startuo, or if trying a common task.)
 Then in the new year when canonical people are back we will have a string case for whatever action might be best
 +1 (re @RikMills: Then in the new year when canonical people are back we will have a strong case for whatever action might be best)
 Makes sense to me as well.
 Oh, Electron apps too? There goes my entire workflow.
 Including an Ubuntu Studio seeded package.
[23:15] <arraybolt3> Electron **AppImages**... though it might affect usual Electron too.
[23:16] <arraybolt3> If you can run Element right now, it's probably not all Electron apps in general.
 Truth be told, I haven't upgraded anything to Noble yet. Being really cautious.
[23:17] <arraybolt3> Alright, confirmed that Angelfish web browser SIGTRAP's
[23:17] <kc2bez> add privacybrowser and notepadqq
[23:18] <arraybolt3> Trying Cantor...
 Not surprised at this stage
[23:18] <arraybolt3> Eickmeyer: ^ might be Edubuntu territory
[23:18] <arraybolt3> oh actually it definitely is
[23:18] <Eickmeyer> arraybolt3: Can you install the freeshow snap and see if it works?
[23:18] <arraybolt3> will try in a sec
[23:19] <kc2bez> snaps might be ok, they have their own apparmor baggage I think.
[23:19] <Eickmeyer> Oh yeah, there's tons in Edubuntu that use qtwebengine and gtkwebkit both.
[23:19] <arraybolt3> Cantor is SIGTRAP on startup
[23:19] <Eickmeyer> *sigh*
[23:19] <arraybolt3> Freeshow installing...
[23:19] <Eickmeyer> kc2bez: True, but still worth checking.
[23:19] <arraybolt3> (my VM is going to blow up so quickly :P)
[23:20] <Eickmeyer> Freeshow is fairly lightweight.
[23:20] <kc2bez> Absolutely worth checking, I wasn't trying to suggest otherwise.
[23:20] <arraybolt3> Interestingly Cantor's sigtrap showed up in the log before the AppArmor block
[23:20] <arraybolt3> probably just a race condition
[23:21] <arraybolt3> Eickmeyer: Freeshow loads!
[23:21] <Eickmeyer> arraybolt3: \o/
[23:21] <arraybolt3> https://i.imgur.com/IdsC3hh.png
[23:21] <arraybolt3> No guarantee that it works perfectly, but it's a start
[23:21] <Eickmeyer> Considering it's a snap, it's confined, so that makes sense.
[23:22] <arraybolt3> It seems to be working so far
[23:22] <Eickmeyer> You'd actually like that one, arraybolt3.
[23:22] <arraybolt3> heh, looks neat
[23:22] <arraybolt3> I played with some of its features to see what would happen
[23:22] <arraybolt3> Alright, next victim is morph-browser
[23:23] <arraybolt3> btw lemme know what you're testing and what happened, I'm compiling a list here
[23:23] <arraybolt3> morph-browser actually works
 ktorrent?
[23:24] <arraybolt3> it uses webengine?
[23:24] <arraybolt3> That would be weird but I'll try it
 I am tired here, and can't really be asked to get started on VM testing
[23:25] <arraybolt3> meh, that's what I'm for
[23:25] <arraybolt3> KTorrent seems to work, I'm dl'ing Lubuntu
[23:25] <kc2bez> No worries @RikMills get some rest
[23:25] <arraybolt3> Going to stop though since my ISP isn't necessarily torrent-friendly
[23:26] <RikMills> nextcloud-desktop?
[23:26] <RikMills> that is fairly high profile
[23:26] <arraybolt3> that sounds like trouble :)
[23:26] <kc2bez> Installing
[23:27] <arraybolt3> Somehow it seems to be surviving
[23:28] <arraybolt3> (my earlier thing about Epiphany working in a Flatpak may have been a red herring - it seems the apt version is working even without apparmor now, so maybe I did something wrong)
[23:28] <RikMills> this is why we test
[23:28] <RikMills> I guess it is where and why qtwebengine is used
[23:28] <arraybolt3> but yeah, Nextcloud Desktop's web browser features seem to be surviving
[23:29] <arraybolt3> PageEdit...
[23:29] <RikMills> it could also be releated to what that app is asking it to do
[23:29] <arraybolt3> SIGTRAPs.
[23:29] <RikMills> urhg
[23:30] <arraybolt3> yuzu is next (a Nintendo Switch emulator... the weird things you have to install as an open-source dev...)
[23:31] <arraybolt3> can't break it easily, so might be OK
[23:31] <RikMills> I have been doing this for years and still find weird stuff in the archive I never knew existed
[23:31] <arraybolt3> I installed a puzzle game on accident while trying to get Epiphany installed :P
[23:32] <kc2bez> Ooops
[23:32] <RikMills> ok. goodnight all
[23:32] <kc2bez> Take care
[23:32] <arraybolt3> syncthingtray... looks like it has a web view, so it's probably affected, but I can't trigger it easily right now
[23:35] <arraybolt3> trying...
 <RikMills> I have been doing this for years and still find weird stuff in the archive I never knew existed - this, 100x over
[23:36] <arraybolt3> Killed it with SIGTRAP trying to open Syncthing.
[23:36] <arraybolt3> So it is affected.
[23:37] <arraybolt3> Sigil -> dies instantly
[23:38] <arraybolt3> rssguard -> dies instantly
[23:39] <arraybolt3> konqueror testing...
[23:40] <arraybolt3> ...dies instantly
[23:40] <RikMills> this really is a shitty thing on a stick
[23:40] <arraybolt3> This is a nightmare, no doubt. Hopefully one that can be cleaned up, but of all the things done in the name of security, this is one of the most disruptive I've ever seen.
[23:41] <arraybolt3> Ironically the easiest workaround is to just disable QtWebEngine sandboxing, which is the exact *opposite* of secure.
[23:42] <arraybolt3> In the name of security, they disabled a critical security feature for most things.
[23:42] <arraybolt3> Kontact -> dies instantly
[23:43] <arraybolt3> KMail -> dies instantly
[23:43] <Eickmeyer> This is bad.
[23:44] <arraybolt3> Now in theory most of these can just have new AppArmor profiles, so this isn't the end of the world. It's just the end of our world at the moment
[23:44] <arraybolt3> kiwix -> dies instantly
[23:44] <Eickmeyer> arraybolt3: Kontact and Kmail aren't technically separate apps. Kontact is a layer on top of a bunch of apps, so that increases the complexity.
[23:45] <arraybolt3> kchmviewer -> dies instantly (which is weird, this one didn't even need JavaScript according to the description so I'm a bit surprised it's sandboxed, maybe I shouldn't be though)
[23:45] <Eickmeyer> Really, all of the KDE PIM suite being affected is more complex than you realize.
[23:46] <arraybolt3> Oh I'm sure. KDE PIM always looked like a complicated beast.
[23:46] <RikMills> Yeah, it may be one on the libs or lower deps 
[23:46] <Eickmeyer> It is. If one component is affected, all of it is affected, and an apparmor profile for each app won't do it.
[23:46] <arraybolt3> goldendict-webengine -> dies instantly
[23:47] <RikMills> :/
[23:47] <RikMills> shitshow
[23:47] <Eickmeyer> ^ This, tby
[23:47] <Eickmeyer> *tbh
[23:47] <arraybolt3> And I see there's a QtWebEngine plugin for the Gambas programming language which probably will send anything that uses it up in flames.
[23:48] <arraybolt3> The big problem here is that you can't set exclusions on the library level, it has to be on the executable binary level.
[23:48] <arraybolt3> So anything, anywhere, regardless of source, *could* explode now, if it uses QtWebEngine for any feature.
[23:48] <RikMills> gotta go to sleep as I have some critical errands to run for a sick relative in the morning/afternoon tomorrow
[23:48] <Eickmeyer> Yep. That's kinda my point with the PIM.
[23:48] <Eickmeyer> Good night, RikMills .
[23:48] <arraybolt3> Same with anything that touches Webkit.
[23:48] <RikMills> running blood sample around the countyr as the NHS are too slow :/
[23:49] <RikMills> Eickmeyer: good night :)
[23:49] <arraybolt3> RikMills: Thanks for everything! May it all go well with you.
[23:49] <RikMills> ty
[23:50] <arraybolt3> The more I think about it, the more this change just looks unsustainable.
[23:50] <arraybolt3> It was a fantastic idea, but this is not going to work AFAICT.
[23:50] <Eickmeyer> Considering all of the apps you just went through are a gigantic chunk, I agree. 
[23:51] <Eickmeyer> And when Amy gets home, I'm going to have to break the bad news to her that a majority of the Edubuntu apps for Noble are broken.
[23:52] <Eickmeyer> Educational apps tend to use QtWebEngine a lot.
[23:54] <arraybolt3> Remmina seems to survive when using its WWW plugin.
[23:54] <arraybolt3> (which uses webkit)
[23:56] <Eickmeyer> arraybolt3: Is that a web server or a web client?
[23:56] <arraybolt3> Web client.
[23:56] <Eickmeyer> Interesting.
[23:56] <arraybolt3> It looks like most WebKit stuff is probably safe. It's things that depend on bwrap that are at risk.
[23:56] <arraybolt3> (Epiphany happening to be one of them)
[23:56] <Eickmeyer> It might already have an apparmor profile, which makes sense considering what it does.
[23:57] <arraybolt3> let's see if I can get Nautilus to break
[23:57] <arraybolt3> I don't see a profile for it
[23:59] <Eickmeyer> I didn't know Nautilus had any browsing capabilities *at all*.
[23:59] <Eickmeyer> s/browsing/html rendering/