[00:13] <keithzg[m]> How might I figure out why the apparmor service isn't starting on a system? I checked journalctl and all it says is fewer error messages than from before it stopped working as it just seems to hang partway through (forever, meaning I cannot boot this *buntu system without disabling apparmor, annoyingly).
[00:17] <sarnold> you could try running apparmor_parser --replace /etc/apparmor.d  -- see if that spits out errors
[00:18] <sarnold> the full path through systemd's service files through the init script functions is pretty gross, but this is an easy one-liner that should come close
[00:32] <keithzg[m]> sarnold: hmm that just seems to spit out the same few lines logged in the journal, and then like when booting nothing further appears to happen essentially indefinitely
[00:36] <sarnold> keithzg[m]: oh very weird :( you could try something like  : for f in /etc/apparmor.d/* ; do if [ -f $f ] ; then apparmor_parser --replace $f ; else echo skipping $f ; fi ; done
[00:36] <sarnold> i'm hoping that will help you find a specific profile that is giving the parser trouble
[04:00] <keithzg[m]> Hmm, got back to this and it spits out the skipping lines for each of these in order: abi, abstractions, disable, force-complain, local. It then just sits there forever a la before. The next entry in alphabetical order is lsb_release, but the next folder is rsynclog.d, which only contains a README file.
[05:19] <rfm> keithzg[m], if I had weird stuff like that going on the first thing I would do is boot a live cd and run fsck on the thing...\]
[05:44] <keithzg> rfm: Full disclosure, this is a home system that first exhibited this back in *April*; at that point I just outright uninstalled apparmor and meant to get back to it but never did, and was reminded of it today because I rebooted and apparently some updates I had applied or something else I'd installed pulled apparmor back in (this time I merely disabled the service). In the intervening months, I outright replaced the entire drive this 
[05:44] <keithzg> install is on!
[05:56] <rfm> keithzg, if you just imaged the new drive from the old, it would not have fixed any crosslinked files, etc.  I suspect you are in "save what you can and wipe/reinstall" territory...
[05:58] <keithzg> rfm: Everyone always wants to wipe, I want to understand what the actual problem is! Anyways my point was I've definitely fsck'd since then, and then some. (I remember doing it as a precaution after the move.)
[05:58] <keithzg> For my part, I'm fine just not having apparmor installed. But I really wanna know *what* is going wrong, if at all possible.
[06:12] <rfm> keithzg, if your serious about it, the next step is to run the apparmor_parser under strace and see what strange loop it has got to in the end. 
[06:16] <rfm> keithzg, unless you'd rather root around for the apparmor_parser source, recompile with debug syms, and run it under a debugger..  been a long time since I had to do stuff like that (and I was getting paid)
[06:17] <keithzg> rfm: Good point; I'm bad at reading strace but it should at least spit out *something* instead of just tightlipped silence, heh
[06:19] <keithzg> Hmm from that I see it's actually hung up on the usr.bin.tcpdump entry; removing that to see what happens it goes one further in alphabetical order and then stops there too...
[06:22] <rfm> keithzg, so if you want to try to bisect this, maybe the one before usr.bin.tcpdump has some poison, and removing that might change the behavior. Once you've found the entry that causes the problem you could try tracing it, or comparing to a fresh system..
[06:24] <rfm> and I'm off to bed...
[06:24] <keithzg> rfm: G'night!
[06:24] <keithzg> Removing the one before (which happens to be usr.bin.man) doesn't change anything, but yeah it seems weird if each and every one after is somehow the problem, hmm. 
[06:25] <keithzg> Although I did just run into one that *didn't* hold things up, oho (usr.sbin.cups-browsed as it happens)
[06:33] <keithzg> Wellll I think I'm barking up the wrong tree there as I've checked and the service actually starts just fine even if I restore everything I was testing tearing out of /etc/apparmor.d, so whatever apparmor's problem is it's somehow only a problem on boot?
[06:34] <keithzg> (I phrased that awkwardly, what I mean is that with no config changes but already having booted with the apparmor service disabled, manually starting the apparmor service appears to succeed.)
[20:15] <habys> Regarding autoinstall; if the installer crashes you can test commands in the install environment via local console with `curtin in-target --target /target cmd`. Running the same thing fails over SSH, is there a way to invoke curtin via SSH?
[20:18] <dbungert> habys: interesting.  in your working case, is that in the debug shell that subiquity offers to create?  or a vt switch?
[20:19] <habys> these are all remote machines, but I can access this "console" via remote VGA (or whatever video standard)
[20:19] <habys> if it were a desktop then, yeah, the screen and keyboard
[20:20] <dbungert> must be some environment difference.  if you're really motivated I've got a terrible script lying around
[20:21] <dbungert> habys: https://paste.ubuntu.com/p/kQg5sMrWzR/ - depending on the subiquity snap you may need to hack it for python3.10 instead of py3.8
[20:21] <habys> I assume it's running in some kind of container.. Sorry really ignorant here. The `curtin` command claims it's not installed if executed over SSH. I would love to see any kind of script as it may help me figure out how this thing works.
[20:22] <dbungert> really the right answer is to fix the subiquity snap to let direct curtin invocation be a thing
[20:22] <habys> that would be really convenient
[20:23] <habys> this script looks very straightforward, let me see if I can get it working
[20:24] <habys> that totally works. I guess it's mainly the $PATH that was tricking me
[20:24] <habys> thank you!
[20:24] <dbungert> :) glad it helps, thanks for the nudge to fix this
[20:25] <habys> I can open a bug if you want
[20:26] <habys> seems like not a hard fix, maybe some dot file updates or something
[20:27] <dbungert> yea, some enviornment variable tweak or similar is all that should be needed
[20:27] <habys> In the mean time I can set these values with my own early_commands
[20:52] <habys> spoke too soon. everything I try to run is `command not found`. Trying /bin/bash and apt etc
[20:55] <dbungert> habys: early commands or late commands?  in early commands, the target is not populated yet.
[22:06] <habys> oh. obviously :S