mupPR snapd#9745 closed: [RFC] seed: enable uc20 devmode snaps in dangerous models <Bug> <UC20> <Created by anonymouse64> <Closed by anonymouse64> <https://github.com/snapcore/snapd/pull/9745>00:11
mupPR snapd#9762 closed: gadget: prepare gadget kernel refs (0/N) <Skip spread> <UC20> <Created by mvo5> <Merged by bboozzoo> <https://github.com/snapcore/snapd/pull/9762>06:47
mborzeckipstolowski: mvo: morning guys08:09
mvogood morning mborzecki and pstolowski08:11
mborzeckiawfully quiet09:54
* ogra rattles the chains 10:00
pstolowskiyes i just restarted my irc client, thought it was misbehaving ;)10:07
mborzeckiholiday season is clearly upon us10:08
ograon that note ... did anyone see my question about https://forum.snapcraft.io/t/scummvm-snap-failing-to-install-on-rpi-4/21394 yesterday ?10:18
ogra(install hook failing because snapctl is not allowed due to "install in progress")10:18
pstolowskiogra: is this with stable snapd, or edge?10:23
ograthats with stable ...10:23
ograarmhf, debian buster based OS10:23
pstolowskiogra: ok. i'll check this thread later (it's pretty long!), and see if i can reproduce10:23
ograyou need PiOS and an rpi for it though10:24
pstolowskiogra: ah, it's Pi specific? doh10:25
pstolowskiogra: anyway, i'll see if i can deduce anything from the forum posts then10:26
ograyeah and sadly scummvm is one of the apps heavily promoted by the pi foundation so it could be a very typical target for a "first snap" people install on their new pi400 they got for christmas10:26
ograi got HW and an install here, the error is pretty clearly some ordering problem (but obviously only happening on that HW/OS)10:27
ograDez 07 11:54:01 raspberrypi scummvm.daemon[2935]: error: error running snapctl: snap "scummvm" has "install-snap" change in progress10:27
ograthats the message i get10:28
pstolowskiogra: i can reproduce it also with x86 vm (on focal, 2.48.1)10:42
pedronispstolowski: hi, is #9429 now ready for re-review?10:42
mupPR #9429: o/daemon: validation sets api and basic spread test <Needs Samuele review> <validation-sets :white_check_mark:> <Created by stolowski> <https://github.com/snapcore/snapd/pull/9429>10:42
ograoh, wow ...10:42
pstolowskiso that's "good", i can play around with it10:42
ograi cant reproduce it on the same system using 20.10 desktop10:42
ogra(and others in the thread see the same)10:43
ograperhaps we're just lucky there though10:44
ogra(race wise)10:44
alan_gIt is always fun to prove races are really fixed10:48
mborzeckipstolowski: interesting, can you post snap change?10:55
pstolowskimborzecki: https://pastebin.ubuntu.com/p/ktRYVjdndZ/10:56
pstolowskisnap.scummvm.daemon.service: Scheduled restart job, restart counter is at 5.10:57
pstolowskiwhat is that?10:57
mborzeckipstolowski: hmm weird, what is the hook trying to run?10:59
ogramborzecki, https://github.com/snapcrafters/scummvm/blob/master/snap/hooks/install11:05
ograjust "snapctl set"11:05
ogra(it shoudl admitedly perhaps just call "snapctl stop --disable snap.scummvm.daemon")11:06
ogra(and start it from the configure hook)11:06
pstolowskipedronis: hi, sorry, missed your question, yes11:14
mupPR snapd#9771 opened: boot: boot config update & reseal <UC20> <Created by bboozzoo> <https://github.com/snapcore/snapd/pull/9771>11:23
pstolowskineed 2nd review for https://github.com/snapcore/snapd/pull/973211:27
mupPR #9732: asserts: snapasserts method to validate installed snaps against validation sets <validation-sets :white_check_mark:> <Created by stolowski> <https://github.com/snapcore/snapd/pull/9732>11:27
mborzeckipstolowski: ogra: taht would be a configure hook calling snapctl restart?11:31
pstolowskii'm unclear what this snap is doing / should do, i will take a closer look later today11:43
pstolowskibut it clearly shouldn't fail like this11:44
pstolowskiwill also check if edge fixes it11:46
alan_gpstolowski, I wrote the hook scripts. It just checks for and sets a default configuration option that is later read by a launch script.11:57
pstolowskialan_g: hi, yes, sorry,  i understand that; what i mean is i don't have the big picture wrt services of this snap, i need to take a closer look11:58
pedronispstolowski: reviewed12:00
alan_gOh, the launch script just stops the service depending on the configuration option.12:00
pstolowskipedronis: ty12:05
mupPR snapd#9772 opened: desktop/notification: test against a real session bus and notification server implementation <Created by jhenstridge> <https://github.com/snapcore/snapd/pull/9772>12:09
mupPR snapd#9162 closed: gadget: change mountedfilesystemwriter to use resolvedSource (3/N) <Squash-merge> <UC20> <Created by mvo5> <Closed by mvo5> <https://github.com/snapcore/snapd/pull/9162>12:14
ogramborzecki, (sorry, was in a meeting) currently it is the install hook calling "snapctl set" to set a parameter that tells the daemon wrapper to start or not start the service12:28
ograwhile i'm sure just moving the hook to be configure instead of install would help with the race, using snapctl from an install hook should indeed still work12:29
mupPR snapd#9773 opened: interfaces/apparmor: do not fail durin initialization when there is no AppArmor profile for snap-confine <Needs security review> <Created by bboozzoo> <https://github.com/snapcore/snapd/pull/9773>13:04
mborzeckipackage dmitri.shuralyov.com/go/generated: unrecognized import path "dmitri.shuralyov.com/go/generated": https fetch: Get "https://dmitri.shuralyov.com/go/generated?go-get=1": dial tcp connect: connection refused13:19
mborzeckiwhy is this package even being pulled?13:21
pedronisI don't see anything that refers to it13:27
jameshmborzecki, pedronis: it is a dependency of https://github.com/gordonklaus/ineffassign, which is run by the static checks.  It looks like the domain in question has expired13:33
mborzeckijamesh: hm whois says it expires next yearhttps://paste.ubuntu.com/p/wpvGK4xSQr/13:34
jameshmborzecki: you're right.  I put in the wrong query13:35
mborzeckianyways, whether shady or not, it is kind of a bummer it's not on github or somehing13:36
jameshand maybe it is back now? https://dmitri.shuralyov.com/go/generated13:40
mborzeckiheh, urls in import paths13:48
jameshIf we switched to modules, I guess we'd avoid this by only depending on the module proxy being up13:56
ijohnsonwe would avoid many things if we could switch to modules :-)13:57
pstolowskiah i think i understand the issue with scummvm snap13:58
pstolowskiit is the daemon script calling snapctl, which conflicts when the install change that is still running. and i  think it might be racy and may succeed14:01
pstolowskiijohnson: hey, did you see my question yesterday about lxd install hook / namespace slowness? i quit irc shortly after so if you answered this, then i missed that14:03
mborzeckii this case it's not even us, but rather the ineffassign tool14:10
alan_gStarting a daemon before install completes sounds pretty racy to me!14:11
alan_gIs there a way the daemon script could detect this?14:12
ograalan_g, why do you use a script at all ? just use the hooks directly ... make the install hook always stop the daemon, put the logic about starting it based on the setting into the configure hook14:25
alan_gIt's the first way I found that worked. Stopping in the install hook isn't enough to deal with reboots and restarts.14:27
ograalan_g, https://github.com/ogra1/pi-fancontrol-snap/blob/master/snap/hooks/install#L8 similar to this14:28
alan_gAnd IIRC snapctl doesn't do disable14:28
ograalan_g, and this in the configure hook https://github.com/ogra1/pi-fancontrol-snap/blob/master/snap/hooks/configure#L1614:28
ograwe use a similar setup in a lot of customer snaps and that works reliabely14:29
ograjust add some extra logic to check for the setting and drop the script alrogether14:30
alan_gAck. I've not seen problems until now. And didn't see your approach when I first came up with this.14:30
alan_gBut how does your approach avoid the service starting after a reboot?14:31
ograit is dsabled14:32
ograthe install hook calls: "snapctl stop --disable ${SNAP_NAME}.${SNAP_NAME} 2>&1 || true"14:32
alan_gOh! When did that become possible?14:33
ograthe configure hook checks if it is inactive (and can check additionally for the setting) and then calls "snapctl start --enable ${SNAP_NAME}.${SNAP_NAME} 2>&1 || true"14:33
alan_gOr did I just not find the right docs?14:33
ograi think that was always there14:33
ograits after all just a frontend to systemd features14:34
pstolowskiyes it has always been there ;)14:34
alan_gIts a long time ago, but I *wanted* to disable and never figured out how. /o\14:35
pstolowskialan_g, ogra i'm in the standup, give me a moment, i've suggestion for this snap14:35
ograno hurry14:35
ogra(before christmas is fine i think 😄 )14:35
alan_gNot my snap anyway14:35
alan_gBut I have the same logic in several of mine14:36
pstolowskialan_g: so, 1) yes, snapctl stop --disable will be the cleanest14:43
pstolowskialan_g: 2) not sure why install hook has logic around snapctl get, install hook is only run once for the first intallation of the given snap where by definition there is no configuration... Such logic should live in configure hook.14:45
alan_gpstolowski, it's just to make it the same as post-refresh. I didn't realise that configure would be run in both cases.14:47
pstolowski3) the error we're seeing here is caused by daemon.sh calling snapctl when (re)starting during install. we currently detect such situation as a conflict so it fails. The solution to this is to use snapctl get.. to get all the configuration from configure hook and generate a config file from that (in snap data dir), and the daemon just reads the config file on start.14:48
pstolowskianyway, i suppose you won't need a config file after using --disable, but mentioning in case you need something more sophisticated elsewhere14:49
alan_gThanks. I'll try updating one of my snaps and report back on the forum14:53
mborzeckialan_g: that hook will need an update to work on core20 reliably, there's no snap_core, so the check should be modified to `grep -e snap_core= -e snapd_recovery_mode=`14:55
ograit would really be nice if we could have a "snapctl is-core" or something in a future release to not having to grep /proc/cmdline from packages14:57
alan_g@mborzecki, thanks, but that's already in my snaps. But I can mention it to the author14:57
ijohnsonalan_g: I think the other way you could get around "snapctl get daemon" from the daemon script is to just run it in a loop until it works, that way it will start working when the install-snap change is finished14:58
ograwell, the daemon should really only start on core ... the snap is for both, core and desktop ... so you dont want something looping constantly in the background14:59
alan_gI wondered about that. But now I know how to disable from the install hook I don't think it is needed at all.14:59
ijohnsonoh I see14:59
ogra--disable from install, --enable from configure and dropping the wrapper is really the cleanest solution15:00
alan_gI think (conditional) --disable from install and leave the user to enable if they want to covers it.15:01
ogranot sure what you want to make conditional in install here15:02
ograi'd just install with stopped by default and do the conditional stuff from configure15:02
ogra(there are no conditions to check on install since you cant "snap set <snapname>" before the snap is installed)15:03
alan_gThe condition is "if grep -q -e snap_core= -e snapd_recovery_mode= /proc/cmdline"15:04
ograoh, that ...15:04
ograi'D still do that from configure .. but yeah indeed15:04
ogra... missed that15:04
alan_gWell, configure might run if the user configures something15:05
alan_gI just want it on install15:05
ograinstallation calls configure once15:06
ograso you dont need to duplicate code15:06
alan_gWhat duplicate code?15:07
ograyou'D still check "snapctl get daemon", no ?15:08
alan_gWhy? I'd do away with the configuration option and let the user enable the daemon15:09
ograso make install just disable it by default and have configure check for both conditions (on core or daemon=true) and enable it if required15:09
pstolowskii've summarized what i wrote above in the forum15:09
ograwell, i thought you want it to start on core in any case ... but also allow the user to start it as daemon on desktop optionaly15:10
ograso you write a single conditional in configure and have it always come up disabled in install15:11
alan_gif /* not on core */; then snapctl stop --disable $SNAP_NAME.daemon; fi15:12
alan_gin install. Nothing in post-refresh, nothing in configure15:12
ograsure and an additional chck for daemon= in configure ...15:13
ograi'm just proposing to have both conditionals in configure to have a central place15:13
ograso you never need to bother about install anymore15:13
ograeven if conditions change15:13
ograbut up to you really ... i just find it a lot more elegant .. but thats personal taste 🙂15:14
* cachio lunch15:14
cachiomvo, this is failing in debian https://paste.ubuntu.com/p/y4kCcgyrqT/15:15
cachiomvo, any idea bout how to fix it15:16
mvocachio: oh, fun - looks like the archive is inconsistent. could you try a "apt full-upgrade -y" before the "apt build-dep" ?15:16
alan_gI think I'm missing your point. What condition might change?15:16
cachiomvo, sure, thanks15:16
ograwell, you just had one that changed 😉 from snap_core to snapd_recovery_mode ...15:17
ograbut really, do as you like ... lets not discuss style as log as we get a fix out 🙂15:18
alan_gAFAICS its harder in configure as we only want to disable during an initial install15:20
alan_gNot on any random change15:20
ograyes, thats why i'D unconditionally always disable it in install15:20
ograand have all the enablement logic in configure15:20
alan_gThe logic is "if (first time && On desktop) then disable"15:22
ograjust do as you like, really15:22
ograboth hoos run in succession anyway15:22
alan_gI still feel that logic is simpler in install as you know it is first time15:23
ograwell, you still need the daemon= logic in configure in any case15:23
ograbecause your user might want to run a kiosk on classic ?15:24
ograi thought thats the purpose of having daemon=15:24
ograso you give additional control15:24
alan_gSo the user enables  $SNAP_NAME.daemo15:24
mborzeckimvo: pedronis: i've updaed #9629 to the latest version of license data15:25
mupPR #9629: spdx: update to SPDX license list version: 3.10 2020-08-03 <Needs Samuele review> <Simple 😃> <⛔ Blocked> <Created by bboozzoo> <https://github.com/snapcore/snapd/pull/9629>15:25
ograah, so you would ask the user to snap start --enable scummvm.daemon ... instead of snap set ... sure ... that works but wouldnt be usable from a gadget on classic15:25
mborzeckii suppose i can drop the blocked label now too15:26
alan_g"wouldnt be usable from a gadget on classic" is the point I was missing. Thanks!15:26
ogranot a super common case ... but possible15:26
ogra(up to now we talked all diigtal signage users into using core anyway 🙂 )15:27
mborzeckiijohnson: a quick observation, i was able to reproduce the rsa veirification error quite reliably every couple of runs when i was building a kernel with yocto in the background15:37
mvomborzecki: thank you!15:39
ijohnsonmborzecki: interesting15:47
ijohnsonperhaps it is so difficult to reproduce for me because I have so many cores that are not busy :-p15:47
mborzecki2020-12-09T14:35:03.9833068Z Dec 09 14:32:59 ubuntu snapd[1702]: 2020/12/09 14:32:59.119476 stateengine.go:150: state ensure error: devicemgr: cannot mark boot successful: cannot check for fde-setup hook in reseal: cannot get kernel info: no state entry for key16:09
mborzeckimvo: any clues what that might be about? ^^16:09
mborzeckihm found more weird logs:16:10
mborzecki2020-12-09T14:35:03.9588849Z [   55.176506] snapd[1702]: 2020/12/09 14:33:01.515032 stateengine.go:150: state ensure error: devicemgr: cannot mark boot successful: cannot identify kernel snap with bootloader grub: cannot read dangling symlink kernel.efi16:10
mborzeckilooks like this happens right after install too, this appears when booting into run mode for the first time:16:13
mborzecki2020-12-09T14:35:03.8893265Z [   32.599045] snapd[885]: stateengine.go:150: state ensure error: devicemgr: cannot mark boot successful: cannot check for fde-setup hook in reseal: cannot get kernel info: no state entry for key16:13
N3bulaKHey guys16:18
N3bulaKI have been referred to ask a question here16:18
N3bulaKI am trying to copy a snap to another machine with a different username16:18
N3bulaKif I just copy the folder over, that doesn't work16:19
ogratechnically you should install the snap newly, take a snapshot on the old host and restore it on the new host ... but that will likely not handle changed user name or changed UID for user data16:20
ograperhaps someone with more insight into snapshots can give a hint if it is possibel to restore snapshots to a new user16:22
N3bulaKBTW, snap is question is bluemail16:23
N3bulaKogra: tried that but that doesn't work due to username16:23
ograyou will definitey need to install the snap anew ... you can surely also restore the system bits from a snapshot ... perhaps then copying the ~/snap/bluemail/current/* content is enough16:24
mvopedronis: I have this feeling that 9149 has too much in it, it's a bit messy, should I split it into one PR that does the "$kernel:ref" validation, one PR that implements gadget.ResolveContentPaths() and one that uses ResolveContentPaths() ? wdyt?16:25
pedronismvo: that's fine with me, it's not very large, but that sequence seems easier to review16:26
mupPR snapd#9149 closed: gadget: provide new gadget.ResolveContentPaths() (2/N) <Needs Samuele review> <Squash-merge> <UC20> <Created by mvo5> <Closed by mvo5> <https://github.com/snapcore/snapd/pull/9149>16:30
mupPR snapd#9774 opened: o/snapshotstate: don't set auto flag in the snapshot file <Needs Samuele review> <Created by stolowski> <https://github.com/snapcore/snapd/pull/9774>16:35
pstolowskipedronis: ^16:39
pedronispstolowski: thx16:41
* ijohnson short break 16:56
alan_gpstolowski, would there be the same problem with using `snapctl is-connected` in a launch script?17:00
ograwhy would you do that from a launch script instead of a hook ?17:07
ograalan_g, https://github.com/ogra1/pi-fancontrol-snap/tree/master/snap/hooks ... se the connect hooks17:09
ogra(and how the configure hook uses is-connected alongside)17:09
alan_gI've existing scripts that check for the wayland and x11 interfaces to figure out how to launch17:10
ograi doubt it makes any difference what comes after snapctl ... the call itself is the issue17:10
alan_gI suspect as much too. But hoped...17:11
alan_gSo configure runs on connect/disconnect?17:11
ograno, the connect hooks do17:12
ograconfigure just uses is-connected and exits zero if a connection is missing17:12
ograbefore it starts (or restarts) the service ....17:12
* alan_g is tempted to keep calling snapctl until it works17:13
ograin a crazy loop 🙂17:15
pstolowskialan_g: no, that should be fine17:17
pstolowskialan_g: but also i was slightly wrong about the source of the conflict, it's actually 'snapctl stop ..' in the daemon-start.sh (not snapctl get) triggering this (it's fine to do this from hooks, but in daemon it conflicts with install as explained earlier)17:19
pedronisit seems we have tests that generate real notifications?17:20
alan_gpstolowski, that seems less awkward. But that means a daemon can't stop itself in the case of persistent problems?17:22
pedronisit can but needs to deal with conflict errors17:23
pedronismaybe we need those to be more detectable17:23
alan_guntil snapctl stop --disable $SNAP_NAME.daemon; do sleep 1; done # Ugh!17:28
pstolowskiwhy the loop?17:31
alan_gto deal with conflict errors17:32
alan_gOr have I misunderstood the failure mode?17:32
pstolowskialan_g: if you do this from hook then it should just work17:33
pstolowskii.e. won't conflict17:33
alan_gBut the hook doesn't know that the daemon has encountered a persistent error17:33
mupPR snapd#9775 opened: gadget,o/devicestate,tests: drop EffectiveFilesystemLabel and instead set the implicit labels when loading the yaml <Cleanup :broom:> <Run nested> <UC20> <Created by pedronis> <https://github.com/snapcore/snapd/pull/9775>17:35
alan_gBut it could `snapctl set killme=true` and the configure hook would process that?17:36
pstolowskisorry, i need to run, need to taxi my daughter, let's talk tomorrow (oe maybe ijohnson can help)17:39
* ijohnson is back17:46
ijohnsonalan_g: I'm a bit confused where you're at right now17:47
alan_gijohnson, I understand the immediate problem and solution. But I'm just imagining the hypothetical circumstance of a daemon that hits a persistent problem at runtime and elects to stop itself. In that case it is necessary to "deal with conflict error". I see two ways to do that:17:50
alan_g1. until snapctl stop --disable $SNAP_NAME.daemon; do sleep 1; done17:50
alan_g2.  `snapctl set killme=true` and the configure hook would process that?17:51
ijohnsonso just to make sure we are on the same page, `snapctl stop --disable ...` needs to be run in a loop because if the daemon runs very fast, snapctl may fail due to an conflict in progress ?17:52
ijohnsoni.e. install-snap in progress or some such error message17:52
alan_gIt's not blocking anything right now. Just want to confirm my understanding.17:54
ijohnsonok, then imho having `snapctl stop --disable` run in a loop until it works is the cleaner solution17:56
ijohnsonI think there is maybe things we can do in snapd to make `snapctl stop --disable` work when there is a conflict like this, but it's unclear how exactly that would be implemented17:56
ijohnsonI guess just as a user seeing `snap get <your-snap>` and seeing `killme: true` would be a bit unexpected and confusing17:57
ijohnsonoh wait actually you can't do that17:57
ijohnsonbecause when you run `snapctl set` that does _not_ trigger the configure hook to run17:58
ijohnsonso `snapctl set killme=true` would be racing with the configure hook itself and would fail anyways17:58
alan_gThe problem with the loop is that it isn't obvious it is needed and "just works" without most of the time. (Until a user hits a weird problem on some new device.)18:01
ograbut why do you need it at all ?18:02
ograthe hooks offer everything ou need18:02
ograand they save you from having to use a wrapper at all usually18:02
alan_gogra, I understand the immediate problem and solution. But I'm just imagining the hypothetical circumstance of a daemon that hits a persistent problem at runtime and elects to stop itself. The hooks are not running.18:03
ijohnsonwell sometimes things are not obvious, that's what code comments are for :-P18:04
=== ijohnson is now known as ijohnson|lunch
ograwell, thats something you'd probably manage via an additioanl watcher service then18:04
* alan_g hits EOD18:06
mupPR snapd#9776 opened: gadget: add validation for "$kernel:ref" style content <UC20> <Created by mvo5> <https://github.com/snapcore/snapd/pull/9776>18:15
=== ijohnson|lunch is now known as ijohnson
mupPR snapd#9777 opened: gadget: add gadget.ResolveContentPaths() <UC20> <Created by mvo5> <https://github.com/snapcore/snapd/pull/9777>18:50
* cachio afk19:29
ijohnsonpedronis: do we have any current assertions or assertion examples where a list is empty? it seems to me that we have no such example and I can't seem to convince the assertion decoding function to understand what an empty list is, which leads me to believe that the assertion format doesn't support empty lists and only allows fields to be omitted if they are empty19:49
ijohnsonindeed, if I try signing a system user assertion json with an empty string for serials, serials is just omitted from the produced assertion19:49
N3bulaK@ogra tried copying the /current/* but still nothing :(20:18
pedronisijohnson: yes, empty and omitted are equivalent20:18
ijohnsonthanks for clarifying20:19
N3bulaKis there a way I can backup snaps to be deployed on another machine with a different username20:19
ograN3bulaK, so yu installed a fres snap from the store, took a snapshot of the system config of the old one and copied the content of current/* (making sure all "dot dirs are included) ?20:20
ograN3bulaK, also whats the actual issue you see with that ? is just data missing, does the app not start etc etc20:20
pinuscHello! I'm having some trouble with my snap install of LXD20:28
pinuscI think the problem is that I configured LXD to set storage pools in /data/lxd (note: /data is a btrfs drive), and snap is not mounting them in /var/lib/snapd/hostfs/data/lxd, which apparently is where lxd expects them20:29
pinuscThe strange thing is, this setup worked flawlessly two weeks ago! Then I shut off the server for a while, powered it on today, and lxd won't even start anymore20:30
pinuscHere's the output of `sudo lxd --debug --group lxd`: https://l.termbin.com/rlaiu20:30
mupPR snapd#9479 closed: tests: replace pkgdb.sh (library) with tests.pkgs (program) <Created by zyga> <Merged by sergiocazzolato> <https://github.com/snapcore/snapd/pull/9479>20:36
ograstgraber, ^^^20:38
stgraberpinusc: your error is about: EROR[12-09|20:11:51] Failed to start the daemon: Failed initializing storage pool "default": Failed to mount '/var/lib/snapd/hostfs/data/lxd/common/lxd/disks/default.img' on '/var/snap/lxd/common/lxd/storage-pools/default': not a directory20:58
stgraberpinusc: oh, I see20:58
stgraberpinusc: what the tell is this mess, we don't support disks/ being anywhere other than /var/snap/lxd/common/lxd/disks/20:59
pinuscI honestly have no idea20:59
pinuscThis was my first time installing lxd and I'm very confused about configuring storage21:00
stgraberpinusc: Does /data/lxd/common/lxd/disks/default.img exist on your system?21:00
pinuscI'm not sure I exactly understand how snap works, but is /var/lib/snapd/hostfs supposed to contain some sort of bind mount of the root fs? Because right now it's completely empty, which I think is the problem21:02
stgraberwhat does `sudo nsenter --mount=/run/snapd/ns/lxd.mnt ls -lh /var/lib/snapd/hostfs/data` show you?21:02
stgraberyou can't see the content of /var/lib/snapd/hostfs from outside the snap, that's normal21:02
pinuscIt shows me the contents of /data21:04
stgraberwhat does `sudo nsenter --mount=/run/snapd/ns/lxd.mnt readlink -f /var/lib/snapd/hostfs/data/lxd/common/lxd/disks/default.img` show you?21:05
stgraberand `readlink -f /data/lxd/common/lxd/disks/default.img` without the nsenter stuff for good measure21:06
pinuscthe nsenter one prints /var/lib/snapd/hostfs/data/lxd/common/lxd/disks/default.img21:06
pinuscWithout nsenter it prints nothing and exits with 121:07
stgraberwhat does `sudo nsenter --mount=/run/snapd/ns/lxd.mnt ls -lh /var/lib/snapd/hostfs/data/lxd/common/lxd/disks/default.img` show you?21:08
pinusc-rw------- 1 root root 11G Nov 18 21:19 /var/lib/snapd/hostfs/data/lxd/common/lxd/disks/default.img21:09
stgraberok, so that environment looks happy enough now, what does `lxc info` show you?21:12
pinuscError: Get "http://unix.socket/1.0": dial unix /var/snap/lxd/common/lxd/unix.socket: connect: permission denied21:13
pinuscWhich is expected because lxd won't start at all21:13
mupPR snapd#9778 opened: asserts/repair.go: add "bases" and "modes" support to the repair assertion <UC20> <Created by anonymouse64> <https://github.com/snapcore/snapd/pull/9778>21:16
N3bulaKogra: App doesn't start, I have tried copying the folders etc but to no avail22:33
N3bulaKbut If I remove the copied folders then it works fine without any previous data22:33
pinuscstgraber: any more ideas on what I could try? Sorry for the insistence, I'm completely lost23:06
stgraberpinusc: tried `sudo lxc info`?23:07
pinuscYup, same Error: Get "http://unix.socket/1.0": dial unix /var/snap/lxd/common/lxd/unix.socket: connect: connection refused23:07
stgraberpinusc: ah, that's better than the permission denied23:07
stgraberpinusc: try `systemctl restart snap.lxd.daemon`23:07
stgraberwell, `sudo systemctl restart snap.lxd.daemon`23:08
pinuscThe command exits cleanly, but the service fails soon23:09
pinuscOh, there's a (new?) error23:10
pinuscFailed initializing storage pool \"default\": Source path '/var/lib/snapd/hostfs/data/lxd/common/lxd/disks/default.img' isn't btrfs"23:10
stgraberI really wonder how you managed to get yourself into such a broken situation in the first place, LXD shouldn't have ever let you put default.img anywhere other than /var/snap/lxd/common/lxd/disks/23:11
pinuscThat's a very good question23:12
pinuscThe weird thing is, it used to work23:12
stgraberthe first failure you got was because LXD started before your /data mount was mounted, now you're hitting an error because LXD assumes that any source path outside of /var/snap/lxd/common/lxd/disks refers to a block device or a path, which your setup definitely doesn't match23:12
pinuscI *might* have created the default.img and then moved it somewhere else and then changed the config to reflect that23:12
stgraberhmm, yeah but LXD wouldn't have let you change the source property, it's read-only. The only way you could make such a mess short of us having a bug that let you do it another way is through `lxd sql` by directly updating the DB23:13
pinuscThat I am sure I did not do23:13
stgraberanyway, do you have enough space on /var/snap/lxd/common/lxd/disks to store that default.img file?23:14
N3bulaKI will ask the question again as there are more people active at this very moment :D23:15
N3bulaKI am trying to copy snap data to another machine with a different  username but can't make it work23:15
pinuscstgraber: yes, I do23:15
N3bulaKsnap is question is bluemail23:16
stgraberpinusc: ok, then move it where it should be at /var/snap/lxd/common/lxd/disks/default.img23:16
stgraberpinusc: do you have more than one storage pool configured?23:16
pinuscstgraber: Also, I went through my .bash_history and I did indeed mv the directory under /data/lxd, and then the next relevant command is `sudo lxc storage edit default`23:16
pinuscstgraber: nope23:17
pinuscNo idea what I did in storage edit, but I guess just changed the path?23:17
stgraberpinusc: yeah, apparently there's a bug that lets you change it... I'll have someone sort that out tomorrow23:18
pinuscOh, that would be good... I just assumed that this setting was fina23:18
pinuscAre you a lxd maintainer?23:18
stgraberpinusc: onece you have default.img moved back where it belongs, you can create a file at "/var/snap/lxd/common/lxd/database/patch.global.sql" containing "UPDATE storage_pools_config SET value='/var/snap/lxd/common/lxd/disks/default.img' WHERE key='source';". Then restart LXD. The database should get updated with the correct path and hopefully things will start back up.23:20
stgraberpinusc: I'm the LXD project leader.23:20
pinuscOh wow, thank you for helping23:22
pinuscThe lxd daemon now starts up fine!23:22
pinuscThough containers fail to start for some reason...23:22
pinuscI'll see if I can debug that23:22
pinuscstgraber: I'm getting Failed to mount rootfs "/var/snap/lxd/common/lxd/containers/synapse/rootfs" onto "/var/snap/lxd/common/lxc/" with options "(null)"23:36
pinuscWhen I try to launch a (existing) container23:37
pinuscNew containers, however, run fine23:37
stgraberpinusc: what's `ls -lh /var/snap/lxd/common/lxd/containers/` showing you?23:37
pinuscLinks to /var/snap/lxd/common/lxd/storage-pools/default/containers/CONTAINERNAME23:38
stgraberok, that part is good then23:38
stgraberls -lh /var/snap/lxd/common/mntns/var/snap/lxd/common/lxd/storage-pools/default/23:38
pinuscSome dirs, including containers/23:39
pinuscOooh, inside containers/ is one dir per container, but the owner might be wrong. I have root:root for everything, except the one I just created (which is the only one which works), which has 1000000:root as permission23:41
pinuscAlso, the old ones are empty, except for a backup.yaml, whereas the new one has other stuff---including rootfs23:44
stgraberCan you check if you see anything at `/var/snap/lxd/common/lxd/storage-pools/default`? you shouldn't but given the current mess, it's not impossible that some of the data ended up there somehow?23:46
pinuscNope, empty23:47
stgraberpinusc: oh, I think I may know what happened but you're not going to like it23:52
stgraberpinusc: were those containers created after default.img got moved but prior to the next system reboot?23:53
pinuscVery likely23:53
stgraberpinusc: and are /data and /var/snap/lxd on different partitions?23:53
stgraberright, then I'm afraid that you're screwed. You see there is no such thing as moving a file between two mounts, when you `mv` between two mounts, the source is copied and then deleted. In your case, the source was still mounted and actively being used. When that happens Linux succeeds in deleting the file but actually keeps it active on disk until such time as the last thing that has it open closes it.23:55
stgraberpinusc: so when you moved default.img, LXD never actually used the moved path in /data, instead it just kept using the now delete file under /var/snap/lxd23:55
pinuscOoh, I see23:55
stgraberpinusc: after a reboot, the data in /var/snap/lxd is gone forever and your data in /data is effectively a copy of a very old state23:56
pinuscSo that also explains why it was working before. On a reboot, it tried to actually access /data for the first time, and failed23:56
pinuscI have to say, it sucks that I lost all that was in the containers... but this is a satisfying answer23:57
pinuscI was dumb and I got bitten23:57
pinuscBefore I proceed, I'll make sure to actually read the documentation and properly set up a storage pool on an external media23:58
pinuscMeanwhile, i guess I'll just have to delete my containers and start from scratch...23:58
stgraberthe best is to have a dedicated disk or partition for LXD23:59
stgraberthen during LXD init you will be prompted for whether you have one of those for your storage pool23:59
stgraberLXD will then automatically mount it for you on startup all inside its mount namespace23:59
pinuscYeah, I guess I'll make a btrfs subvolume23:59

Generated by irclog2html.py 2.7 by Marius Gedminas - find it at mg.pov.lt!