[00:32] <mup> PR core20#55 closed: hooks: make systemd-modules-load depend on mounts at /usr/lib/{firmware,modules} <Created by anonymouse64> <Closed by anonymouse64> <https://github.com/snapcore/core20/pull/55>
[00:57] <mup> PR snapd#8623 opened: tests/lib/prepare.sh: delete patching of the initrd <UC20> <⚠ Critical> <Created by anonymouse64> <https://github.com/snapcore/snapd/pull/8623>
[05:49] <mborzecki> morning
[05:57] <mup> PR snapd#8622 closed: cmd/snap-bootstrap/initramfs-mounts: add sudoers to dirs to copy as well <UC20> <Created by anonymouse64> <Merged by bboozzoo> <https://github.com/snapcore/snapd/pull/8622>
[06:30] <mborzecki> mvo: hey
[06:31] <mborzecki> mvo: we can land https://github.com/snapcore/snapd/pull/8623
[06:31] <mup> PR #8623: tests/lib/prepare.sh: delete patching of the initrd <UC20> <⚠ Critical> <Created by anonymouse64> <https://github.com/snapcore/snapd/pull/8623>
[06:31] <mvo> mborzecki: cool
[06:32] <mvo> mborzecki: done
[06:32] <mborzecki> mvo: thanks
[06:33] <mup> PR snapd#8623 closed: tests/lib/prepare.sh: delete patching of the initrd <UC20> <⚠ Critical> <Created by anonymouse64> <Merged by mvo5> <https://github.com/snapcore/snapd/pull/8623>
[06:33] <mborzecki> seems like that go 1.9 failure in wrappers is random after all
[06:51] <zyga> good morning
[06:51] <zyga> eventful evening?
[06:52] <mvo> zyga: good morning - we did all the bits missing for beta I think
[06:52] <mvo> zyga: snapd snap just building, that was the last piece, then we hopefully have a working beta
[06:52] <zyga> that's impressive
[06:53] <zyga> mvo: I'll do what claudio mentioned
[06:53] <zyga> boot it and leave it running
[06:53] <zyga> an aging test
[06:53] <mvo> zyga: nice
[06:53] <mborzecki> zyga: hey
[06:54] <zyga> :-)
[06:54] <zyga> guys, if you have daughters
[06:54] <mborzecki> hm?
[06:54] <zyga> then spend 30zł / < 10 euro
[06:55] <zyga> and play it with them
[06:55] <zyga> https://www.gog.com/game/foxtail
[06:55] <zyga> this game is incredibly beautiful and fun
[06:55] <zyga> localized, with click-to-advance in dialogue, so kids can read at their own pace
[06:55] <mborzecki> zyga: hmm looks like an old school adventure game?
[06:55] <zyga> it is
[06:56] <mborzecki> zyga: do you remember teenagent? :P
[06:56] <zyga> and the mood and looks are just so nicely crafted
[06:56] <zyga> mborzecki: yes but this is way better :)
[06:56] <zyga> (I bought teenagent as a teen)
[06:56] <mborzecki> no way xD
[06:56] <zyga> yeah :D
[06:57] <mborzecki> too bad gog doesn't support their client on linux
[06:57] <zyga> there's no DRM and it supports linux as well
[06:57] <zyga> mborzecki: yeah but the game is just a zip to download and run
[06:59] <mborzecki> zyga: well, it is, but it's clearly a work in progress title, so on linux you'll have trouble keeping up with the updates, none of the open source clients i tried seem to support updates nicely and there's some issue with gog updates api that apaprently makes it hard to integrate
[06:59] <zyga> mborzecki: this game is updated roughly once a year
[06:59] <zyga> mborzecki: anyway, just consider it
[07:00] <zyga> it really is worth the money
[07:00] <zyga> mborzecki: each update brings the next episode (3 out of 7 are available now)
[07:00] <zyga> it is made by a tiny studio
[07:03] <pstolowski> morning
[07:03] <zyga> good morning pawel!
[07:13] <mborzecki> anyone able to reproduce the unit test timeout in wrappers package?
[07:13] <zyga> mborzecki: on master?
[07:15] <zyga> mborzecki: I started go test -count 1000 ./wrappers/
[07:15] <zyga> I'll let you know if it triggers
[07:16] <mborzecki> pedronis: hi, were you able to reproduce the unit test timeout in wrappers by any chance?
[07:16] <pedronis> no
[07:17] <mborzecki> hm intersting, also the PRs that ijohnson opened yesterday were green today
[07:17] <pedronis> it's definitely weird
[07:17] <pedronis> also it might be that something else failed and the panic covers it because it doesn't get printed
[07:18] <mborzecki> yeah, that's true
[07:18] <pedronis> mvo: hi, it's not needed for the beta but this also needs to be backported https://github.com/snapcore/snapd/pull/8602  I don't see it in 2.45 yet
[07:18] <mup> PR #8602: configcore: only reload journald if systemd is new enough <Squash-merge> <Created by stolowski> <Merged by stolowski> <https://github.com/snapcore/snapd/pull/8602>
[07:21] <pedronis> mvo: this also wasn't ported  https://github.com/snapcore/snapd/pull/8622/ ?
[07:21] <mup> PR #8622: cmd/snap-bootstrap/initramfs-mounts: add sudoers to dirs to copy as well <UC20> <Created by anonymouse64> <Merged by bboozzoo> <https://github.com/snapcore/snapd/pull/8622>
[07:30] <mvo> pedronis: yes, that is right, let me fix that
[07:34] <zyga> pedronis: I moved https://github.com/snapcore/snapd/pull/8566 to snaplock
[07:34] <mup> PR #8566: cmd/cmdutil: add run inhibition operations <Created by zyga> <https://github.com/snapcore/snapd/pull/8566>
[07:34] <zyga> once this is +1 I will adjust the dependencies
[07:34] <pedronis> thx
[07:35] <pedronis> need to do some other things before getting back to reviews
[07:35] <zyga> sure
[07:35] <zyga> I have plenty of things to write so not blocked
[07:35] <mup> PR snapd#8624 opened: cmd/snap: fix the order of positional parameters in help output <Simple 😃> <Created by bboozzoo> <https://github.com/snapcore/snapd/pull/8624>
[07:37] <mborzecki> trivial fix ^^
[07:38] <zyga> mborzecki: I reproduced the panic
[07:38] <zyga> https://www.irccloud.com/pastebin/vmigrXPM/
[07:46] <zyga> mborzecki: so
[07:46] <zyga> mborzecki: this trace is funny
[07:46] <zyga> mborzecki: we clearly exec something
[07:46] <zyga> yet the SetUpTest says
[07:46] <zyga>     s.systemctlRestorer = systemd.MockSystemctl(func(cmd ...string) ([]byte, error) {
[07:46] <zyga>         s.sysdLog = append(s.sysdLog, cmd)
[07:46] <zyga>         return []byte("ActiveState=inactive\n"), nil
[07:46] <zyga>     })
[07:46] <zyga> which seems to suggest this mock is one thing
[07:47] <zyga> but calling "systemctl" executable is another
[07:47] <mborzecki> zyga: which go version is that?
[07:47] <zyga> 1.13.8
[07:47] <zyga> brb
[07:48] <mborzecki> ok, so not go specific, probably just borked setup
[07:48] <mborzecki> or test
[08:35] <zyga> bugs, bugs, bugs
[08:35] <zyga> but also progres
[08:40] <pedronis> #8564 got a +1 from pstolowski and now I applied his suggestions and is ready for 2nd reviews
[08:40] <mup> PR #8564: asserts: introduce Pool <Bulk assert refresh :scroll::scroll::scroll:> <Created by pedronis> <https://github.com/snapcore/snapd/pull/8564>
[08:53] <zyga> pedronis: what is the 'h' assertion header for?
[08:53] <ackk> mvo, hi, I canceled today sync as there isn't really anything to talk about
[08:53] <mvo> ackk: ok, I see there is a bug mentioned, I have a look at that
[08:54] <ackk> mvo, ah yeah, not sure if it's really relevant for snapd, but if you want we can talk about it. it's just me today
[08:54] <ackk> mvo, mostly, I added it there to ask if there was a best practice for knowing the underlying OS details from a snap
[08:55] <mvo> ackk: no need to meet just for that :)
[08:55] <mvo> ackk: we can cover it next time
[08:55] <ackk> right
[09:01] <pedronis> zyga: they are test-only assertions, they look a bit like snap-declarations and snap-revisions
[09:02] <pedronis> zyga: but unless you reviewed the other ones in this chain, I wouldn't recommend looking a tit
[09:02] <pedronis> heh
[09:02] <pedronis> at it
[09:04] <pedronis> mborzecki: so that test failed again with the timeout
[09:05] <pedronis> mvo: some the unit tests changes that came with jamesh PR that was merged yesterday are failing regularly but is not super clear what is going on
[09:06] <pedronis> mvo: do you need to do a pre4 ?
[09:08] <ackk> mvo, I have a very basic snap with just meta/snap.yaml (to test the system-files interface). I snap pack it but when I install I get - Run configure hook of "testsnap" snap if present (run hook "configure": cannot snap-exec: no such file or directory)
[09:08] <ackk> mvo, why is it trying hooks when there are none?
[09:09] <zyga> ackk: do you have meta/hooks/configure?
[09:09] <ackk> zyga, nope
[09:09] <zyga> though the snap-exec error is also curious
[09:09] <ackk> just meta/snap.yaml and "script" , which is a bash script
[09:09] <ackk> I'm using base: bare if that mattes
[09:09] <ackk> *matters
[09:09] <zyga> ah
[09:10] <zyga> bare base has no shell?
[09:10] <zyga> no /bin/sh
[09:10] <ackk> oh,
[09:10] <zyga> though the configure hook is weird still
[09:10] <zyga> it's bare for a reason :D
[09:10] <zyga> in fact, it has nothing
[09:10] <zyga> you must provide a static binary
[09:10] <ackk> right, I forgot
[09:11] <mborzecki> pedronis: can you reproduce it?
[09:11] <mborzecki> pedronis: or was that in some PR?
[09:11] <pedronis> mborzecki: a PR
[09:12] <mvo> pedronis: not sure if we need a pre4, I hope we can test what we have in beta and if it's all working fine I can do a 2.45 for real but it depends a bit. do you know already something that is broken?
[09:12] <pedronis> mvo: no, just wondered because sudoers bits are not in pre3
[09:13] <pedronis> so you cannot really test recover
[09:13] <pedronis> afaiu
[09:13] <mvo> pedronis: yeah, that was silly of me, I cherry picked it and pushed to release/2.45 and rebuild the beta. so it will be ~pre3+git but that should be ok for testing :)
[09:14] <mborzecki> pedronis: commit d16c44f0aa mentiones some issues with shutting down the session agent
[09:15] <pedronis> I know
[09:15] <pedronis> maybe we should try undoing and then it happens for often and try to really understand
[09:15] <pedronis> s/for/more/
[09:16] <pedronis> anyway this is very annoying
[09:25] <mborzecki> hm wonde rif it's possible that the agent hits idle timeout and stops listening
[09:26] <pedronis> mborzecki: ah
[09:26] <pedronis> maybe should be possible to write a test to see what happens in that case
[09:26] <mborzecki> trying to provoke that in tests
[09:26] <pedronis> but there's shoould be an Accept around
[09:26] <pedronis> in that case, no?
[09:26] <pedronis> sorry, there shouldn't be
[09:27] <mborzecki> ah right
[09:30] <pedronis> zyga: I did another pass on #8566
[09:30] <mup> PR #8566: c/snaplock/runinhibit: add run inhibition operations <Created by zyga> <https://github.com/snapcore/snapd/pull/8566>
[10:48] <pstolowski> hmm we could use systemd version check to avoid LazyUnmount on core16 (which generates a lot of noise in system log)
[10:49] <zyga> pedronis: thank you for the review, I replied to one comment and I'll adjust the rest later today
[10:49] <zyga> pstolowski: +1
[10:49] <zyga> pstolowski: but then we never rewrite .mount units
[10:49] <zyga> pstolowski: so -1 unless you know how to do that on systemd upgrades
[10:50] <pstolowski> ah, hmm
[10:50] <pstolowski> anyway, something to think about
[11:01] <mup> PR snapd#8625 opened: wrappers: tweak the  order of restoring, use client timeout in services test teardown <Skip spread> <Created by bboozzoo> <https://github.com/snapcore/snapd/pull/8625>
[11:01] <mborzecki> pedronis: ^^ maybe this will give us more insight
[11:07] <mborzecki> hm removed that thing poking the client in teardown and can reproduce the timeout now
[11:11] <pstolowski> zyga: maybe we could make systemd version part of systemkey
[11:12] <zyga> pstolowski: we don't rewrite mount units on system key
[11:12] <pstolowski> zyga: yes, but maybe then we would
[11:12] <zyga> pstolowski: what happens when you rewrite mount unit
[11:13] <zyga> I tried to propose that but it got stuck on this discussion
[11:13] <zyga> do we remount things? we cannot really
[11:13] <pstolowski> allright... just thinking aloud
[11:13] <zyga> no no, that's good
[11:13] <zyga> it's just we need to think how to do that in practice
[11:15] <pedronis> zyga: interesting, firefox does this HOME: $SNAP_USER_COMMOM in its snap.yaml
[11:15] <zyga> I hope it's COMMON :)
[11:15] <zyga> but interesting, yeah
[11:15] <zyga> we give people the means and they get creative
[11:26] <mborzecki> meh, go test -timeout is kinda silly
[11:26] <mup> PR snapd#8626 opened: tests: fix passing stdin via session-tool <Created by zyga> <https://github.com/snapcore/snapd/pull/8626>
[11:27] <zyga> mborzecki: ^ a trivial patch and an annoying discovery
[11:27] <pedronis> mborzecki: your PR is green,  so maybe we were trying to talk to the real agent?
[11:27] <pedronis> do we need that poking code?
[11:28] <mborzecki> yeah, but it'd hard to explain why there would be another agent
[11:28] <zyga> maybe we activate it via socket?
[11:28] <zyga> do we mask the real agent?
[11:28] <mborzecki> however, go test ./... runs all package tests in parallel, if there's another package starting the agent, we could try to talk to taht other agent ?
[11:28] <zyga> ah, unit tests
[11:28] <mborzecki> (assuming mocking elewhere is incorrect too)
[11:29] <pedronis> mborzecki: that sounds fragile,  well there are agents own tests of course
[11:30] <pedronis> mborzecki: anyway I'm still unsure why we need that poking code
[11:30] <mborzecki> right
[11:31] <mborzecki> perhaps we could land #8625 while i'll try to poke further
[11:31] <mup> PR #8625: wrappers: tweak the  order of restoring, use client timeout in services test teardown <Skip spread> <Created by bboozzoo> <https://github.com/snapcore/snapd/pull/8625>
[11:31] <pedronis> mborzecki: yes, that's fine
[11:38] <zyga> it's finally warm enough to use the standing desk in the colder corner of the office :)
[11:44] <cachio> zyga, hi
[11:44] <zyga> hi
[11:44] <cachio> yesteday I left some errors related to session tool
[11:44] <cachio> I see you created a new PR with fixes
[11:44] <zyga> oh?
[11:44] <zyga> I didn't see those
[11:45] <zyga> which fixes? :)
[11:45] <cachio> zyga, is it related to this? https://paste.ubuntu.com/p/9Svw3Qd7yB/
[11:45] <cachio> https://paste.ubuntu.com/p/k7fyPJch78/
[11:45] <zyga> no
[11:45] <zyga> restore EOFs?
[11:45] <cachio> zyga, ahh, these fixes #8626
[11:45] <mup> PR #8626: tests: fix passing stdin via session-tool <Test Robustness> <Created by zyga> <https://github.com/snapcore/snapd/pull/8626>
[11:46] <zyga> that is related to another PR but not to the pastebin
[11:46] <zyga> is this EOF a one-off or something that always happens/
[11:46] <cachio> zyga, restore EOF and also the other test
[11:46] <cachio> zyga, the test failing on this https://paste.ubuntu.com/p/k7fyPJch78/
[11:48] <cachio> zyga, cannot connect to server -> curl --unix-socket /run/user/12345/snapd-session-agent.socket -D- -X POST -H 'Content-Type: application/json' -d '{"action": "daemon-reload"}' http://localhost/v1/service-control
[11:50] <cachio> zyga, this is also failing on uc20 https://paste.ubuntu.com/p/NgHhWVVPFW/
[11:50] <cachio> zyga, the problem is that as it is failing on restore, it breaks the whole test suite
[11:50] <cachio> zyga, my concern is why it is not failing in google execution
[11:51] <cachio> but fails on edge and beta validation
[12:06] <zyga> re
[12:06] <zyga> cachio: probably random
[12:07] <zyga> cachio: unless it happens each time when executing a specific test
[12:07] <mborzecki> cachio: can you reproduce the problem in tests/main/snap-session-agent-service-control?
[12:07] <cachio> zyga, mborzecki it happens 100% of the time
[12:07] <cachio> mborzecki, yes I can
[12:07] <zyga> cachio: did you collect information about the other session?
[12:08] <zyga> cachio: from the session-tool failure?
[12:08] <zyga> ah
[12:08] <zyga> wait
[12:08] <zyga> sorry,  I misread
[12:08] <zyga> it's the EOF
[12:08] <zyga> so no new data
[12:08] <mborzecki> perhaps the session agent isn't running yet/already/at all
[12:08] <cachio> zyga, no, but I can do it
[12:08] <zyga> it's only interesting iff we can reproduce it in isolation
[12:08] <zyga> cachio: if you can run session-tool test _alone_
[12:09] <cachio> zyga, sure
[12:09] <zyga> ok
[12:10] <cachio> zyga, running
[12:10] <cachio> gime me 5 minutes until it fails
[12:21] <cachio> zyga, I reproduced the error
[12:21] <cachio> is any info which I could provide?
[12:22] <cachio> I have an ssh opened
[12:22] <zyga> what's the last thing that was logged in the spread run?
[12:24] <cachio> 2020-05-08 09:15:58 Error restoring external:ubuntu-core-18-64:tests/main/session-tool:test (external:ubuntu-core-18-64) : + session-tool --restore -u test
[12:24] <cachio> 2020-05-08 09:15:58 Error debugging external:ubuntu-core-18-64:tests/main/session-tool:test (external:ubuntu-core-18-64) : EOF
[12:24] <cachio> 2020-05-08 09:15:58 Restoring external:ubuntu-core-18-64:tests/main/ (external:ubuntu-core-18-64)...
[12:24] <cachio> 2020-05-08 09:15:58 Error restoring external:ubuntu-core-18-64:tests/main/ (external:ubuntu-core-18-64) : EOF
[12:25] <cachio> I could run with the flag to show the output
[12:25] <zyga> cachio: how do you log into the system?
[12:25] <cachio> it is a vm here
[12:25] <zyga> cachio: how does spread log into the system?
[12:25] <cachio> I it does not log
[12:26] <zyga> well, does it execute test by magic?
[12:26] <cachio> I have a spreac which show ssh on real time
[12:26] <zyga> it must connect to the system somehow
[12:26] <cachio> zyga, I have an image to reproduce the error if you want
[12:26] <zyga> no
[12:26] <zyga> I'd like to understand if this is a normal spread run that uses ssh to connect
[12:27] <cachio> zyga, it is a normal spread run
[12:27] <zyga> ok
[12:27] <cachio> using external backend
[12:27] <zyga> is it using ssh to log in as the root user on the device under test?
[12:28] <zyga> or is there any other user used
[12:29] <cachio> zyga, it uses root
[12:29] <cachio> same as in google backend
[12:29] <mup> PR snapd#8627 opened: c/snap-bootstrap: port mount state mocking to the new style on master (2.45) <Created by pedronis> <https://github.com/snapcore/snapd/pull/8627>
[12:29] <zyga> ok
[12:29] <zyga> cachio: can you log into the system
[12:29] <zyga> and run, as root
[12:30] <zyga> session-tool --prepare -u test
[12:30] <zyga> session-tool --restore -u test
[12:32] <cachio> zyga, done
[12:32] <zyga> did it EOF?
[12:32] <cachio> no
[12:32] <zyga> ok
[12:32] <zyga> is there anything in journal from the time of the failure?
[12:34] <cachio> zyga, https://paste.ubuntu.com/p/bfJYGthXwB/
[12:35] <cachio> this is the only I see which seems to be relevant
[12:35] <cachio> I see it many times
[12:35] <zyga> cachio: what specifically?
[12:36] <cachio> May 08 12:15:58 localhost systemd[1]: session-tool-c24e7dda-59d7-4fbd-a871-fc2431b4f5d1.service: Main process exited, code=exited, status=1/FAILURE
[12:36] <zyga> the test is running "false"
[12:37] <zyga> Starting session-tool running false as test...
[12:37] <zyga> to check that exit status is forwarded
[12:37] <cachio> ah
[12:39] <zyga> cachio: K w
[12:39] <zyga> I wonder if this is just a network timeout over ssh
[12:39] <zyga> this test takes a while to run
[12:39] <zyga> maybe tweak it so that there's fewer iterations
[12:39] <zyga> and see if that changes anything
[12:40] <cachio> I am running again but now showing the output
[12:40] <cachio> so I'll see on which line fails
[12:41] <cachio> zyga, perhaps it helps
[12:41] <zyga> yeah, let's try
[12:43] <cachio> it is looping
[12:56] <zyga> cachio: any failures?
[12:56] <cachio> still preparing
[12:56] <zyga> ok
[12:56] <ackk> zyga, am I doing something wrong here https://bugs.launchpad.net/maas/+bug/1876217/comments/5 ?
[12:56] <cachio> I had to re-execute
[12:56] <mup> Bug #1876217: Controllers report Ubuntu Core version in the Snap <MAAS:In Progress by ltrager> <snapd:Incomplete> <https://launchpad.net/bugs/1876217>
[12:56] <cachio> because once it fails, then the session is not correctly cleaned and fails the next run
[12:57] <ackk> zyga, (wrt system-files plug)
[12:57] <cachio> zyga, I see same behaviour
[12:57] <cachio> as it calls itself it goes into an infinite loop
[12:58] <zyga> ackk: yeah
[12:58] <zyga> ackk: probably ..l
[12:58] <zyga> ls -ld /etc/os-release
[12:58] <zyga> is it a symlink?
[12:58] <ackk> zyga, outside or inside the snap?
[12:59] <zyga> well
[12:59] <zyga> outside
[12:59] <zyga> the hostfs one is failing
[12:59] <ackk> /etc/os-release -> ../usr/lib/os-release
[12:59] <zyga> so
[12:59] <ackk> ah I see
[12:59] <ackk> so I have to use the actual file?
[12:59] <zyga> apparmor grants you rights to read /var/lib/snapd/hostfs/etc/os-release
[12:59] <zyga> but you that is a symlink
[12:59] <zyga> so you need /var/lib/snapd/hostfs/usr/lib/os-release
[13:00] <zyga> and you need to understand this in your app
[13:00] <zyga> it kind of sucks because symlinks seen via hostfs are "hard"
[13:00] <zyga> it's good we don't have absolute paths in them
[13:00] <zyga> extend the read section
[13:00] <zyga> to say /var/lib/snapd/hostfs/usr/lib/os-release
[13:00] <zyga> (in addition to the etc line)
[13:00] <zyga> and you should be good
[13:00] <ackk> zyga, so I suspect that won't work for the purpose of being os-agnostic, as on other OSes that might not be a symlink
[13:00] <zyga> yeah
[13:00] <ackk> zyga, so it won't work even if the interface allows both?
[13:00] <zyga> the denial would also tell you
[13:00] <zyga> well
[13:01] <zyga> the symlink can in theory point anywhere
[13:01] <zyga> in practice it either is not a symlink
[13:01] <zyga> or it points to a finite set of files
[13:01] <zyga> which do differ by OS flavour (I recommend booting a fedora workstation)
[13:02] <ackk> zyga, I see, thanks
[13:02] <zyga> good luck! :)
[13:02] <zyga> oh
[13:02] <zyga> standup!!
[13:03] <zyga> ackk: I would prefer a snapctl os-release -like API
[13:03] <zyga> where you could just ask
[13:04] <zyga> and we'd tell you without this mess
[13:04] <ackk> zyga, yeah, that'd be good
[13:05] <ackk> zyga, alternatively the actual /etc/os-release content could be exposed in some other file, like /etc/os-release-host or something
[13:05] <ackk> although that would only work for core* bases
[13:05] <zyga> ackk: that's tricky too
[13:05] <zyga> it's easier to have an API
[13:05] <ackk> zyga, yeah I assumed it would be trickier :)
[13:06] <cachio> zyga, this is the last part of the output
[13:06] <cachio> https://paste.ubuntu.com/p/mysVkdXQ4M/
[13:06] <zyga> "last" :D
[13:07] <zyga> I don't know where to look
[13:07] <zyga> the end looks like just cleanup
[13:08] <cachio> I am logging the full run now
[13:08] <zyga> no
[13:08] <zyga> maybe just look at the log
[13:08] <zyga> I mean
[13:08] <zyga> I just don't know what to look at
[13:08] <zyga> there are 35K lines there
[13:09] <zyga> so a longer log doesn't really help much
[13:09] <zyga> do you see where it failed?
[13:09] <cachio> zyga, https://paste.ubuntu.com/p/2NT6zGG5QB/
[13:09] <cachio> this could help
[13:09] <zyga> what is the status of user-1001.slice?
[13:10] <zyga> cachio: that does not seem to be a fragment of the earlier log
[13:10] <zyga> as I cannot see the EOF messages there
[13:11] <cachio> zyga, no, it is a new run
[13:11] <cachio> zyga, this is from journal https://paste.ubuntu.com/p/hkcJwwpsV3/
[13:12] <zyga> May 08 13:07:54 localhost sshd[19701]: pam_unix(sshd:session): session closed for user test
[13:12] <zyga> this is weird
[13:12] <zyga> how come are we logging in as the test user?
[13:12] <cachio> cheking
[13:13] <zyga> so
[13:14] <zyga> cachio: let me tell you what I think
[13:14] <zyga> session-tool --restore does stop the slice of the test user
[13:14] <zyga> that stops all the services running as the test user
[13:14] <zyga> and kills all the test users's processes
[13:14] <zyga> the only explanation for the EOF
[13:14] <zyga> is that we ssh as the test user
[13:14] <zyga> not as root
[13:14] <zyga> maybe both, dunno
[13:14] <zyga> but that message looks like we just ssh as test
[13:19] <cachio> zyga, I ran with -vv and I see it is running as root
[13:19] <cachio> export USER="root"
[13:20] <zyga> cachio: can you find the moment where we ssh
[13:20] <zyga> or log into the machine and see if we've ssh'd as root or as test/
[13:21] <zyga> pstolowski: maybe move the window to the mac's main screen?
[13:22] <zyga> mvo: if you review the shell fix, please merge it, it failed on mount-protocol-error :/
[13:23] <mvo> zyga: will do
[13:23] <zyga> thanks
[13:28] <zyga> cachio: I know what the problem is
[13:28] <zyga> cachio: look at spread.yaml
[13:28] <zyga> cachio: and look for user: test
[13:28] <zyga> that's the problem
[13:29] <zyga> we do connect as the test user
[13:29] <zyga> no idea why
[13:29] <zyga> it's probably not a good idea
[13:29] <zyga> (username: test)
[13:29] <cachio> zyga, yes
[13:29] <cachio> you are right
[13:30] <cachio> Let me update the test user and try again
[13:30] <zyga> ok
[13:31] <zyga> some tests failed on the store
[13:32] <zyga> updates: got unexpected HTTP status code 408 via POST to "https://api.snapcraft.io/v2/snaps/refresh"
[13:32] <zyga> pedronis: ^ is that expected?
[13:37] <cachio> zyga, https://paste.ubuntu.com/p/fww8t62NbC/
[13:37] <cachio> thanks!
[13:37] <zyga> excellent
[13:37] <zyga> thanks!
[13:37] <cachio> I'll create a PR with the fix
[13:40] <mup> PR snapd#8628 opened: cmd/snap: coldplug auto-import assertions from all removable devices <UC20> <Created by bboozzoo> <https://github.com/snapcore/snapd/pull/8628>
[13:44] <zyga> mvo: if you look at https://tracker.debian.org/pkg/snapd it says there were 8K new commits since release
[13:44] <zyga> https://qa.debian.org/cgi-bin/vcswatch?package=snapd
[13:50] <zyga> mvo: overrides for https://github.com/snapcore/snapd/pull/8626 and https://github.com/snapcore/snapd/pull/8614 would be great
[13:50] <mup> PR #8626: tests: fix passing stdin via session-tool <Test Robustness> <Created by zyga> <https://github.com/snapcore/snapd/pull/8626>
[13:50] <mup> PR #8614: tests: don't create root-owned things in ~test <Test Robustness> <Created by zyga> <https://github.com/snapcore/snapd/pull/8614>
[13:50] <zyga> I could re-trigger but that would just waste time
[13:56] <mvo> zyga: sure, OMW
[13:57] <zyga> thank you
[13:57] <mup> PR snapd#8614 closed: tests: don't create root-owned things in ~test <Test Robustness> <Created by zyga> <Merged by mvo5> <https://github.com/snapcore/snapd/pull/8614>
[13:57] <mup> PR snapd#8626 closed: tests: fix passing stdin via session-tool <Test Robustness> <Created by zyga> <Merged by mvo5> <https://github.com/snapcore/snapd/pull/8626>
[13:58] <zyga> thanks!
[14:04] <mup> PR snapcraft#3111 opened: elf: fix parsing of notes after patchelf mangling <Created by cjp256> <https://github.com/snapcore/snapcraft/pull/3111>
[14:13] <ijohnson> mvo: you might want to cherry-pick https://github.com/snapcore/snapd/pull/8623 onto 2.45 as well, since spread tests on the release/2.45 branch will fail there without that PR
[14:13] <mup> PR #8623: tests/lib/prepare.sh: delete patching of the initrd <UC20> <⚠ Critical> <Created by anonymouse64> <Merged by mvo5> <https://github.com/snapcore/snapd/pull/8623>
[14:13] <ijohnson> mvo: I milestoned the PR as 2.45 for you
[14:17] <mvo> ijohnson: \o/
[14:35] <mup> PR snapd#8629 opened: tests: new test user is used for external backend <Created by sergiocazzolato> <https://github.com/snapcore/snapd/pull/8629>
[14:36] <zyga> cachio: thank you :)
[14:37] <cachio> zyga, yaw
[14:37] <cachio> zyga, thaanks to you for helping with the issue
[14:37] <zyga> :)
[14:37] <zyga> yeah, it was a fun thing to discover :)
[14:39] <pedronis> #8577 needs a 2nd review, maybe ijohnson ?
[14:39] <mup> PR #8577: secboot,cmd/snap-bootstrap: move initramfs-mounts tpm access to secboot <UC20> <Created by cmatsuoka> <https://github.com/snapcore/snapd/pull/8577>
[14:39]  * zyga is sorry for missing the TGIF
[14:39] <zyga> lunch
[14:43] <ijohnson> pedronis: yes it's been in my queue, will take a look
[14:44] <pedronis> I forgot about your 8559
[14:46] <pedronis> mborzecki: so I double checked, the workaround would have poked the real agents, so it cannot have a useful effect
[14:49] <mborzecki> pedronis: i think we should just drop that bit that pokes the agent and see how it will perform in the unit tests accross more runs
[14:49] <mborzecki> let me push that change actually
[14:51] <pedronis> mborzecki: one other thing I notice is that the agent uses SdNotify but I don't see it mocked
[14:52] <mborzecki> pedronis: that code shoudl be noop mostly withouth the env vars
[15:09] <mup> PR snapd#8630 opened: tests: inject snapd from edge into seeds of the image in manual preseed test <Created by stolowski> <https://github.com/snapcore/snapd/pull/8630>
[15:10] <pstolowski> cachio: ^ this should fix the problem, passes for me on 19.10 & 20.04
[15:11] <mborzecki> pedronis: so i dropped that bit that poked the session agent, and it failed now ;P
[15:12] <pedronis> mborzecki: it failed here too, but only with 1.9
[15:12] <zyga> :D
[15:12] <mborzecki> maybe that shitdown is buggy
[15:12] <zyga> is it a bug in go?
[15:12] <mborzecki> shutdown
[15:13] <zyga> shitdown :D
[15:13] <mborzecki> heh, funny typo
[15:13] <zyga> shit happens
[15:13] <zyga> shutdown may hang
[15:13] <pedronis> likely yes, but is not obvious because the shutdown code seems the same
[15:14] <pedronis> so it would be lower level
[15:14] <zyga> maybe it relies on infra beneath
[15:14] <mborzecki> if we only could restart successful unit test jobs
[15:14] <zyga> oh about that
[15:14] <zyga> mvo: what about bit pusher?
[15:14] <mvo> zyga: bit-cacher? still an option, someone needs to review my pr
[15:15] <mborzecki> got to go, my father's name day, bringing him some cake and all
[15:15] <zyga> mborzecki: ^ you have a solution there
[15:15] <zyga> mborzecki: o/
[15:21] <pedronis> mmh
[15:21] <pedronis> there is something odd about the code
[15:21] <pedronis> maybe it relates to the problem
[15:21] <pedronis> or not
[15:28] <seb128> kenvandine, btw, I reported https://bugs.launchpad.net/snapd/+bug/1877610 now
[15:28] <mup> Bug #1877610: The gdb option takes over the user configurations <snapd:New> <https://launchpad.net/bugs/1877610>
[15:29] <seb128> about the 'gdb screws the configuration and breaks your application'
[15:29] <kenvandine> seb128: thanks
[15:29] <cachio> pstolowski, great, thanks
[15:37]  * cachio lunch
[15:45] <mup> PR snapd#8631 opened: image,cmd/snap,tests: add support for store-wide cohort keys <Created by xnox> <https://github.com/snapcore/snapd/pull/8631>
[15:49] <pedronis> maybe I understand what is going on
[16:18] <mup> PR snapd#8624 closed: cmd/snap: fix the order of positional parameters in help output <Simple 😃> <Skip spread> <Created by bboozzoo> <Merged by mvo5> <https://github.com/snapcore/snapd/pull/8624>
[16:18] <mup> PR snapd#8627 closed: c/snap-bootstrap: port mount state mocking to the new style on master (2.45) <Created by pedronis> <Merged by mvo5> <https://github.com/snapcore/snapd/pull/8627>
[16:19] <mup> PR snapd#8629 closed: tests: new test user is used for external backend <Simple 😃> <Created by sergiocazzolato> <Merged by mvo5> <https://github.com/snapcore/snapd/pull/8629>
[16:23] <mup> PR snapd#8632 opened: configcore: only reload journald if systemd is new enough (2.45) <Created by mvo5> <https://github.com/snapcore/snapd/pull/8632>
[16:27] <mup> PR snapd#8633 opened: tests: detect and report root-owned files in /home <Created by zyga> <https://github.com/snapcore/snapd/pull/8633>
[16:29] <zyga> ijohnson: I guess with this ^ we could drop the FIXME comments in the pulseaudio tests
[16:30] <ijohnson> zyga: there's a lot of things there, which one?
[16:30] <pedronis> I think I have a fix for the wrappers mystery, actual the workaround would have also helped (if it talks to the right agent as now in Maciej PR)
[16:30] <zyga> ijohnson: ah sorry, this code says BZZ BROKEN if a test leaks root-owned files in /home
[16:31] <zyga> ijohnson: this is the code that showed the three tests fixed today
[16:31] <pedronis> the issue is that on older go it seems if you call Shutdown before Serve, you get a race and Shutdown will not quite work
[16:31] <zyga> pedronis: nice
[16:31] <zyga> pedronis: what was the problem?
[16:31] <zyga> ah :)
[16:31] <pedronis> in normal usage that shouldnd't happen, but because this is setting up the agent also for a lot of cases that don't use it
[16:31] <pedronis> it happens
[16:31] <zyga> ijohnson: if you scroll to the bottom, you can see how this works
[16:32] <zyga> pedronis: which go version is fixed?
[16:32] <zyga> I was looking at some hangs that I could not explain and perhaps those are related
[16:32] <ijohnson> zyga: no sorry I meant which PR were you referring to, there was like 8 messages from mup above your msg
[16:32] <ijohnson> the "^" was ambiguous
[16:32] <zyga> ijohnson: ah :D
[16:33] <zyga> ijohnson: see here please https://github.com/snapcore/snapd/pull/8633/files#diff-01bb876cbb65b7a336cd99a41c4b97a9R5
[16:33] <mup> PR #8633: tests: detect and report root-owned files in /home <Created by zyga> <https://github.com/snapcore/snapd/pull/8633>
[16:33] <pedronis> zyga: I don't know
[16:33] <pedronis> actually
[16:33] <ijohnson> ack looking now
[16:33] <pedronis> zyga: it's not a change in Shutdown itself
[16:33] <pedronis> so it's not very easy to track
[16:33] <pedronis> and haven't looked deeply
[16:34] <zyga> ijohnson: some things we do in ad-hoc way could move there, like looking for selinux denials or apparmor denials or messages from broken udev rules
[16:34] <zyga> ijohnson: I have one more checker, that shows we don't have processes running as "test" user
[16:35] <ijohnson> this checker looks nice and I like having more checker things like this that make sure tests clean up after themselves
[16:35] <zyga> one thing this buys us
[16:35] <zyga> is perhaps a way to accelerate prepare/restore
[16:35] <zyga> as now it's somewhat heaveyweight because it must be very conservative
[16:36] <zyga> we could maybe make it faster over time
[16:36] <ijohnson> right
[16:37] <ijohnson> It would be good to see what kind of performance that is though, because I wouldn't be surprised if it's not actually that slow
[16:37] <zyga> that fast or that slow?
[16:38] <ijohnson> or rather than not doing automatic cleanup doesn't net us that much total performance
[16:38] <zyga> ah, I see
[16:38] <ijohnson> *performance gain
[16:38] <zyga> I would be happy if we get a clear message from broken tests :)
[16:38] <ijohnson> yes I think that's  the MVP we should be looking for right now :-)
[16:38] <zyga> and not "one of the tests before, guess which please" but "this test right here" :)
[16:39] <zyga> agreed
[16:39] <ijohnson> also zyga what happened to "let's write all the -tool's in python :-P
[16:39] <ijohnson> "
[16:39] <zyga> well
[16:39] <zyga> I did
[16:39] <zyga> believe me
[16:39] <zyga> but it's 20K lines now
[16:39] <ijohnson> haha what
[16:39] <zyga> and I just realized this will never get merged
[16:39] <zyga> I wrote this
[16:39] <zyga> waaaay more complex
[16:39] <zyga> that's testbed-tool
[16:40] <zyga> it supports python and shell plugins
[16:40] <zyga> and has tests
[16:40] <zyga> and man
[16:40] <zyga> this shell script is good enough
[16:40] <ijohnson> ah I see what you mean
[16:40] <ijohnson> indeed
[16:40] <zyga> but
[16:40] <ijohnson> also much easier to review than 20000 lines of python
[16:40] <zyga> since this is not a source-me shell program
[16:40] <zyga> we can :)
[16:40] <zyga> that's the important distinction
[16:41] <pedronis> zyga: seems it's this issue: https://github.com/golang/go/issues/20239 and it's still thee 1.10 but fixed in 1.11
[16:42] <zyga> and we are building with 1.10?
[16:42] <pedronis> we use 1.9 and 1.10
[16:42] <zyga> I see
[16:42] <pedronis> and 1.13
[16:42] <zyga> well, maybe after beta we could discuss having more recent builds
[16:42] <pedronis> in our tests
[16:42] <zyga> maybe for snapd.snap
[16:42] <zyga> dunno
[16:43] <zyga> I guess apart from core / snapd nobody is on go this old anymore
[16:43] <zyga> but after beta
[16:43] <zyga> pedronis: and great find!
[16:44] <zyga> ok I should EOW now
[16:45] <zyga> ijohnson: I'll push a few more checker patches that covert existing random checks into something more organized but probably later tonight only
[16:45] <ijohnson> sure sounds godo
[16:45] <ijohnson> good even
[16:45] <zyga> thanks, have a great weekend everyone :)
[16:45] <zyga> o/
[16:45] <ijohnson> you too zyga o/
[16:47] <mup> PR snapd#8625 closed: wrappers: tweak the  order of restoring, use client timeout in services test teardown <Skip spread> <Created by bboozzoo> <Closed by pedronis> <https://github.com/snapcore/snapd/pull/8625>
[16:53] <mup> PR snapd#8634 opened: usersession/agent,wrappers: fix races between Shutdown and Serve <Created by pedronis> <https://github.com/snapcore/snapd/pull/8634>
[16:53] <pedronis> mvo: ijohnson: ^
[16:54] <ijohnson> mmm pedronis I haven't been following this discussion super close, is it ok if I review a bit later, trying to wrap up the reboot from recover -> run PR to propose first
[16:54] <pedronis> yes
[16:55] <pedronis> mostly I pointed it out because it should help with some of the red we are seeing
[16:55] <ijohnson> ack, sounds good
[17:36] <mvo> pedronis: nice!
[17:59] <cmatsuoka> mvo: is the network fix for recover mode already in 2.45?
[18:12] <cmatsuoka> mvo: well it seems so, it worked now (but for some reason it failed once before)
[18:30] <ijohnson> cmatsuoka: yes it should be in the kernel snaps available on edge/beta
[18:35] <mup> PR snapd#8635 opened: o/devicestate: support doing system action reboots from recover mode <UC20> <Created by anonymouse64> <https://github.com/snapcore/snapd/pull/8635>
[18:44] <cachio> zyga, any idea for this? https://paste.ubuntu.com/p/MRVj2xkQJf/
[18:44] <cachio> it is happening with the new bionic image
[18:45] <cachio> this happens with session tool as I see
[19:06] <pedronis> ijohnson: I added some small comments/questions to that PR
[19:06] <ijohnson> pedronis: ack I'm addressing them now
[19:06] <zyga> cachio: re
[19:07] <zyga> cachio: I think it's the race I mentioned during standup, I'll send a patch that probably fixes it next week
[19:07] <cachio> zyga, nice, thanks
[19:22] <zyga> so, why do our test images have an "ubuntu" useR?
[19:22] <zyga> sorry
[19:22] <zyga> why does spread-created core-16 and core-18 have an ubuntu user?
[19:22] <zyga> it's really weird
[19:22] <zyga> there's a /home/ubuntu/.ssh/authorized keys that's empty
[19:23] <zyga> but /home/ubuntu is owned by root:root
[19:23] <zyga> but /home/ubuntu/.ssh is ubuntu:ubuntu
[19:23] <zyga> do we have a snap changes change for creating users?
[19:30] <zyga> ok, found the bug
[19:37] <cachio> zyga, did you find it?
[19:37] <zyga> no, it was a false trail
[19:37] <zyga> I added a few more checks
[19:37] <zyga> if you know I'm all ears :)
[19:38] <cachio> zyga, I also see this error + session-tool -u test --has-systemd-and-dbus
[19:38] <cachio> grep error: pattern not found, got:
[19:38] <cachio> no user dbus.socket
[19:38] <cachio> is it related?
[19:38] <zyga> no
[19:38] <zyga> where do you see it?
[19:38] <zyga> 16.04?
[19:38] <cachio> bionic
[19:38] <cachio> using the new image
[19:39] <cachio> which I didnt publish yet
[19:39] <cachio> image: test-bionic-1
[19:39] <cachio> for this test google:ubuntu-18.04-64:tests/main/session-tool-support:test
[19:39] <zyga> no-install-recommends probably ate dbus-user-session
[19:40] <cachio> should I install dbus-user-session ?
[19:40] <cachio> as a dependency
[19:41] <zyga> if it was preinstalled in the other image, yeah
[19:41] <zyga> or adjust the test, the test really shows us what the system defaults are
[19:43] <pedronis> now we hit a different snapd-session-agent issue
[19:44] <cachio> zyga, it is not installed in the previous version
[19:45] <zyga> cachio: that's odd, the test really checks if you have dbus.socket as systemctl --user
[19:45] <zyga> cachio: and we seem to have it in a vanilla bionic image
[19:45] <zyga> cachio: otherwise it would not pass
[19:45] <zyga> cachio: perhaps something just removed it
[19:45] <zyga> cachio: if you have a debug shell, maybe it is in apt/dpkg history log
[19:46] <cachio> zyga, I think the problem is that now we have installed dbus-x11 installed and previously we didnt
[19:46] <zyga> pedronis: what did you run into?
[19:46] <zyga> cachio: dbus-x11 is not checked by session-tool-support
[19:46] <zyga> cachio: look at the test and at the filesystem please
[19:46] <pedronis> zyga: test are failing in on core 16 for user services saying there's a start limit issue with the snapd-session-agent
[19:47] <zyga> pedronis: start limit, as in it starts and crashes quicky?
[19:47] <pedronis> maybe
[19:47] <zyga> pedronis: core 16 and core 18 did not ship dbus for user sessions  (20 did)
[19:47] <zyga> but did that test ever pass there?
[19:47] <pedronis> maybe is my code change, but unsure it happens only on core 16
[19:47] <zyga> I see
[19:47] <zyga> oh fun
[19:47] <zyga> Friday :/
[19:48] <pedronis> I'm trying to run the test alone, to see if it's flaky or always fail
[19:57] <pedronis> zyga: alone it works
[19:57] <pedronis> fwiw
[20:01] <pedronis> zyga: do we start sessions for root in tests?
[20:02] <zyga> pedronis: if the test explicitly wants to, as in runs session-tool --prepare (-u root is default) and session-tool ....
[20:02] <zyga> pedronis: IIRC some tests do but most don't
[20:02] <zyga> pedronis: probably only tests that I wrote do that
[20:02] <pedronis> maybe those tests are leaving something behind
[20:02] <zyga> pedronis: try with -repeat 30 or something like this
[20:02] <zyga> it is good at catching weird mistakes
[20:03] <zyga> like test breaking itself
[20:03] <zyga> as well as some chance of catching races
[20:03] <zyga> pedronis: perhaps, which test did you see fail?
[20:03] <zyga> if you dump all the information you have here I may have luck over weekend
[20:04] <pedronis> zyga: https://github.com/snapcore/snapd/pull/8634/checks?check_run_id=657046716
[20:04] <mup> PR #8634: usersession/agent,wrappers: fix races between Shutdown and Serve <Test Robustness> <Created by pedronis> <https://github.com/snapcore/snapd/pull/8634>
[20:05] <zyga> - Make snap "test-snapd-user-service" (unset) available to the system (Post http://0/v1/service-control: read unix @->/run/user/0/snapd-session-agent.socket: read: connection reset by peer)
[20:05] <pedronis> yes, see /0 is root
[20:05] <zyga> yeah
[20:05] <zyga> let me check the test quickly
[20:05] <pedronis> but the tests really set up a session only for test
[20:05] <pedronis> sorry the test
[20:06] <pedronis> I think there's an issue with the tests but also probably a robustness issue in general
[20:06] <zyga> well
[20:06] <zyga> the root user has a session
[20:06] <zyga> because spread logs in over ssh
[20:06] <zyga> perhaps we should mask session services for root
[20:07] <zyga> at least for testing
[20:07] <zyga> until we understand what's going on
[20:07] <zyga> so, without using session-tool, there is a session
[20:07] <zyga> loginctl shows it
[20:07] <zyga> it's the ssh we use to connect
[20:07] <pedronis> maybe yes
[20:07] <zyga> so you get session services there
[20:07] <zyga> including auto-start and other things
[20:07] <zyga> if snapd does something to per-user sessions now
[20:08] <zyga> (I kind of forgot how this works)
[20:08] <zyga> it's plausible it talks to root's session as well
[20:08] <zyga> as a quick way to check
[20:08] <pedronis> it does, the logic is very naive in some ways
[20:08] <pedronis> that's why I said we might have to think a bit
[20:08] <pedronis> but indeed my concern right now is not even the feature
[20:09] <pedronis> but more test robustness
[20:09] <zyga> systemctl --user mask snapd.session-agent.{socket,service}
[20:09] <zyga> right
[20:09] <zyga> that line in core-16 prepare should disable this for the root user
[20:09] <zyga> so no socket, no autostart
[20:09] <zyga> if you want to get sanity quickly, maybe that's the solution
[20:10] <pedronis> not today at this point
[20:10] <zyga> sure,
[20:10] <zyga> it's super late
[20:10] <pedronis> I probably should look a bit more how the world looks when it works
[20:10] <zyga> the one problem with root session
[20:10] <zyga> is that we cannot "wipe the slate" like we do for test
[20:10] <zyga> since we're connected as root
[20:11] <zyga> so I guess we must be more careful with what happens across tests
[20:13] <pedronis> well it seems we get the user agent for root not working somehow
[20:14] <pedronis> ijohnson: I don't think I will get to re-review that PR today, also seems the check I made you add is redundant otoh I don't know how covered all that is
[20:14] <zyga> the connection reset by peer is simply the agent closing?
[20:14] <ijohnson> pedronis: no problem, I think it's reasonably well tested the only thing I don't have more tests for there that I could add would be that doing various combos of some mode to a differnet mode with a different system
[20:14] <ijohnson> pedronis: but as mentioned all such cases currently just fail
[20:15] <ijohnson> err are not supported, and thus fail
[20:15] <pedronis> zyga: it's socket activated
[20:15] <pedronis> or should be
[20:15] <zyga> maybe the agent crashes?
[20:15] <zyga> do we get a journal entry?
[20:15] <zyga> note
[20:15] <zyga> it's funny
[20:15] <pedronis> or mabye the stop on idle is naive
[20:15] <zyga> because journalctl --user is distinct
[20:15] <zyga> so maybe we should look at journalctl --user as ewll
[20:15] <zyga> *well
[20:16] <zyga> the test does this in debug
[20:16] <zyga> + session-tool -u test systemctl --user status snapd.session-agent.service
[20:16] <zyga> we should also do
[20:16] <zyga> + session-tool -u root systemctl --user status snapd.session-agent.service
[20:16] <zyga> or really just
[20:16] <zyga> systemctl --user status
[20:16] <zyga> (without the extra fanfare)
[20:28] <zyga> May 08 20:23:40 may082014-475733 passwd[1570]: password for 'ubuntu' changed by 'root'
[20:28] <zyga> cloud init?
[20:28] <zyga> is our core16 system affected by cloud init data from GCE?
[20:29] <zyga> ha
[20:29] <zyga> it seems so
[20:29] <zyga> :D
[20:29] <zyga> May 08 20:23:40 may082014-475733 passwd[1570]: password for 'ubuntu' changed by 'root'
[20:30] <zyga> that's from cloud-init.service
[20:30] <zyga> pedronis: ^^ lol
[20:30] <zyga> is this expecteD?
[20:38] <zyga> cachio: so, do you know why we get the "ubuntu" user on core16?
[20:38] <zyga> cachio: we do have a weird ubuntu user that belongs to grup "niemeyer"
[20:38] <zyga> this looks very much like an artefact of cloud-init
[20:38] <zyga> but I cannot put my finger on it
[20:39] <zyga> specifically the group is surprising
[20:40] <pedronis> how do we build images, do we take a image, ssh to it, add package and then clean up and snapshot it?
[20:42] <zyga> we build an image and copy some bits but I confirmed that /home/ubuntu does *not* exist at that stage
[20:42] <zyga> it really shows up during boot
[20:42] <zyga> even timestamps support that
[20:42] <zyga> I'm unfamiliar with cloud init detials, looking at some of what the logs say now
[20:44] <zyga> (so the directory and the user get added on first boot)
[20:44] <zyga> and it really seems that it's cloud init
[20:44] <zyga> and I would not really complain
[20:44] <zyga> except that it's buggy
[20:44] <zyga> the owner of /home/ubuntu is root :D
[20:46] <zyga> another fun find
[20:46] <zyga> core 16 ships this file
[20:46] <zyga>  /usr/share/click/frameworks# cat ubuntu-core-15.04-dev1.framework
[20:47] <zyga> (messed up but you get the idea)
[20:47] <zyga> we also have /usr/share/upstart
[21:10] <zyga> https://bugs.launchpad.net/ubuntu/+source/cloud-init/+bug/1639955
[21:10] <mup> Bug #1639955: bad test for snappy systems <cloud-init:Confirmed> <cloud-init (Ubuntu):Confirmed> <https://launchpad.net/bugs/1639955>