[04:40] <bryce__> On Ubuntu Core, how can I find a listing of packages that are available for `snappy install`?
[04:41] <bryce__> Just guessing names hasn't gotten me very far
[04:44] <bryce__> Hmm, reading through the channel description, I might be in the wrong room...sorry! Is there a better room for these questions?
[05:05] <RAOF> bryce__: Not to my knowledge.
[05:06] <bryce__> Gotcha! Is there a syslog package available that you know of?
[05:06] <RAOF> Nah, I'm pretty out of the loop vis-a-vis Ubuntu Core.
[05:06] <RAOF> Sorry. :)
[05:06] <bryce__> cool -- thanks anyway :-)
[05:07] <bryce__> is there a better room or place to ask ubuntu core questions?
[05:07] <bryce__> it seems like an awesome project
[05:08] <RAOF> bryce__: Hm, https://developer.ubuntu.com/en/snappy/ suggests “snappy search syslog” might be relevant to your interests.
[05:09] <RAOF> bryce__: Ah! #snappy on freenode is apparently your winning ticket.
[05:09] <bryce__> woot! thanks
[05:22] <infinity> slangasek: It's not an Ubuntu start page bug, it's because you upgraded firefox and didn't kill/restart it.
[05:22] <infinity> slangasek: The old running firefox has a hissy with the new language bits, ignores your en-us/en preference, and picks the first one in its internal list, which is Arabic.
[05:23] <infinity> slangasek: Old windows and tabs are fine, entire new processes are fine (since they're the new firefox), but new windows as children to the old firefox lose their mind.
[05:23] <infinity> chrisccoulson: ^
[05:24] <StevenK> Given slangasek's language preferences, that make even be an excuse to learn Arabic? :-P
[05:24] <infinity> Every time it happens to me, I'm reminded that I keep meaning to learn Arabic, yes.
[05:24] <infinity> But man, right-to-left web browsing is confusing.
[05:25] <StevenK> I don't tend to see pages in Arabic. It tends to show up for me as some elements not rendering or the interface itself having issues
[05:26] <infinity> Different symptom, same bug.
[05:26] <infinity> In your case, it's the old xulrunner and new xul disagreeing.
[05:26] <infinity> In our case, old firefox and new firefox locales.
[05:27] <infinity> In every case, "restart firefox" is the answer.
[05:27] <infinity> After killing it with fire.
[05:27] <infinity> Or just a reboot, for the "Adam's parents" demographic.
[05:27] <StevenK> Haha
[06:23] <pitti> Good morning
[08:06] <dholbach> good morning
[08:14] <seb128> hey dholbach, happy friday!
[08:15] <dholbach> hi seb128
[08:15] <dholbach> and the same to you!
[08:16] <seb128> thanks :-)
[08:41] <LocutusOfBorg1> hi people!
[08:44] <LocutusOfBorg1> pitti, sorry did you receive the mail? ;)
[08:44] <pitti> hey LocutusOfBorg1
[08:44] <pitti> LocutusOfBorg1: yeah, fished it out of my spam filter this morning; I'll get to it ASAP
[08:48] <LocutusOfBorg1> this is why I asked :)
[09:53] <brainwash> didrocks: hey, thanks for packaging xdg-utils. sadly it looks like something broke. one of the patches was created against another one which was dropped.
[09:53] <brainwash> http://bazaar.launchpad.net/~ubuntu-branches/ubuntu/vivid/xdg-utils/vivid/view/head:/debian/patches/xfce-blanking.diff
[09:55] <brainwash> if you follow the upstream link in the patch header, you will see that the applied upstream patch has a case for "xfce" and for "''". however, the patch in the ubuntu package replaces the "''" one with "xfce".
[09:55] <brainwash> original author is gone for a week or so :/
[09:56] <didrocks> brainwash: I didn't review the first commit as it was already approved by a core dev and discussed, I thought that this was intended
[09:57] <didrocks> brainwash: ok, so, the fix is to have "xfce" and "''"?
[09:58] <brainwash> didrocks: yes
[09:59] <didrocks> brainwash: mind giving a debdiff? I will just sponsor it then :)
[10:00] <brainwash> didrocks: ok, but it will take some time
[10:01] <brainwash> I'm not a packaging expert :)
[10:02] <didrocks> brainwash: well, if there is no hurry as long as it's fixed for today, it's a good practice :)
[10:02] <didrocks> brainwash: tell me if you need any help, sorry for the breakage :)
[10:04] <brainwash> didrocks: alright. luckily we did not SRU it to trusty right away :D
[10:05] <didrocks> yep!
[10:37] <xnox> didrocks: totally agree with you on http://lists.freedesktop.org/archives/systemd-devel/2014-November/025288.html
[10:37] <xnox> didrocks: trying to have sane /etc and sane /usr is impossible.
[10:38] <didrocks> xnox: ah, you climbed that long thread! how long did it take you? :)
[10:38] <didrocks> xnox: I plan to bring back the topic at the hackfest, we need to settle on something
[10:39] <xnox> didrocks: well i'm eager to solve it as well.
[10:39] <xnox> didrocks: imho it's outright wrong that .wants/foo.service -> actually does not matter where it points.
[10:40] <xnox> didrocks: i raised same topic again yesterday on the mailing list "Unwants" and was pointed to your thread.
[10:41] <didrocks> xnox: yeah, I saw your unwants one, and thought "ah, this is actually how we started this thread" ;)
[10:41] <xnox> didrocks: right.
[10:41] <didrocks> xnox: I think you, pitti and I should meet before the hackfest and try to summarize (I have http://pad.ubuntu.com/systemd-enablement-handling and http://pad.ubuntu.com/systemd-usr-etc)
[10:41] <didrocks> so that we can think of all cases (I tried at least) and sum that up
[10:42] <xnox> didrocks: note that i'm only interested in the context of system-image updates, which themself have been compiled / assembled from packages.
[10:42] <xnox> didrocks: which i think is broadly the scope of the problem.
[10:43] <xnox> didrocks: i'm in brussels from thursday evening.
[10:44] <didrocks> xnox: I think that the 2 use cases should converge, basically, it's "we don't want cruft" :)
[10:44] <didrocks> an defaults in /usr
[10:44] <didrocks> and*
[10:44] <didrocks> xnox: I am as well, let's wait for pitti if he can be available?
[10:44] <xnox> yeah
[10:46] <xnox> didrocks: i hate pad and it's insistency on doing 2fa
[10:46] <pitti> didrocks, xnox: what's up?
[10:46] <ogra_> pitti, did you test your systemd upload on a phone ? (stgraber: same question to you for lxc) ... seems we cant boot anymore in vivid ...
[10:46] <xnox> pitti: we want a secret kabahl within the systemd kabahl.
[10:46]  * pitti reads backscroll; sorry, snappy day today..

[10:47] <ogra_> (and systemd as well as lxc look most suspicious on http://people.canonical.com/~ogra/touch-image-stats/74.changes)
[10:48] <pitti> ogra_: no, I didn't test it there; what fails?
[10:48] <pitti> the container? early boot?
[10:48] <ogra_> dunno, i only have an unsupported phone with vivid atm, but it doesnt even seem to get to mount a rootfs here
[10:48] <rsalveti> should show up on the emulator
[10:48] <ogra_> which points to udev perhaps ...
[10:48] <rsalveti> but it seems mako is not even starting lightdm/system-compositor
[10:49] <pitti> yeah, I'm currently building an emulator image
[10:49] <ogra_> (wildly guessing here)
[10:49] <sil2100> rsalveti, ogra_: checking logs on my bricked device, but yeah it's either udev, lightdm, lxc or systemd
[10:49] <sil2100> This image had all of them
[10:49] <rsalveti> :-)
[10:49]  * xnox cannot boot any ubuntu emulator with vivid, ever. can't find rootfs - either on trusty or vivid, with distro goget tools or compiled trunk (from launchpad)
[10:49] <rsalveti> xnox: that was fixed with latest goget-ubuntu-touch
[10:50] <rsalveti> https://launchpad.net/ubuntu/+source/goget-ubuntu-touch/0.14-0ubuntu1
[10:50] <ogra_> sil2100, oh, so yours gets far enough to adb shell in ?
[10:50] <xnox> rsalveti: where is that - in vivid distro, some ppa, bzr launchpad upstream?
[10:50] <xnox> thanks.
[10:50] <rsalveti> bug 1412495
[10:50] <rsalveti> vivid distro
[10:50] <sil2100> ogra_: no... it stays on the ubuntu logo
[10:50] <sil2100> (I mean, manufacturer logo)
[10:51] <sil2100> But I booted up into recovery and try to see what's up
[10:51] <rsalveti> yeah, ubuntu logo would mean lightdm is up
[10:52] <ogra_> sil2100, ok, i see the same here
[10:52]  * ogra_ rolls back to 73
[10:57] <pitti> mount: mounting /dev/mtdblock2 on /root/android//cache failed: Invalid argument
[10:57] <pitti> ogra_: ^ is that something to worry about, or normal?
[10:57] <pitti> it still seems to be stuck in initramfs
[10:58] <ogra_> if it is actually not mounted thats something to worry about indeed
[10:58] <ogra_> thouh /cache is not essential for the boot
[10:58] <rsalveti> for emulator that is fine
[10:58] <rsalveti> afaik
[10:58] <ogra_> k
[10:59] <pitti> ok, thanks
[10:59] <ogra_> i was only thinking about /init running set -e
[10:59] <ogra_> (i wasnt sure if we shouled mount failures)
[10:59] <ogra_> *shield
[11:00] <rsalveti> Begin: Running /scripts/init-bottom ... done.
[11:00] <rsalveti> [    1.418683] init: ureadahead-touch main process (462) terminated with status 5
[11:00] <rsalveti> [    2.180823] init: no-cpu-hotplug main process (610) terminated with status 1
[11:00] <rsalveti> that's all I get here
[11:00] <ogra_> not gettin out of initrd means it is likely neither lightdm or lxc
[11:00] <pitti> right, so most likely udev
[11:00] <pitti> [    1.025651] initrd: starting adbd for USB debugging
[11:00] <pitti> [    1.026668] adbd[283]: segfault at c ip 0804cb29 sp bf9aa4b8 error 4 in adbd[8048000+20000]
[11:00] <rsalveti> pitti: did you remove that other rule I had a while ago?
[11:01] <pitti> meh, no adb, and busybox doesn't really work either (no prompt)
[11:01] <rsalveti> yeah, it seems adb is busted
[11:01] <pitti> rsalveti: no, that's still there
[11:01] <pitti> but adb crashed before too, that's nothing new (it doesn't seem to work that early)
[11:01] <rsalveti> yeah
[11:01] <pitti> rsalveti: it's now in /lib/udev/rules.d/61-persistant-storage-android.rules
[11:02] <ogra_> right, we dont have any adbd binary that works in initrd anymore currently
[11:02] <pitti> rsalveti: which is easier to maintain than an eternal patch
[11:02] <pitti> oh, perhaps that needs to go into initramfs?
[11:02] <ogra_> needs a special build
[11:03] <pitti> rsalveti: if the initramfs need the PARTNAME symlink to boot, that sounds like a very plausible candidate
[11:03] <rsalveti> yeah
[11:04] <pitti> ogra_, rsalveti: ok, I'll see to hack the .img files to include that rule and try again; it's system.img that I want, right?
[11:05] <rsalveti> yeah, we're looking for the partname userdata when mounting things around
[11:05] <rsalveti> so that might indeed be it
[11:06] <rsalveti> yup, the ubuntu one
[11:06] <xnox> didrocks: http://lists.freedesktop.org/archives/systemd-devel/2014-November/025296.html -> are good semantics that I like. Didn't check the pad yet (need to fetch phone for 2fa)
[11:06] <pitti> rsalveti: hm, system.img doesn't look like the one I want
[11:07] <ogra_> no, you want the initrd
[11:07] <pitti> ramdisk.img?
[11:07] <pitti> boot.img?
[11:07]  * pitti looks through them all
[11:08] <pitti> ok, not ramdisk
[11:08] <didrocks> xnox: they are some corner cases with that one (explained later in the thread I guess), but yeah, it's the closest to the best solution
[11:08] <pitti> not boot.img
[11:09] <pitti> ogra_, rsalveti: oh, it's system.img, that contains a system.img, which finally seems to be the fs that I need :)
[11:09] <ogra_> most likely boot.img
[11:09] <ogra_> oh
[11:09] <ogra_> ok
[11:09] <sil2100> Inception
[11:09] <rsalveti> yeah
[11:10] <pitti> meh no -- that boot/ is empty
[11:10] <pitti> where is that darn initramfs?
[11:10] <xnox> didrocks: so reading this thread - i want to start shouting at people "there is no way to upgrade preset" - I hope that comes up in the thread. Ie. if one had "enable sshd.service, disable sshd@.socket" and wants to move to reverse "disable sshd.service, enable sshd@.socket" it cannot be done at upgrade time with preset.
[11:10] <xnox> sure the preset changes, but one cannot run "preset-all" without trumping local user configuration.
[11:11] <xnox> in fact there is no way to specify for the admin, as the "old state" is assumed to be "distro/preset state" when in fact it's a mix of "admin & distro/preset"
[11:11] <pitti> rsalveti: /boot/android-ramdisk.img  on system.img?
[11:12] <pitti> nope, that's really android
[11:12] <rsalveti> pitti: for emulator that is available in your host side
[11:12] <rsalveti> .local/share/ubuntu-emulator/test_x86/ramdisk.img
[11:12] <rsalveti> e.g.
[11:12] <pitti> rsalveti: so boot.img and ramdisk.img don't contain anything, system.img is android, and sdcard.img doesn't contain an ubuntu initramfs; /me confused
[11:13] <pitti> rsalveti: oh, ramdisk.img isn't a qemu img, Isee
[11:13] <pitti> rsalveti: ok, that explains the confusion; thanks!
[11:13] <rsalveti> the previous persistent storage rule is there
[11:13] <rsalveti> ./lib/udev/rules.d/60-persistent-storage.rules
[11:14] <pitti> yeah, I'll just add my new one
[11:14] <pitti> phone, brb
[11:14] <didrocks> xnox: I mentioned it multiple times in the thread :)
[11:14] <didrocks> xnox: just be patient :p
[11:18] <xnox> didrocks: and "copy the unit to /etc/ modify Install section to change wants" does not work -> that is .wants/* symlink in /usr/ is still honoured. thus as far as i can tell there is never a way to unwant.
[11:21] <xnox> didrocks: my alternative idea was to make "systemctl enable/disable/mask/unmask" to write out a user preset file in /etc/systemd/system-preset
[11:21] <xnox> not have any symlinks
[11:21] <xnox> make systemd actually load the preset files and parse them in memory, rather than on the filesystem level.
[11:24] <xnox> "but I would have thought that presets should only be applied on install-time, and not on upgrade." -> and what should you do on upgrade?! +) and imho he is confusing package install/upgrade with OS install/upgrade.
[11:27] <LocutusOfBorg1> dear developers, I would like to have your opinion on bug 1413947
[11:30] <didrocks> xnox: keep reading, I don't think we would go to not having symlinks, but yeah ;)
[11:31] <xnox> didrocks: i would be happy if symlinks are user generated only. And system presets are applied in memory without any symlinks anywhere - neither in /usr/ nor /etc nor /run
[11:32] <didrocks> xnox: that would be a nice solution
[11:32] <didrocks> xnox: we still need the unwants: with this
[11:32] <didrocks> or the /dev/null
[11:32] <xnox> "There should be no need to "stick" some of the services as distro choices are only applied at install time, and never on upgrade." -> because RPM upgrade choices are encoded in systemd upstream.
[11:33] <didrocks> xnox: well, it's more because they don't enable any service by default
[11:33] <didrocks> so you install openssh-server
[11:33] <xnox> didrocks: imho unwants is foo.service.wants/undesired.service -> /dev/null (symlink)
[11:33] <didrocks> xnox: agreed
[11:33] <didrocks> you don't have it started on install or next reboot
[11:33] <xnox> yes, presets are optimized for fedora/rpm pagadim of upgrades.
[11:34] <didrocks> xnox: not sure if you read that yet, but I also talk about the alternatives issue with that (and aliases)
[11:34] <didrocks> xnox: like lightdm, kdm, and so on…
[11:34] <didrocks> and various flavors
[11:38] <rbasak> infinity: could you look at this mysql apt upgrade thing for me please? My test packages are in ppa:mysql-ubuntu/collab. Easy to reproduce - just install mysql-server from vivid, then add the ppa and dist-upgrade.
[11:41] <xnox> didrocks: no matter how one looks at this - current status is optimised for "disabled by default" policy. Since unit file by itself, is never considered to be enabled unless symlinks are generated. If symlinks are in /usr it's a "should not disable" unit, if in /etc -> "it's user enabled, or most-users-enable-this-distro-enabled-it-at-first-installation"
[11:43] <didrocks> xnox: yeah, I guess debian/ubuntu are the first distros who wanted enabled by default
[11:43] <pitti> ogra_, rsalveti: so diffing the previous and current udev debs, this is the only significant change, so I uploaded -5ubuntu2 with that fix
[11:43] <didrocks> for most of the things
[11:44] <ogra_> pitti, cool
[11:44] <ogra_> i'll do a vivid re-build once it is in the archive then
[11:44] <pitti> ogra_, rsalveti: I seem to be unable to munge the initramfs manually though, it still fails; I'll keep debugging, but if that's indeed the problem (likely) then we'll at least have a fixed package ASAP
[11:44] <pitti> and if it's something else still, well it's still a valid and necessary bug fix
[11:44] <xnox> didrocks: despite upstream implicit "enable *" without any preset policy
[11:44] <rsalveti> pitti: yeah, failed for me here as well
[11:44] <rsalveti> but cool, thanks for that
[11:45] <pitti> rsalveti: the rule said it's also necessary for flash-kernel, so maybe something else depended on that during image build, which I didn't catch with the initramfs tampering?
[11:45] <rsalveti> ogra_: guess we need to bump initrd and android after that, right?
[11:45] <ogra_> oh, right
[11:45] <ogra_> hmm
[11:45] <didrocks> xnox: I think they are not in the "we can upgrade defaults to your users" mindset
[11:45] <pitti> was it bumped for -5ubuntu1?
[11:45] <rsalveti> nops
[11:45] <ogra_> we didnt bump it between 73 and 74
[11:46] <pitti> I mean, the upload broke it automatically, it shoudl fix itself automatically?
[11:46] <didrocks> xnox: despite my numerous examples in the thread (and I can add more of what we needed to do in ubuntu), they just don't see that as desirable :p
[11:46] <ogra_> yes, it should
[11:46] <rsalveti> nothing in the initrd changed, so it would indeed still be able to boot
[11:46] <pitti> rsalveti, ogra_: anyway, sorry about that! I'll see to it this afternoon if things still go haywire with this
[11:46] <rsalveti> not sure why it silently fails though
[11:46] <didrocks> xnox: the only point I'm unsure is "if you enable manually an unit that is already enabled by default, and then, we disable the unit as per new distro default, should that unit still be enabled?"
[11:46] <ogra_> probably because the inird ships an olde udevd ?
[11:47] <ogra_> *older
[11:47] <didrocks> xnox: they are prons and cons to both, if the answer is "yes", then, we need a reset command…
[11:47] <didrocks> there*
[11:47] <pitti> rsalveti: unpackign ramdisk.img shows that this was built with the updated udev rules (i. e. droping the PARTNAME rule), but was missing the new 61-persistent-android file
[11:47] <pitti> ogra_: the udev binary didn't change at all, just the PARTNAME rule moved to a separate file
[11:47] <rsalveti> hm, we didn't rebuilt the initrd
[11:47] <ogra_> ah, k
[11:48] <pitti> (it should eventually live in some touch specific package, not udev, but I kept it there for now)
[11:48] <ogra_> rsalveti, does the emulator rebuild it on image creation ?
[11:48] <rsalveti> shouldn't, and I thought I still had that rule here
[11:48] <rsalveti> let me extract it again
[11:48] <pitti> rsalveti: oh, you are  right
[11:49] <rsalveti> so the init logic, until it boots upstart, is the same
[11:49] <xnox> didrocks: the answer is yes. At the moment there is no way to tell systemd "i know better" since .wants symlinks / wants dependencies are honoured from lower level directories even if the unit is outright overriden.
[11:49] <pitti> it already is in the original ramdisk.img, which explains why my hand-editing didn't make a difference
 [    1.418683] init: ureadahead-touch main process (462) terminated with status 5
 [    2.180823] init: no-cpu-hotplug main process (610) terminated with status 1
 that's all I get here
[11:50] <ogra_> these messages are not from /init
[11:50] <pitti> yeah, same
[11:50] <didrocks> xnox: right, there is no way to support that case for now, I'm unsure about the "should", I tend to answer yes as well, but I can accept counter-arguments :)
[11:50] <ogra_> but already from /sbin/init
[11:50] <ogra_> so we are past the initrd here
[11:51] <rsalveti> yeah, already in upstart
[11:51] <pitti> ogra_: oh, we are? ok
[11:51] <pitti> so the PARTNAME thing woudl probably have hit us as soon as we did update the initramfs
[11:51] <rsalveti> yeah
[11:51] <pitti> so good that we found it, but it's not that one
[11:51] <ogra_> right, unlikely to be the actual issue we see here
[11:52] <ogra_> do we put the partnames into fstab at fstab creation ?
[11:53] <pitti> so, ironically it does boot further with systemd; that at least proves that it does get out of the initramfs
[11:53] <ogra_> right
[11:53] <pitti> it fails to start the container, though
[11:53] <ogra_> aha !
[11:53] <pitti>          Starting ifup for lxcbr0...
[11:53] <pitti> [FAILED] Failed to start ifup for lxcbr0.
[11:53] <pitti> well, at least the brige, not sure how crucial that is
[11:53] <ogra_> oops, i thought we had that disabled
[11:53] <pitti> or if it's a red herring
[11:53] <pitti> but it doesn't start lightdm either
[11:54] <ogra_> right, lightdm only starts after the container is up
[11:54] <rsalveti> right
[11:54] <rsalveti> http://paste.ubuntu.com/9833781/ upstart with debug
[11:54] <ogra_> the container initialized the graphics drivers atm
[11:54] <pitti> rsalveti: oh nice, how do you do that on the kernel cmd line? I tried "debug", but that's not it
[11:54] <ogra_> [    2.235346] init: lxc-android-boot state changed from waiting to starting
[11:55] <ogra_> never gets out of the starting state
[11:55] <rsalveti> pitti: yeah, adding --debug
[11:55] <ogra_> hm, buut thats only the boot jobs
[11:56] <pitti> rsalveti: where?
[11:56] <rsalveti> pitti: in the kernel cmdline
[11:56] <pitti> oh, how unusual; thanks
[11:56] <pitti> *nnng* want shell
[11:57] <rsalveti> ubuntu-emulator run test_x86 --memory=1024 --kernel-cmdline="--debug"
[11:57] <rsalveti> yeah
[11:57] <ogra_> [    2.003008] init: lxc-android-config state changed from post-starting to post-start
[11:57] <ogra_> there we go
[11:57] <ogra_> it never gets to "started"
[11:57] <pitti> ogra_: you mean "running"?
[11:57] <ogra_> right
[11:58] <ogra_> (it never emits the started event, so lightdm doesnt fire)
[11:58] <ogra_> (or adb)
[11:58] <ogra_> there should be some lxc log
[11:58] <pitti> I had a regression with user containers, filed as bug 1413927; maybe that somehow affects that too
[11:58] <ogra_> ot at least the upstart log for the job
[11:59] <pitti> ogra_: well, they should be written to /var/log/upstart/ in system.img?
[11:59] <pitti> on the sdcard.img qemu?
[11:59] <ogra_> in the wriable equivalent, yeah
[11:59] <rsalveti> yeah, better trying to find the lxc logs
[11:59] <rsalveti> should be there
[12:03] <pitti> userdata.img is zero bytes, /me looks in sdcard.img
[12:06] <rsalveti> hm, log for lxc android is 0 bytes
[12:07] <rsalveti> cat lxc-monitord.log
[12:07] <rsalveti>    lxc-monitord 1422014624.322 NOTICE   lxc_monitord - lxc_monitord.c:main:416 - pid:585 monitoring lxcpath /var/lib/lxc
[12:07] <rsalveti> nothing useful
[12:07] <pitti> hm, I killed my workstation, I think mount /dev/nbd0 hanging forever
[12:07] <pitti> can't reboot, running a snappy test for mvo..
[12:11] <rsalveti> pitti: ogra_: http://paste.ubuntu.com/9833964/
[12:11] <rsalveti> lxc android with debug
[12:11] <ogra_> the kernel misses the network namespace ?
[12:12] <ogra_> i'm pretty sure we had switched all networking off for the container in the past
[12:13] <pitti> meh, no upstart log for the lxc container start?
[12:13] <ogra_> there should be an lxc-android-config log
[12:13] <pitti> cgmanager: Could not look up /run/cgmanager/fs/freezer/freezer.state
[12:13] <pitti> cgmanager:get_value_main: Pid 506 may not access /run/cgmanager/fs/freezer/freezer.state
[12:13] <pitti> not sure if that's relevant
[12:13] <rsalveti> nops
[12:13] <rsalveti> because what seems to be failing is lxc itself
[12:13] <pitti> ogra_: no, there isn't; that's what I meant
[12:13] <ogra_> only for apps perhaps :)
[12:14] <rsalveti>       lxc-start 1422015012.041 WARN     lxc_confile - confile.c:config_personality:1021 - unsupported personality 'armhf'
[12:14] <rsalveti> also weird, this is the x86 emulator
[12:14] <rsalveti> but well, basically the container failed to start
[12:15] <pitti> Jan 23 10:56:11 ubuntu-phablet kernel: [    5.028352] init: lxc-android-config post-start process (574) terminated with status 1
[12:15] <pitti> ok, so definitively that, yes
[12:17] <pitti> lxc-start[717]: lxc: cgmanager.c: lxc_cgmanager_escape: 329
[12:17] <pitti>  call to cgmanager_move_pid_abs_sync(blkio) failed: A proxy is required
[12:18] <pitti> that's the closest error I can fine
[12:18] <pitti> find
[12:18] <pitti> cgproxy failed due to
[12:19] <pitti> cgproxy:get_value_main: Failed reading string from cgmanager: No child processes
[12:19] <pitti> and cgmanager failed due to the above freezer.state thingy
[12:19] <rsalveti> we should be able to get adb if you change the adb upstart job to not depend on lightdm I guess
[12:19] <pitti> so, no cgmanager -> no cgproxy -> no lxc -> boom
[12:19] <rsalveti> right
[12:20] <ogra_> yeah
[12:20] <ogra_> but cgmanager didnt change
[12:20] <ogra_> nor did the kernel
[12:21] <pitti> ah, maybe the new lxc makes a new type of request to cgmanager which it hadn't before?
[12:24] <ogra_> might be, but then i would have expected stgraber to update them together
[12:26] <rsalveti> ogra_: pitti: yeah, got rev 74, updated lxc, rebooted, boom
[12:27] <ogra_> so i guess we need stgraber
[12:46] <rbasak> xnox: are you planning on merging php5? Or do you want me to take it over again?
[12:53] <xnox> rbasak: there was something rather upstart job about it. let me check.
[13:18] <xnox> didrocks: posted my summary of that thread.
[13:20] <didrocks> xnox: reading
[13:24] <xnox> didrocks: talking in multiple channels is fun -> i love how nobody replied to your alternatives question.
[13:25] <didrocks> xnox: :p yeah… as you can see the thread was going to be not ending, that's why I decided that the hackfest would be better to restart the discussion
[13:25] <xnox> as far as i can tell, rpm distros hate supporting alternatives and the way it's done essentially is that last one wins on packaging level similar to how in debian everything does "conflicts/provides: only-one-service"
[13:25] <didrocks> xnox: lennart answered on the alternatives + preset though
[13:26] <xnox> didrocks: where?
[13:26]  * xnox can't see
[13:26] <didrocks> xnox: let me try to fetch it
[13:27] <xnox> -enochan
[13:30] <rbasak> xnox: did you find out about php5?
[13:31] <didrocks> xnox: starting with http://lists.freedesktop.org/archives/systemd-devel/2014-December/026008.html
[13:47] <xnox> rbasak: yes, please merge php5 there is nothing crazy about it.
[13:53] <rbasak> xnox: OK will do, thanks.
[14:01] <xnox> pitti: did lazr.restfulclient fix the bug you saw with python3 launchpadlib?
[14:01] <xnox> pitti: and do you have more bugs? =)
[14:02] <xnox> 0.13.4-3
[14:05] <pitti> xnox: ah, I didn't get back to that yet, sorry; too much snappy :)
[14:06] <xnox> pitti: snap louder...?!
[14:07]  * xnox is not sure of an appropriate cheer up and a pun at the same time
[14:10] <cyphermox> good morning!
[14:21] <pitti> xnox: hmm, staging doesn't let me log in, seems its 2FA got out of sync
[14:22] <pitti> (I tried the "enter three codes" after "having problems", but it doesn't re-sync)
[14:22] <xnox> pitti: staging has differen 2fa tokens...
[14:22] <pitti> oh, right
[14:22] <xnox> pitti: or rather you should have a different 2fa token for staging.
[14:23] <xnox> pitti: alternative is fake account, or leave 2fa-demo-group
[14:23] <pitti> xnox: I actually do *blush*, just looked at the wrong one in google auth :(
[14:23] <xnox> but i guess you can't leave 2fa-demo-group due to other memberships that include that one.
[14:23] <pitti> so with py2 I saw the "The authorization page:..." message, which doesn't happen with py3
[14:28] <pitti> xnox: so with py3 and unauth'ed, I get http://paste.ubuntu.com/9835355/
[14:28] <pitti> xnox: with py2, this leads me to the LP login page, where I login
[14:28] <pitti> once auth'ed, py2 works, and with py3 I now seem to get a py3 incompat in my own code, so that looks promising!
[14:29] <pitti> xnox: but yes, the original bug is definitively fixed, thanks!
[14:31] <xnox> pitti: keep them comming. i cannot run the full test-suite =/ as i've only ported the critical path to runtime, rather than all test-suite deps.
[14:31] <xnox> all test-suite deps dig deep into lazr.* stack and pretty much a lot of launchpad.
[14:31] <pitti> xnox: right, I'll file a proper bug report soon
[14:32] <pitti> xnox: if this works while already being authenticated, that's a huge leap already
[14:55] <stgraber> ogra_: do you have version 0.35 of cgmanager on that device?
[14:56] <stgraber> ogra_: I've got a meeting in 5min but I'll go grab my nexus4 and see if I can reproduce this
[14:56] <ogra_> stgraber, whatever is current in vivid
[14:57] <ogra_> its today image build
[14:57] <ogra_> *todays
[14:58] <stgraber> oh and my nexus4 is on RTM and with my custom debug kernel, so I guess my test wasn't terribly good since it was older cgmanager and a kernel which doesn't require cgproxy
[15:01] <ogra_> ah, yeah :)
[15:01] <stgraber> now that we have reasonably recent kernels for touch, I should really get the kernel team to include those, it'd be one less daemon running on it and one less source of problems
[15:22] <pitti> stgraber: it reproduces just fine in the emulator, might be easier than bricking your n4
[15:22] <pitti> stgraber: sudo ubuntu-emulator create --channel=ubuntu-touch/devel-proposed dprop
[15:23] <stgraber> pitti: well, I don't use my nexus4 so I don't mind :)
[15:24] <stgraber> and it's faster than the emulator
[15:25] <pitti> stgraber: really?
[15:25] <pitti> stgraber: I find the emulator delightfully fast, and one has a serial console for free :)
[15:31] <stgraber> ogra_: what's the bug# ?
[15:32] <ogra_> i dont think anyone has opened one ... sil2100 ?
[15:34] <slangasek> infinity: ok well it went away on its own regardless, without restarting the browser.
[15:35] <sil2100> ogra_: I didn't hear about any, sadly
[15:36] <stgraber> ok, anyway, I'm about done downloading the rootfs here, so should have a phone with the bug in about 30min
[15:36] <sil2100> stgraber: thanks :)
[15:36] <stgraber> I'll then work with hallyn to figure out what's going on exactly
[15:37] <stgraber> since it looks like it's some odd lxc talking to cgmanager through cgproxy problem
[15:37] <ogra_> stgraber, you will need to hack the adb upstart job to get in
[15:37] <stgraber> (as we're not seeing the problem at all when talking straight to cgmanager on recent kernels)
[15:37] <ogra_> from recovery or so
[15:37] <stgraber> ogra_: yeah, I'm pretty used to doing that :)
[15:37] <ogra_> (it nowadays waits for container and lightdm to be up)
[15:37] <ogra_> ok
[15:37] <stgraber> (since the only times I use that phone is to debug lxc on it)
[15:47] <hallyn> stgraber: do we have an ysort of logs?
[15:47] <stgraber> hallyn: I should have some very soon
[15:55] <pitti> xnox: filed bug 1414055
[15:59] <hallyn> looks like pitti and jodh are running into the same bug (which i think stgraber noticed yesterday?)
[15:59] <stgraber> hallyn: http://paste.ubuntu.com/9836490/
[15:59] <pitti> right, that's what I saw on the emulator
[15:59] <pitti> I suppose upstart is trying to restart it, so it appears several times
[16:00] <pitti> cgproxy fails on that, and lxc fails on not having a cgproxy
[16:00] <jodh> hallyn: https://bugs.launchpad.net/ubuntu/+source/lxc/+bug/1413922 ?
[16:02] <hallyn> jodh: yup
[16:02] <stgraber> so those are two separates bugs. Touch runs privileged containers.
[16:02] <stgraber> so touch is the first priority, then we can look into the other one
[16:03] <stgraber> ogra_: ok, so I'm apparently failing to get adbd to start even when changing its start on condition...
[16:03] <hallyn> stgraber: yeah pitti also reported failing unpriv containers, that's all
[16:04] <pitti> jodh: ah, sounds like my bug 1413927 indeed, dupes?
[16:04] <ogra_> stgraber, well, ssh then ?
[16:04] <pitti> jodh: we must have read stgraber's g+ post at the same time, we are only 5 bug IDs apart :)
[16:04] <stgraber> ogra_: yeah, I'll try to set that up
[16:05] <ogra_> android-gadget-service enable ssh
[16:05] <ogra_> in the terminal app
[16:08] <hallyn> stgraber: so this sounds like touch may be trying to do something not allowed.  Any way of getting the /proc/$$/cgroup of the proxy ?
[16:08] <hallyn> iow it sounds like the security bug fixed by 6a30adc22cc05b5f9e138b8756408c64946452bc was being exploited by touch
[16:16] <pitti> xnox: I'm afraid I do keep them coming :/ -- bug 1414063
[16:17] <xnox> pitti: tah.
[16:21] <stgraber> ogra_: doesn't NM and wifi depend on android?
[16:22] <ogra_> nope
[16:22] <stgraber> did we find a way to get the nexus4 wifi firmware loading to happen from Ubuntu now?
[16:23] <ogra_> oh, N4 might inded be special
[16:23]  * ogra_ hasnt touched an N4 in months 
[16:25] <stgraber> ogra_: 73 was working and 74 is broken, correct?
[16:25] <ogra_> stgraber, yep
[16:26] <stgraber> ok, reflashing on 73 then, will get adbd working from there and then upgrade to 74
[16:26] <ogra_> ++
[16:27] <stgraber> hallyn, pitti, jodh: what's odd about that unprivileged containers failing bug is that I'm running current cgmanager, current lxc and current lxcfs but on trusty and I'm not running into that problem at all
[16:28] <pitti> stgraber: note that I'm currently under systemd; I haven't tried running under upstart yet, but I suppose jodh is?
[16:28] <pitti> stgraber: maybe different cgroupfs layout or so?
[16:28] <pitti> stgraber: btw, once that works again I'm happy to test Lennart's upstream patch for the /dev/urandom segfault
[16:29] <stgraber> pitti: testing on vivid with upstart now
[16:29] <stgraber> (while my phone is flashing)
[16:29] <stgraber> pitti: starts fine under upstart
[16:30] <stgraber> rebooting with systemd now
[16:31] <jodh> stgraber: pitti: actually, running systemd on this box :)
[16:31] <stgraber> good, so I didn't break default vivid :)
[16:32] <bdmurray> pitti: Did you see my question the other day about package hooks on devices?
[16:33] <pitti> stgraber: indeed, under vivid/upstart it starts up fine
[16:34] <pitti> stgraber: so maybe jodh is also running systemd
[16:34] <stgraber> pitti: he just said he did :)
[16:34] <pitti> bdmurray: no, it seems I missed that completely
[16:34] <pitti> stgraber: ah, missed his reply then while I was rebooting
[16:35] <bdmurray> pitti: are package hooks not run because whoopsie wouldn't send all data or for performance reasons?
[16:35] <pitti> bdmurray: for the former case mostly
[16:36] <pitti> bdmurray: also, apport-noui doesn't provide - surprise - an UI, so everything which expects interactivity will just crash
[16:36] <pitti> (but that's ok, we'll just skip those hooks)
[16:36] <bdmurray> pitti: skip how?
[16:36] <pitti> bdmurray: if a hook crashes, apport just goes on and calls the next one, i. e. that's not fatal for the whole bug
[16:37] <bdmurray> pitti: alright, thanks
[16:45] <stgraber> ogra_: starting to wonder if the nexus4 doesn't need android to setup usb or something...
[16:45] <ogra_> shouldnt
[16:45] <ogra_> but adbd needs a password set
[16:45] <ogra_> else it will refuse to start
[16:46] <stgraber> well, I got adbd working fine pre-upgrade including the modified start on condition
[16:46] <stgraber> then I upgraded to 74 and now it won't start
[16:50] <pitti> xnox: ... and bug 1414075; I attach reproducers for all of them
[16:50] <xnox> pitti: thanks a lot!
[16:50] <xnox> will fix.
[16:50] <pitti> xnox: heh, thanks to you! :-)
[16:51] <xnox> it "works" for "some" api calls =)
[16:51] <pitti> xnox: but that's an useful exercise still, I got apport's lplib backend half-ported to py3 now, too
[16:51] <pitti> str vs. bytes yay
[16:52] <pitti> and .next()  vs. __next__(), makes the code really ugly
[16:54]  * pitti waves good bye, enjoy the weekend!
[17:00] <LocutusOfBorg1> bye pitti
[17:00] <LocutusOfBorg1> you too
[17:06] <stgraber> ogra_: ok, so I can't get a shell on the nexus4 and I can't get a shell in the emulator either, so I really don't know how I can help you with that stuff...
[17:07] <hallyn> stgraber: to be clear, it's not hta tyou can't get a shell bc of cgmangaer bug right?
[17:07] <hallyn> (just making sure, else i'll really panick)
[17:08] <stgraber> hallyn: well, it kinda is, cgmanager is preventing the lxc container from starting and it's that lxc container which sets up the display, network, ...
[17:08] <hallyn> ogra_: can you tell me what fails with that bug?  is it just an error msg on login?  failure to login?  apps failing to start?
[17:08] <hallyn> stgraber: but cgproxy would be in the container, and cgproxy is having the problem, so the container must be *starting*
[17:08] <stgraber> hallyn: so basically all we get is a blank screen and no way to connect to it... I can pull logs from recovery or straight from the partition but that's about it
[17:08] <stgraber> hallyn: no, cgproxy runs on the host
[17:08] <ogra_> hallyn, stgraber i dont really work much on the phone atm, i just jumped in to help debugging
[17:08] <stgraber> hallyn: it's a very old kernel
[17:08] <hallyn> still?
[17:09] <stgraber> yep
[17:09] <hallyn> ok well that's messed up - cgproxy on the host should be running in cgroup /
[17:09] <hallyn> so, hm.
[17:09] <stgraber> well, I guess we could force cgproxy to start on a recent kernel and see if the problem happens there too
[17:10] <stgraber> might be simpler than trying to get a working shell on that bloody phone
[17:10] <hallyn> ogra_: how can i deliver a cgmanager package to you to help debugging?  basically i'd like to print out the relevant cgroups at line 583 of cgmanager.c
[17:10] <ogra_> i dont even have a phone with vivid atm
[17:11] <hallyn> ok, let's do as stgraber suggested then
[17:11]  * ogra_ has been fully working on snappy the last two weeks, sorry 
[17:11] <gnuoy`> pitti, zul, I'm planning to look at merging python-flake8 2.1.0-1ubuntu2 -> 2.2.2-1 unless you have any objections?
[17:11] <stgraber> hallyn: I've got an x86 emulator here, so if you get me an i386 build of a test cgmanager, I can install it and get you the upstart log afterwards
[17:12] <stgraber> what I can't get is any kind of interactive debugging...
[17:12] <zul> gnuoy`:  you know what...i dont :)
[17:12] <hallyn> stgraber: ok
[17:15] <stgraber> hallyn: can't reproduce the issue here by just spawning cgproxy and starting a conainer
[17:15] <stgraber> *container
[17:16] <hallyn> stgraber: drat.  well, maybe it's a good thing...
[17:16] <stgraber> well, it shows that cgproxy isn't completely busted :)
[17:17] <hallyn> stgraber: i386 build of cgmanager on vivid?
[17:18] <stgraber> yep
[17:18] <stgraber> I'm doing a test run now with --debug passed to cgmanager and cgproxy and -l debug -o debug to lxc
[17:21] <xnox> cjwatson: are you ssh upstream? and do you have weight in https://bugzilla.mindrot.org/show_bug.cgi?id=1585 ?
[17:21] <stgraber> hallyn: I think I may have found the problem
[17:22] <stgraber> well, we have two problems. 1) lxc's debugging sucks 2) android doesn't work with lxc.autodev
[17:22] <stgraber> I just set lxc.autodev = 0 in the container config and everything works now
[17:23] <stgraber> ogra_: I'll upload a new lxc-android-config now which should fix the issue (it did in the emulator anyway)
[17:23] <hallyn> oh
[17:23] <hallyn> stgraber: well then it may be the same problem that pitti and jodh are seeing
[17:23] <ogra_> stgraber, awesome !
[17:23] <hallyn> time to rewrite lxc
[17:24] <stgraber> hallyn: I doubt it, seeing how I tested lxc.autodev = 1 for all supported Ubuntu releases, both privileged and unprivileged
[17:24] <stgraber> hallyn: for Android, I think the problem is that we're passing it a pre-populated /dev usually and now with autodev, that gets masked
[17:25] <stgraber> so not something I'd fix upstream since our default behavior still makes sense. I just should have though of setting autodev to 0 in lxc-android-config ...
[17:25] <stgraber> ogra_: uploaded. I'll kick a new image as soon as it's published, and test on my nexus4
[17:26] <stgraber> sorry about that, I should really make sure to test on a clean device, rather than my hacked up one :)
[17:27] <hallyn> i suppose we *could* have lxc look for any devices which we don't normally created in the rootfs /dev,
[17:28] <hallyn> and not do autodev if we find anything
[17:28] <hallyn> (or even more work - bind-mount the old /dev to /tmpdev, bind mount devices from there to /a new tmpfs mounted on /dev, and then drop /tmp/dev
[17:32] <cjwatson> xnox: no, and my track record with getting openssh upstream to do config things I want is not noticeably better than others :)
[17:33] <xnox> =))))))
[17:33] <xnox> cjwatson: will you be at fosdem in brussels next week?
[17:33] <cjwatson> xnox: afraid not.  one of these years I will make it
[17:33] <bdmurray> xnox: could you have a look at bug 1412624?
[17:34] <xnox> bdmurray: is this another dupe of the bug that is fixed in vivid?
[17:34] <bdmurray> xnox: I don't know what bug that is.
[17:34] <xnox> bug 1387303
[17:35] <xnox> bdmurray: needs cherry pick of the fix from vivid.
[17:35] <xnox> i guess i should do it. or anyone else can pick it up
[17:35] <xnox> i guess people are not finding that bug because, default search excludes bugs fixed in latest devel series....
[17:36] <bdmurray> xnox: if you do it, then I could review them for you.
[17:37] <xnox> bdmurray: that is true.... =)
[17:37] <xnox> and i should have finished work 7 minutes ago.
[17:37] <bdmurray> I'm sure it would only take another 7 minutes. ;-)
[17:41] <xnox> bdmurray: as in, i can stop working and start volunteering.
[17:42] <bdmurray> xnox: ah-ha, great! I always feel bad adding more things to the SRU queue anyway, since I'm on the team that has to review them.
[17:43] <bdmurray> xnox: not that it stops, I just feel like for everything I add I should take one thing away. ;-)
[17:43] <xnox> =))))
[17:44] <xnox> i thought you used your timedelay two pairs of eyeballs review strategy
[17:44]  * xnox was reading complex bug on waitid failures the other day, only to see that the post was authored by myself year and a half ago
[17:47] <bdmurray> xnox: well, yes that does happen eventually
[17:52] <smoser> infinity: https://bugs.launchpad.net/ubuntu/+source/slof/+bug/1374568
[17:52] <smoser> if you're interested.
[17:53] <stgraber> pitti: btw, I've got a pretty simple network config  which completely hangs systemd at boot time
[17:53] <smoser> that is easy reproduce failure case.
[17:53] <stgraber> hallyn: reproduced the issue here
[17:54] <infinity> smoser: Ta.
[17:55] <stgraber> hallyn: http://paste.ubuntu.com/9837914/
[17:57] <stgraber> hallyn: looking at the cgroup setup, it looks like this may be because /sys/fs/cgroup/systemd/user.slice/user-1000.slice/session-1.scope/ doesn't actually belong to the user
[17:57] <infinity> hallyn: How do you feel about backporting that "qemu-kvm on all arches" change to trusty and utopic as SRUs?
[18:03] <hallyn> infinity: well, as i recall we said months ago we intended to do that
[18:04] <hallyn> infinity: i'll ad dit to my list o fthings to sru
[18:04] <hallyn> infinity: next 1.5 weeks i need to focus on lxc/lxd, then i need to quickly merge qemu 2.2 into v
[18:04] <hallyn> (it's working fine in a ppa)
[18:04] <hallyn> stgraber: that's with systemd-ssyv on the host?
[18:08] <infinity> hallyn: Kay.  I'm also going to backport SLOF from U to T as an SRU and promote the version in updates to main, so you could fix that dep as well while you're in there.
[18:08] <stgraber> hallyn: no, just booted with init=/lib/systemd/systemd
[18:09] <hallyn> stgraber: indeed, the name=systemd cgroup is not owned by the user
[18:10] <hallyn> (when i'm logged in with systemd as pid1)
[18:10] <hallyn> so we need systemd to fix that
[18:10] <stgraber> why did that ever work? was that because of the cgmanager bug?
[18:14] <hallyn> no
[18:14] <hallyn> bc the /user/user-1000.slice/session-whatever used to get created owned by the user
[18:15] <hallyn> i thin kit' sbc cgmanager used to handle name=systemd when passed controller=all and now it does not
[18:15] <hallyn> hm, no, we didn't change that
[18:15] <hallyn> so looks like a systemdlogind change?
[18:15] <hallyn> but systemd on host gives me far worse errors than that
[18:16] <hallyn> i.e. if i chown the name=systemd cgroup so tha ti can start the container, al lhell breaks loose.
[18:18] <stgraber> hallyn: ah?
[18:18] <stgraber> hallyn: here the chown does the trick and the container starts fine
[18:20] <hallyn> stgraber: then run 'top'
[18:20] <hallyn> then exit and look around on the host.  /proc should be the container's proc, shutdown fails;
[18:22] <stgraber> hallyn: ah yeah, that one was reported upstream a few days ago
[18:22] <stgraber> hallyn: looks like an OMG-critical kernel bug to me
[18:22] <hallyn> yeah
[18:22] <hallyn> unless it's something we're doing wrong,
[18:22] <stgraber> well, even if it's us doing it wrong, we're doing it as an unpriv user
[18:23] <hallyn> haha good point
[18:23] <hallyn> is apw still around? :)
[18:23] <stgraber> unpriv user can destroy /proc, sounds kinda bad to me :)
[18:23] <apw> hallyn, vaguly, s'up
[18:23] <hallyn> apw: unpriv user on vivid with systemd can hose the host
[18:23] <hallyn> ("why would anyone be doing that?"  wrong q)
[18:23] <stgraber> hallyn: so looking at /proc on the host, it looks like the /proc mount in the container propagated back to the host
[18:23] <apw> hallyn, well its a good job system isn't the default
[18:24] <hallyn> stgraber: yes
[18:24] <apw> hallyn, how are they able to do that
[18:24] <hallyn> apw: yeah but i assume systemd isn't the cause
[18:24] <apw> systemd is always the cause :)
[18:24] <hallyn> i suspect that making / r-shared on upstart may do the same thing
[18:25]  * stgraber boots that box on his trusted 3.13 kernel to see exactly how screwed we are
[18:25] <stgraber> (might be worth taking this somewhere slightly more private)
[18:25] <apw> stgraber, concur ...
[18:57] <stgraber> sil2100, ogra_: new touch image building
[19:34] <sil2100> stgraber: oh, thanks!
[20:59] <stgraber> sil2100, ogra_: new image works fine here
[21:00] <ogra_> stgraber, great news ! thanks a lot for fixing it
[21:02] <Laney> xnox: fosdem> see you there!
[21:02] <xnox> \o/ good
[21:03] <xnox> Laney: and thank you for testing my desktop notifications in circ irc client
[21:03] <Laney> here to help
[22:08] <Bluefoxicy> shit, that wasn't the OOM sysctl key
[22:10] <Bluefoxicy> Again:  Page cache is essential; if you don't OOM at low page cache, the system will lock up.  That means having 16GB RAM and 15.75GB used is a disk thrashing condition, and the only two ways out of it are OOM or powering the machine off.