#ubuntu-s390x 2016-03-31
* ChanServ changed the topic of #ubuntu-s390x to: Ubuntu for IBM LinuxONE and z Systems | Public mailing list at https://launchpad.net/~ubuntu-s390x | Wiki https://wiki.ubuntu.com/S390X
<dave__> Hello
<dave__> Hello, Dimitri
<xnox> dave__, hello =)
<dave__> Thanks for the note about the availability of the beta...I am anxious  to get it installed here.
#ubuntu-s390x 2016-04-01
<jfh> good morning
<andrewc> jfh, good morning!
<andrewc> jfh, sorry to hear that you're having difficulty getting online :-(
<andrewc> jfh, as well as your mail, you could also try asking for assistance on the "canonical-sysadmin" channel on freenode...
<jfh> good morning andrewc - well, I hope I can figure that out soon ... seems to be an issue with the Sign-in through Canonical ... let's see ...
<jfh> good point - will try that, too
<zachman> :D
<cpaelzer> jfh: welcome to the dark side
<jamespage> o/
<cpaelzer> hi jamespage
<cpaelzer> mihajlov: borntraeger: hi I hope you have a good weekend soon, but we would have a question regarding libvirt/kvm/openstack on s390
<cpaelzer> I hope it is one of the former two so we can get a quick solution without asking the OS guys :-)
<cpaelzer> jamespage: here in the channel hit thie bug just a few minutes ago
<cpaelzer> https://bugs.launchpad.net/ubuntu/+source/nova/+bug/1564831
<ubottu> Launchpad bug 1564831 in nova (Ubuntu) "s390x: error booting instance" [Undecided,New]
<borntraeger> cpaelzer, ?
<cpaelzer> mihajlov: borntraeger: and I wondered what/why you don't hit that with z/KVM+OS
<cpaelzer> the title is rather misleading IMHO
<cpaelzer> libvirtd[21610]: this function is not supported by the connection driver:  cannot update guest CPU data for s390x architecture
<cpaelzer> borntraeger: mihajlov: that is closer to where things might start to break
<borntraeger> cpaelzer, I would assume that this is about the "not yet available" cpu model support
<borntraeger> cpaelzer, mihajlov : but there was a workaround in libvirt for that
<jamespage> http://libvirt.org/git/?p=libvirt.git;a=blobdiff;f=src/cpu/cpu_s390.c;h=23a7f9d8d38a00dc9c673d224f797cf8a17aa5d1;hp=f9d7e216aec847df321d7c7d3a050415ee8550fd;hb=59403018893cf2c4f9a6f5145e387cefbd44399a;hpb=b789db36ae1cb5a48986c3b9e3bfb64131367872
<jamespage> looks relevant but we appear to have that in the libvirt version in xenial - just double checking
<jamespage> yah - confirmed in 1.3.1
<cpaelzer> jamespage: hmm - to be sure is that OS against libvirt/KVM ?
<cpaelzer> or containers anywhere in between?
<borntraeger> cpaelzer, jamespage , mihajlov its certainly a message from libvirt
<borntraeger> cpaelzer, jamespage, mihajlov , but I have not seen it here
<cpaelzer> jamespage: could you identify the exact (api) call it made to trigger that?
<jamespage> cpaelzer, borntraeger: actually yes there is a container in the way here
<jamespage> I think that's the cause of the problem...
<jamespage> empty /proc/cpuinfo is not helping I suspect
<cpaelzer> jamespage: do you want to give it a try without containers just with KVM ?
<xnox> jamespage, is missing /proc/cpuinfo an lxc/lxd bug, given that it needs to emulate/whitelist/synthesise it or some such?
<jamespage> xnox, yes I think so
<jamespage> cpaelzer, not just yet
<xnox> jamespage, i guess a manual provider can be mixed into the thing... ?
<jamespage> xnox, figured out how to bind mount the hosts cpuinfo into the container...
<xnox> ^_^
<jamespage> xnox, getting alot of "Failed to allocate directory watch: Too many open files"
<jamespage> xnox, cpaelzer: lack of /proc/cpuinfo is a problem for LXD, but does not appear to be the cause of this...
<jamespage> xnox, cpaelzer: trying a trick to add the host machine to the deployment, but just hit the wall with the 2G root disk size...
<xnox> jamespage, what's your host? you should be able to active e.g. additional drives and add them to the vgroup.
<xnox> jamespage, btw i can reboot s1lp7 and give it to you as well, as an additional resource it should have ~100GB large rootfs.
<jamespage> xnox, my problem is that all of the control plan IP addresses are on the local bridge and not generally accessible...
<jamespage> 2016-04-01 11:21:49 INFO install E: Write error - write (28: No space left on device)
<jamespage> not unexpected...
<cpaelzer> xnox: does d-i in guided partitioning try to create a swap disk as huge as memory?
<cpaelzer> xnox: the disk of james on a 40G memory system had split the available ~41G into 38.x swap and 2G root
<cpaelzer> xnox: s390 is the land of small disks and (sometimes) a lot of memory
<cpaelzer> xnox: there should/could be a cap on the swap size
<xnox> cpaelzer, jamespage: this is a classic d-i/partman bug. there are no caps, just multiples.
<xnox> deactivate swap, remove it, enlarge partition, enlarge rootfs....
<cpaelzer> xnox: already done that
<cpaelzer> I just wanted to avoid the next oen running into it
<cpaelzer> xnox: classic means the bug exists and is open=
<cpaelzer> ?
<xnox> https://bugs.launchpad.net/ubuntu/+source/partman-auto/+bug/1032322
<ubottu> Launchpad bug 1032322 in partman-auto (Ubuntu) "Swap space allocation for large memory systems needs improvement" [Medium,Confirmed]
<cpaelzer> great, thanks
<xnox> first opened in 2012-08-02 but it has been around since forever, and typically reported by installer testers in e.g. qemu vms with like
<xnox> "i gave it 16GB of ram and 8GB rootfs disk"
<xnox> thinking about it.
<xnox> cpaelzer, does it even make sense to have swap on lpar / z/VM?
<cpaelzer> xnox: don't get this started in a public channel
<cpaelzer> nooooo
<cpaelzer> you did it
<cpaelzer> this is like vim and emacs
<xnox> lpars should be big enough, and z/VM can over-commit x2 RAM
<cpaelzer> it can overcommit up to whatever you can accept performance wise and often I've seen 2-3x
<xnox> but on z/VM only, not on LPAR, right?
<cpaelzer> even on kvm it works reasonably most of the time although there could be soom improvements
<cpaelzer> LPAR is only partitioning, no overcommit for memory
<xnox> thinking about it, maybe there should be a safe guard that e.g. swap cannot be more than 10% of total disk space, regardless of the sizing relative to RAM
<cpaelzer> IMHO the Host should swap not the guests
<xnox> or maybe the 200% should be from the smallest of (ram, disk) sizes
<cpaelzer> but there are quite a lot of cases where that alone is not the truth
<cpaelzer> I think I have seen some logic that groups into three categories by ram size
<cpaelzer> ram <2G, try 2*ram
<cpaelzer> else swap = ram size
<cpaelzer> but
<xnox> can one at all hybernate lpar & z/VM? cause on server hibernate on emergency power shut down is a poor mans choice for redundant power.
<cpaelzer> never go over 64G
<cpaelzer> and never go over x% of the disks
<xnox> why 64G? why not 65G? why not 63G?
<cpaelzer> xnox: suspend and resume is implemented
<cpaelzer> arbitrary choice, like the old 2x, why not 1.8x
<xnox> suspend&resume is not hibernate&thaw. E.g. swap is not needed for suspend, as RAM remains powered/active.
<cpaelzer> if science people are involved we could suggest a smooth scaling formular no one would understand :-)
<xnox> 2x -> is reasonable to have a good chance at hibernate, when things have overcommited ram.
<xnox> cause one needs to dump all of ram to swap, to hibernate, plus whatever got overcommited/spilled over to swap.
<cpaelzer> ah you mean to disk
<xnox> yes, hibernate.
<cpaelzer> never cares too much about that, I'd have to check if that works as well
<cpaelzer> hca: ^^ ?
<xnox> is there hibernate on lpar / z/vm -> if not, i'll just remove swap from default recipes full stop, and people can install swapfile package to add swap.
<cpaelzer> wait for hca's answer
<cpaelzer> but then power failure is so boo low end
<cpaelzer> I mean most cpu calculations are doen twice for quantum effects of random particles
<cpaelzer> power failure - pffff
<xnox> i think mainframe deployements have better power failure mode handling than other architectures.
<cpaelzer> well - eventually they have way better handling, but then this is (sadly) one of the things the business/finance people cut costs
<cpaelzer> it works the same without that battery pack, well then ...
<xnox> >_<
<xnox> cpaelzer, reading all the bug reports it's like "high memory system -> too large swap" and "swap not large enough to hibernate"
<xnox> the most reasonable comment is from superm1
<xnox> https://bugs.launchpad.net/ubuntu/+source/partman-auto/+bug/576790
<ubottu> Launchpad bug 576790 in partman-auto (Ubuntu) "Partman should support disabling swap in impractical scenarios" [Undecided,New]
<xnox> e.g. it should be possible to have a flag to essentially "skip swap" and calculate that for "impractical scenarios" e.g. RAM >> root disk (high memory system)
<xnox> with a threshold as to what a high memory system is
<xnox> and be able to preseed that key.
<xnox> imho "high-memory" is anything where RAM >> 10% of total disk space
<xnox> (specifically 10% of the /usr partition)
<xnox> well... no.
 * xnox needs to look at partman-auto to see if it has total disk size numbers available
<cpaelzer> I'm ok with almost any limit, as the hard part is creating the infrastructure not defining the exact ratio/size of the limit
<jamespage> cpaelzer, borntraeger: OK so after looping around and re-deploying with the compute node directly on an LPAR running Ubuntu Xenial, I still see the same problem
<cpaelzer> jamespage: so you now run without Containers just KVM&Openstack
<cpaelzer> ?
<jamespage> cpaelzer, well the control plane bits are still in containers but the hypervisor is not
<cpaelzer> ok
<cpaelzer> didn't xnox already say it worked for him, maybe he has the workaround you need
<xnox> not with latest nova generated libvirt config for our cloud image
<jamespage> xnox, yeah - I suspect this is a break in nova's used of libvirt but not 100% sure yet...
<xnox> so we will need to debug the generated libvirt config i guess.
<xnox> jamespage, is the one you pasted on the bug report accurate?
<xnox> most recent
<jamespage> xnox, yes
<xnox> cool, i'll give it a poke in a few.
<xnox> need to finish a few things up, and have a call, and then will be able to look into it.
<jamespage> xnox, having a punt at setting the cpu-mode flags for nova to host-passthrough
<jamespage> xnox, we do the same for ppc64el
<jamespage> xnox, have you hit this "too many open files" warning/error on s390x?  I think its actually impacting my deployment
<jamespage> I see it on the host and in containers as well..
<mpavone> Hi, I have updated a comment to https://bugs.launchpad.net/ubuntu/+source/nova/+bug/1564831 regarding instance not starting on s390x
<ubottu> Launchpad bug 1564831 in nova (Ubuntu) "s390x: error booting instance" [Undecided,New]
#ubuntu-s390x 2018-03-28
<joelkraehemann> hi all
<joelkraehemann> I try to setup a s390x VM
<joelkraehemann> but I don't get a screen
<jfh> joelkraehemann: are you trying a z/VM guest installation? what is the hw you are running?
<joelkraehemann> jfh: I do it using qemu
<joelkraehemann> on debian
<joelkraehemann> qemu-system-s390x -M s390-ccw-virtio -hda ./ubuntu.qcow -cdrom ./ubuntu-17.10-server-s390x.iso -boot d -m 512
<jfh> joelkraehemann: do you know if qemu on debian is emulating generation EC12 or higher? Because Ubuntu is compiled for EC12 or higher ...
<xnox> joelkraehemann, what is your mainframe level? Ubuntu only supports zEC12
<xnox> joelkraehemann, also you cannot use '-cdrom' option to qemu-system-s390x, it has to be a 'virtio iscsi drive' -> yes it is an iso, but that's how they boot it with qemu-system-s390x.
<xnox> and thus specify that one, as the first hard drive.
<xnox> joelkraehemann, -m 512 might be too small
<xnox> joelkraehemann, you probably need -enable-kvm too
<joelkraehemann> https://github.com/qemu/qemu/blob/master/pc-bios/s390-ccw.img
<joelkraehemann> ^^ I had to copy this file
<jfh> joelkraehemann: that's just one aspect - the other is that EC12 instructions are needed (for Ubuntu)
<joelkraehemann> no idea :/
<jfh> well, you may just try to run Debian - to verify if it works in general ... (can't say out of my head if qemu in Debian comes with EC12 support ...)
<joelkraehemann> does it help install ubuntu on my notebook?
<joelkraehemann> well the same on my laptop s390x won't start
<xnox> joelkraehemann, i believe you need hardware KVM accelration, as in qemu-system-s390x cannot emulate zEC12 instruction set on non zEC12+ mainframe hardware. I do not believe you can run s390x Ubuntu qemu guests, on e.g. Intel laptop.
<xnox> joelkraehemann, what are you trying to do?
<xnox> joelkraehemann, can you simply use Launchpad PPA?
<joelkraehemann> I think PPA doesn't support s390x
<joelkraehemann> and I need a interactive debug session
<borntraeger> xnox, joelkraehemann when you use a qemu from git, this seems to work with code compiled for zEC12
<xnox> joelkraehemann, PPAs do support s390x, you simply need to check the tickbox in the PPA settings page.
<xnox> joelkraehemann, what would you like to debug? can it be scripted, to dump tracebacks in the build log?
<xnox> (it's there, simply not enabled by default - one can enable full set of supported architecutres in any ppa, e.g. arm64 armhf amd64 i386 ppc64el s390x)
<jfh> joelkraehemann: alternatively you may also go for a KVM VM at IBM's LinuxONE Community Cloud at Marist (no-charge)
#ubuntu-s390x 2018-03-29
<joelkraehemann> jfh: but there is no ubuntu on LinuxONE Community Cloud
<pppingme> joelkraehemann https://developer.ibm.com/linuxone/home-ubuntu/
<joelkraehemann> pppingme, thank you
<joelkraehemann> I was able to track down the problem using SLES 12.3
<jfh1> joelkraehemann: SLES is compiled for older s390x generations - that may work ... (sorry missed your question on L1CC)
<joelkraehemann> jfh1: I was able to reproduce
<joelkraehemann> for short, I have 2 versions one working the other not
<joelkraehemann> the ubuntu version is not working
<joelkraehemann> further I just looked at the diff and there is actually a problem with the ubuntu version
<joelkraehemann> http://codepad.org/mKZ5c5dA
<joelkraehemann> ^^ the stack-trace of the crash
<joelkraehemann> http://codepad.org/1pUsFFpl
<joelkraehemann> ^^ I think this diff snipped is the root cause
<xnox> joelkraehemann, if these are ubuntu binaries, you should enable dbgsyms repository and install debug symbols to see more info
<joelkraehemann> I compiled from source :)
<xnox> joelkraehemann, note Desktop is not supported on s390x on Ubuntu; despite just being a test case binary....
<joelkraehemann> well I was able to run the tests with X11Forwarding
<xnox> joelkraehemann, sure, but you want debug symbols for the shared libraries you did not compile. e.g. #36 0x000003fffc294082 in  () at /usr/lib64/libgobject-2.0.so.0
<xnox> etc.
<xnox> and so on, no?
<joelkraehemann> xnox: the crash was introduced while the critical warning
<joelkraehemann> I am thinking of a memory corruption
<joelkraehemann> here, you need to use your brain ;)
<xnox> yeah, annoying
#ubuntu-s390x 2018-03-30
<phita1505> .-.            .-.
<phita1505> /   \          /   \
<phita1505> |   _ \        / _   |
<phita1505> ;  | \ \      / / |  ;
<phita1505> \  \ \ \_.._/ / /  /
<phita1505> '. '.;'    ';,' .'
<phita1505> './ _    _ \.'
<phita1505> .'  a __ a  '.
<phita1505> '--./ _,   \/   ,_ \.--'
<phita1505> ----|   \   /\   /   |----
<phita1505> .--'\   '-'  '-'    /'--.
<phita1505> _>.__  -- _.-  `;
<phita1505> .' _     __/     _/
<phita1505> /    '.,:".-\    /:,
<phita1505> |      \.'   `""`'.\\
<phita1505> '-,.__/  _   .-.  ;|_
<phita1505> /` `|| _/ `\/_  \_|| `\
