[19:30] <harlowja> arg, harmw so close, except need to modify the ufs image and rhel doesn't come with ufs module that seems to work, lol
[19:40] <SnoFox> No topic ...? I didn't realize that was possible.
[19:53] <harlowja> seems like even ubuntu creates a RO ufs module, arg
[20:29] <harmw> harlowja: what do you try to do?
[20:30] <harlowja> need to put some files in /tmp (jsonpatch...), but instance under libvirt doesn't have networking yet, and durn RH and ubuntu ship read-only ufs modules
[20:30] <harmw> why doesn't it have networking?
[20:31] <harlowja> ya, not sure, haven't investigated that much yet :-P
[20:31] <harmw> i can imagine rc.conf lacking some networking settings, though running dhclient should just give you an adress
[20:31] <harlowja> kk, let me try
[20:35] <harlowja> ah, it works
[20:35] <harlowja> sweet
[20:36] <harlowja> arg, of course no wget, lol
[20:43] <harmw> fetch :)
[20:43] <harmw> and perhaps even curl, not sure though
[20:47] <harlowja> hmmm
[20:47] <harlowja> this should let me download jsonpatch and stuff
[20:47] <harlowja> pita, lol
[20:47] <harmw> this beeing fetch?
[20:47] <harlowja> ah, i got wget installed :-P
[20:48] <harmw> in usr/local/sbin perhaps?
[20:48] <harlowja> let me check
[20:48] <harlowja> crazy bsd, lol
[20:48] <harlowja> ;)
[20:49] <harlowja> oh, i guess i can put all the needed files that aren't installed on a fat disk, and then avoid all this stuff, hmmm
[20:50] <harmw> or you just wget/fetch the jsonpatch and cloudinit tarballs and unzip those ;)
[20:50] <harlowja> ya ya :)
[20:51] <harlowja> gonna get this to work :-P
[20:58] <harmw> harlowja: http://paste.openstack.org/show/59693/ I'm seeing this on accasion, while creating/destroying instances
[20:58] <harlowja> ah, good ole neutron
[20:58] <harlowja> :-/
[20:58] <harmw> any ideas where/how to debug this? since it doesn't always happen 
[20:59] <harlowja> hmmm
[20:59] <harlowja> i bet thats your keystone timing out
[20:59] <harmw> lol, i thought the same thing
[20:59] <harlowja> how are u setting up keystone?
[21:00] <harmw> mysql+memcached
[21:00] <harlowja> multi-process?
[21:00] <harlowja> 1 process?
[21:00] <harmw> no, just 1
[21:00] <harmw> keystone-all
[21:00] <harlowja> ya, i bet thats causing it
[21:00] <harmw> hm, is multi-process something new in havana?
[21:01] <harlowja> nah, its been an ongoing fight for that
[21:01] <harmw> because this only started since after the upgrade from grizzly
[21:02] <harlowja> https://www.mail-archive.com/openstack@lists.openstack.org/msg03715.html
[21:02] <harmw> keystone token-get is slow aswell, sometimes it returns in <1s while it can also take an average of 10s
[21:02] <harlowja> hmmm
[21:02] <harmw> (I should mention its a rather old Atom cpu though)
[21:02] <harlowja> do u have a script running that cleans up the token database?
[21:02] <harmw> keystone is running 100%cpu most of the time
[21:02] <harmw> my token table is empty
[21:02] <harlowja> k
[21:03] <harmw> since memcached takes care of that
[21:03] <harlowja> right
[21:03] <harlowja> 100%cpu  not good
[21:03] <harlowja> http://docs.openstack.org/developer/keystone/apache-httpd.html
[21:03] <harlowja> u can try that
[21:04] <harlowja> so much stuff hits keystone that its a good idea to scale it > 1
[21:04] <harmw> true
[21:04] <harlowja> its the first thing to fall over really
[21:04] <harmw> though why didn't this happen with grizzly?
[21:05] <harlowja> unsure
[21:05] <harmw> mind you, I'm running a tiny tiny cloud with just 2 nodes 
[21:05] <harlowja> :-/
[21:05] <harlowja> and its running at 100%
[21:05] <harmw> indeed
[21:05] <harlowja> that seems off :-/
[21:05] <harlowja> bb, food
[21:06] <harlowja> keystone logs useful?
[21:06] <harmw> :)
[21:06] <harmw> ahno
[21:06] <harmw> i had verbose+debug on, nothing interesting
[21:09] <harmw> i think this Atom just sucks bigtime at doing all the PKI stuff, causing it to max at 100
[21:38] <harmw> bedtime now harlowja, let me know how your fbsd stuff ends
[22:31] <harlowja> ends, probably never ends, haha :-P