[01:17] <PeteBB> Busy channel :)
[04:09] <Kildurin_> any clue what could cause maas web UI to fail login repeatedly?
[09:40] <mup> Bug #1531493 opened: MAAS 1.9.0 fails to detect storage for KVM instances <storage> <MAAS:Confirmed> <https://launchpad.net/bugs/1531493>
[10:34] <mup> Bug #1642916 opened: [1.9] Unable to delete users with reserved IPs <landscape> <MAAS:New> <https://launchpad.net/bugs/1642916>
[11:38] <mup> Bug #1639202 changed: Cannot enlist/commission machines in MAAS 2.1 with usb network adapter <sts> <MAAS:Invalid> <maas-images:New> <initramfs-tools (Ubuntu):New for chiluk> <linux (Ubuntu):Invalid> <https://launchpad.net/bugs/1639202>
[13:35] <mup> Bug #1642960 opened: [wishlist] No info field for nodes marked broken <ui> <MAAS:New> <https://launchpad.net/bugs/1642960>
[13:55] <lynxn0t> hello all, is there a plan to allow maas to deploy nodes with bridged interfaces? Actually it seems that only physical interfaces can be defined in webUI
[14:35] <mup> Bug #1642960 changed: [wishlist] No info field for nodes marked broken <ui> <MAAS:New> <https://launchpad.net/bugs/1642960>
[16:38] <jacekn> has anyone seen very slow tgt traffic beteween host being provisioned and rack controller?
[16:39] <jacekn> TFTP part works just fine with ~3MB/s but then MAAS starts using tgt which drops to 43k
[16:41] <kiko> jacekn, suspect a NIC or switch port config issue?
[16:41] <kiko> do we have ethernet frame droppage? packet loss?
[16:43] <jacekn> kiko: no nothing like that. But that's normal I think, why would packet drop kick in for TGT but not TFTP or any other transfer?
[16:43] <jacekn> kiko: also we deploy boxes on the same physical host (iths KVM instance that I'm deploying) and they work fine
[16:43] <kiko> jacekn, well, how can you tell it's tgt only?
[16:44] <kiko> jacekn, ah, you have more traffic
[16:44] <jacekn> kiko: because I can see bandwidth graphs and console
[16:44] <kiko> my first guess was just based on the fact that a relatively low amount of data moves over dhcp+tftp
[16:44] <jacekn> kiko: yeah. 3MB/s (24Mbit/s) during tftp and then it drops significantly for tgt
[16:44] <kiko> so you could be getting enough to get the dhcp shake to work, and the kernel and initrd over..
[16:45] <kiko> how about an MTU mismatch?
[16:45] <kiko> is all the traffic going out on the same NIC?
[16:45] <jacekn> both ends on 1500
[16:45] <jacekn> yeah same NIC
[16:45] <kiko> do you suspect tgt itself being problematic?
[16:45] <kiko> it's a bit of a crap daemon tbh
[16:45] <kiko> known for leaks, hangs, etc
[16:46] <jacekn> still I'd expect more than 46kbps and it is way faster in other places
[16:46] <kiko> but the recent version has been stable enough -- I think jason had some issues with it eating up memory in OIL occasionally but that's not been mentioned in the past 6 months
[16:46] <kiko> no kidding
[16:46] <kiko> we've never had it be the problem
[16:46] <kiko> the symptom is that enlist/commission takes forever?
[16:47] <jacekn> commissioning was fine it's deployment that's taking forever
[16:48] <jacekn> hmm there are many retransmits, I wonder why
[16:48] <kiko> hmm, that's interesting
[16:48] <kiko> commissioning and deployment use the same (smallish) image over tgt
[16:48] <kiko> but deployment of the OS also pulls the main image over HTTP
[16:48] <kiko> which is usually not that small
[16:48] <kiko> could it be that the actual issue is HTTP, not tgt/iSCSI?
[16:49] <jacekn> I can't see any traffic on http
[16:50] <kiko> that can't be right..
[16:50] <kiko> curtin pulls images over http, I'm pretty sure -- roaksoax, mpontillo, newell?
[16:50] <jacekn> kiko: so are you saing that TGT is not needed for deployment?
[16:50] <mup> Bug #1643001 opened: Moonshot iLO4 'Power HW address' prevent ipmitool from working <MAAS:Incomplete> <https://launchpad.net/bugs/1643001>
[16:51] <kiko> jacekn, tgt is used to provide the ephemeral ubuntu environment, which is what curtin runs inside
[16:51] <kiko> but it's not used to provide the actual image curtin will write out
[16:52] <jacekn> kiko: so slow TGT would cause http traffic to not show up correct?
[16:52] <kiko> well
[16:52] <kiko> slow TGT would first mean that enlistment and commissioning would not work or be very slow
[16:52] <kiko> there is nothing special about the TGT aspect of deployment -- it runs in the same ephemeral environment as enlist/commission
[16:53] <jacekn> kiko: alright I'll keep digging
[16:53] <jacekn> thanks
[16:54] <kiko> jacekn, is there a proxy or firewall issue?
[17:16] <jacekn> kiko: rack controller and boxes deployed have to be in the same subnet no? they certainly are in my case so no FW, just L2 between them
[17:19] <mpontillo> kiko, I'm pretty sure we still use tgt to boot ephemeral and run curtin to deploy
[17:30] <jacekn> mpontillo: kiko: is it possible that commissioning uses trusty? That culd explain it - trusty initrd pulls commissionig image just fine (I've just recommissioned it was super quick). When I try to deploy xenial it's super slow
[17:44] <jhegge> jacekn: you can choose to commission with trusty or xenial
[17:44] <jacekn> nice, I'll try that
[17:45] <jhegge> we've still been adding trusty image for more reliable commissioning and hoping that the need for trusty goes away
[17:45] <jacekn> jhegge: sorry how do you choose that? All I see is "Allow ssh", "Retein network" and "Retain storage" options
[17:46] <jhegge> jacekn: Settings page
[17:46] <brendand> jacekn, it's a global setting, you can't set it for individual nodes
[17:46] <jacekn> brendand: jhegge: hmm only 14.04 on the list, I'm on maas 1.9.4
[17:47] <brendand> jacekn, you want to use xenial?
[17:47] <jhegge> jacekn: you'll have to add the images, not sure how that works on 1.9.4....i'm on 2.0
[17:47] <brendand> jacekn, select it from the Images tab
[17:48] <jacekn> jhegge: brendand: I have xenial images, I can deploy xenial. I wanted to try commissionig with xenial to confirm whether tgt slowness is somethign xenial specific
[17:48] <brendand> jacekn, so you have them but you can't choose to commission with it? hmmm. that might be the case
[17:50] <jhegge> can't recall 1.9 options, that was long ago and i only use the API now...
[17:50] <jacekn> i could just increase juju bootstrap timeout and try again I guess
[17:54] <brendand> no i guess 1.9 can only commission with trusty
[17:54] <brendand> you can however select a later kernel, if that will help
[17:55] <brendand> hwe-x will use the xenial kernel
[18:08] <jacekn> brendand: the problem only happens when I deploy _xenial_, I wanted to commission with xenial to verify OS version is the root cause
[18:59] <roaksoax> jacekn: you cannot commission with Xenial on 1.9
[18:59] <roaksoax> jacekn: 1.9 only supports trusty
[18:59] <roaksoax> jacekn: you can use hwe-x if you need it though, but not xenial
[20:09] <mup> Bug #1642996 opened: [2.1.1] 2.1.0 preseeds with '{{node_disable_pxe_data|escape.shell}}' cause a failed deployment if preseed wasn't upgraded on 2.1.1. <hwcert-server> <MAAS:New> <MAAS 2.1:New> <https://launchpad.net/bugs/1642996>
[20:51] <kiko> jacekn, found anything else out?
[21:09] <mup> Bug #1643057 opened: juju2 with maas 2.1.1 LXD containers get wrong ip addresses <landscape> <juju:New> <MAAS:New> <https://launchpad.net/bugs/1643057>
[22:09] <mup> Bug #1573304 changed: [2.0b3] maas-provision causing connection refused with TFTP back-end <MAAS:Invalid> <https://launchpad.net/bugs/1573304>
[22:09] <mup> Bug #1573400 changed: [2.0b3] maas-dhcp not started after enabling DHCP via CLI <MAAS:Invalid> <https://launchpad.net/bugs/1573400>
[22:09] <mup> Bug #1573768 changed: [2.0b3] Can't change machine name when in 'New' state <MAAS:Invalid> <https://launchpad.net/bugs/1573768>
[22:33] <mup> Bug #1558747 opened: [1.9.1] Deployment for IBM S822LC  8335-GTA  and S812L TN71-BP012 fails to boot local disk following  curtin install <blocks-hwcert-server> <oil> <curtin:Invalid> <MAAS:Confirmed for newell-jensen> <https://launchpad.net/bugs/1558747>
[22:35] <spaok> blake_r: heya