[01:17] Busy channel :) [04:09] any clue what could cause maas web UI to fail login repeatedly? [09:40] Bug #1531493 opened: MAAS 1.9.0 fails to detect storage for KVM instances [10:34] Bug #1642916 opened: [1.9] Unable to delete users with reserved IPs [11:38] Bug #1639202 changed: Cannot enlist/commission machines in MAAS 2.1 with usb network adapter

[13:35] Bug #1642960 opened: [wishlist] No info field for nodes marked broken [13:55] hello all, is there a plan to allow maas to deploy nodes with bridged interfaces? Actually it seems that only physical interfaces can be defined in webUI [14:35] Bug #1642960 changed: [wishlist] No info field for nodes marked broken [16:38] has anyone seen very slow tgt traffic beteween host being provisioned and rack controller? [16:39] TFTP part works just fine with ~3MB/s but then MAAS starts using tgt which drops to 43k [16:41] jacekn, suspect a NIC or switch port config issue? [16:41] do we have ethernet frame droppage? packet loss? [16:43] kiko: no nothing like that. But that's normal I think, why would packet drop kick in for TGT but not TFTP or any other transfer? [16:43] kiko: also we deploy boxes on the same physical host (iths KVM instance that I'm deploying) and they work fine [16:43] jacekn, well, how can you tell it's tgt only? [16:44] jacekn, ah, you have more traffic [16:44] kiko: because I can see bandwidth graphs and console [16:44] my first guess was just based on the fact that a relatively low amount of data moves over dhcp+tftp [16:44] kiko: yeah. 3MB/s (24Mbit/s) during tftp and then it drops significantly for tgt [16:44] so you could be getting enough to get the dhcp shake to work, and the kernel and initrd over.. [16:45] how about an MTU mismatch? [16:45] is all the traffic going out on the same NIC? [16:45] both ends on 1500 [16:45] yeah same NIC [16:45] do you suspect tgt itself being problematic? [16:45] it's a bit of a crap daemon tbh [16:45] known for leaks, hangs, etc [16:46] still I'd expect more than 46kbps and it is way faster in other places [16:46] but the recent version has been stable enough -- I think jason had some issues with it eating up memory in OIL occasionally but that's not been mentioned in the past 6 months [16:46] no kidding [16:46] we've never had it be the problem [16:46] the symptom is that enlist/commission takes forever? [16:47] commissioning was fine it's deployment that's taking forever [16:48] hmm there are many retransmits, I wonder why [16:48] hmm, that's interesting [16:48] commissioning and deployment use the same (smallish) image over tgt [16:48] but deployment of the OS also pulls the main image over HTTP [16:48] which is usually not that small [16:48] could it be that the actual issue is HTTP, not tgt/iSCSI? [16:49] I can't see any traffic on http [16:50] that can't be right.. [16:50] curtin pulls images over http, I'm pretty sure -- roaksoax, mpontillo, newell? [16:50] kiko: so are you saing that TGT is not needed for deployment? [16:50] Bug #1643001 opened: Moonshot iLO4 'Power HW address' prevent ipmitool from working [16:51] jacekn, tgt is used to provide the ephemeral ubuntu environment, which is what curtin runs inside [16:51] but it's not used to provide the actual image curtin will write out [16:52] kiko: so slow TGT would cause http traffic to not show up correct? [16:52] well [16:52] slow TGT would first mean that enlistment and commissioning would not work or be very slow [16:52] there is nothing special about the TGT aspect of deployment -- it runs in the same ephemeral environment as enlist/commission [16:53] kiko: alright I'll keep digging [16:53] thanks [16:54] jacekn, is there a proxy or firewall issue? [17:16] kiko: rack controller and boxes deployed have to be in the same subnet no? they certainly are in my case so no FW, just L2 between them [17:19] kiko, I'm pretty sure we still use tgt to boot ephemeral and run curtin to deploy [17:30] mpontillo: kiko: is it possible that commissioning uses trusty? That culd explain it - trusty initrd pulls commissionig image just fine (I've just recommissioned it was super quick). When I try to deploy xenial it's super slow [17:44] jacekn: you can choose to commission with trusty or xenial [17:44] nice, I'll try that [17:45] we've still been adding trusty image for more reliable commissioning and hoping that the need for trusty goes away [17:45] jhegge: sorry how do you choose that? All I see is "Allow ssh", "Retein network" and "Retain storage" options [17:46] jacekn: Settings page [17:46] jacekn, it's a global setting, you can't set it for individual nodes [17:46] brendand: jhegge: hmm only 14.04 on the list, I'm on maas 1.9.4 [17:47] jacekn, you want to use xenial? [17:47] jacekn: you'll have to add the images, not sure how that works on 1.9.4....i'm on 2.0 [17:47] jacekn, select it from the Images tab [17:48] jhegge: brendand: I have xenial images, I can deploy xenial. I wanted to try commissionig with xenial to confirm whether tgt slowness is somethign xenial specific [17:48] jacekn, so you have them but you can't choose to commission with it? hmmm. that might be the case [17:50] can't recall 1.9 options, that was long ago and i only use the API now... [17:50] i could just increase juju bootstrap timeout and try again I guess [17:54] no i guess 1.9 can only commission with trusty [17:54] you can however select a later kernel, if that will help [17:55] hwe-x will use the xenial kernel === CyberJacob is now known as zz_CyberJacob [18:08] brendand: the problem only happens when I deploy _xenial_, I wanted to commission with xenial to verify OS version is the root cause === zz_CyberJacob is now known as CyberJacob [18:59] jacekn: you cannot commission with Xenial on 1.9 [18:59] jacekn: 1.9 only supports trusty [18:59] jacekn: you can use hwe-x if you need it though, but not xenial [20:09] Bug #1642996 opened: [2.1.1] 2.1.0 preseeds with '{{node_disable_pxe_data|escape.shell}}' cause a failed deployment if preseed wasn't upgraded on 2.1.1. [20:51] jacekn, found anything else out? [21:09] Bug #1643057 opened: juju2 with maas 2.1.1 LXD containers get wrong ip addresses [22:09] Bug #1573304 changed: [2.0b3] maas-provision causing connection refused with TFTP back-end [22:09] Bug #1573400 changed: [2.0b3] maas-dhcp not started after enabling DHCP via CLI [22:09] Bug #1573768 changed: [2.0b3] Can't change machine name when in 'New' state [22:33] Bug #1558747 opened: [1.9.1] Deployment for IBM S822LC 8335-GTA and S812L TN71-BP012 fails to boot local disk following curtin install [22:35] blake_r: heya