=== Aaton is now known as Aaton_off [00:40] This officially sucks .. [00:40] now that I have the MaaS WoL thing more or less worked out, juju is not working... [00:40] juju bootstrap turns the machine up but, there is no IP address to that machine ... and I can't log in to the node either. [00:41] Does anyone know which kickstart is used to load the machine up when it is in the "Ready" state ? I need to add a password to the ubuntu user there. [00:47] I need to take a break from this for a few minutes ... I'll be back in a few [01:06] negronjl: don't do that, juju gives the ubuntu user your public ssh key [01:06] bigjools: don't do which part ... I've done so many things that I am going crazy and losing track. [01:07] Does anyone know which kickstart is used to load the machine up when it is in the "Ready" state ? I need to add a password to the ubuntu user there. [01:09] bigjools: I am having issues with juju now ( got past the WoL issues ) [01:10] bigjools: after the nodes are "Ready" I can't get juju bootstrap to finish [01:10] bigjools: I am able to do "juju bootstrap" or "juju bootstrap --constraints maas-name=quantal" [01:10] negronjl: how do you know it's not finished? [01:11] bigjools: I have a monitor hooked up to the node [01:11] bigjools: it WoL the system, installs everything that needs to be installed ( system, juju, etc. ) [01:13] bigjools: after i end up @ the login promt, I try "juju status" but, I get "no route to host" [01:13] bigjools: i can't even ping the node. [01:15] bigjools: I was wondering how to get inside of the node ( the juju bootstrap node ) to see if I can figure out what is happening [01:17] are you relying on zeroconf for name lookup, or dns on the maas server? [01:19] dns on maas server [01:19] and is it correct? [01:19] it is ... when I "host " it replies correctly [01:20] can you ssh ubuntu@host? [01:20] no ... "no route to " [01:20] it doesn't seem to be a DNS issue as I cannot get to the node by address [01:21] I am working on 192.168.1.0/24 and I have checked every IP ... the node doesn't appear to have an IP [01:22] it doesn't sound like a juju or maas problem [01:22] did the dhcpd assign an ip? [01:22] bigjools: according to the DHCP ... no [01:24] bigjools: The node doesn't appear to be asking for one [01:27] bigjools: That's why I was asking for a way to get in the node so I can ( with the monitor/keyboard ) check things out. [01:30] negronjl: are you running precise? is that what was used to do the demo ? [01:30] lifeless: yes and yes [01:30] ok cool [01:30] uhm [01:31] bigjools: ip allocation will be dhcp on each node right ? [01:31] negronjl: have you got tcpdump / wireshark running on the dhcp server, see what comes in ? [01:32] lifeless: yes but, when I do "juju bootstrap" I don't see the node asking for an address [01:32] ok, even though it powers up ? [01:34] lifeless: yes .. the node powers up and it appears to install correctly ... it then reboots and it leaves me at the login prompt. [01:34] during the install, it must do networking - do you see dhcp requests during that period ? [01:34] I then do "juju status" and I get a "no route to host" [01:35] If you don't, it suggests that the wireshark config is wrong, if you do, but don't later, it tells us either that a) dhcp is being more persistent than usual :P[unlikely] or b) dhcp isn't trying on the reboot. [01:36] negronjl: you could try using arp to locally associate the address used during the install with the node, and see if it responds to that via ping/ssh [01:36] ahh ... I could try that lifeless [01:36] lifeless: give me a minute to try it out [01:36] sure [01:36] also check you see DHCP for the install phase. [01:37] lifeless: i will [01:39] I have had it up to HERE with ubuntu problems already today [01:40] welcome back [01:40] bigjools: amen to that :) [01:40] three reboots [01:40] and it's 11:40am [01:54] just so we are all on the same page .. I am monitoring DHCP traffic at the maas server with tcpdump -i eth0 -n port 67 and port 68 [01:55] I just did a "juju bootstrap" ( leaving maas to pick a node for me ) [01:55] I see the DHCP requests from the node and the responses from the DHCP server === wgrant_ is now known as wgrant [01:58] hah .. i found something [01:59] i found that the IP that the node got differs from what's expected. [02:00] ie: the node's name is natty so, when i do "host natty" I get "192.168.1.103" but, when I look at the tcpdump output, I see that the address is 192.168.1.168 [02:00] I cannot ping 192.168.1.103 but, I CAN ping 192.168.1.168 [02:01] I CANNOT ssh into 192.168.1.168 though ... it asks me for password but, I don't have one [02:01] is cobbler set to manage DNS and DHCP? [02:02] it is supposed to set the same IP in both [02:02] if it doesn't then things will break like this [02:02] cobbler is managing DNS but, not DHCP [02:02] there's your problem [02:02] there's one of my problems ... [02:02] :) [02:02] optimist :P [02:02] ha [02:02] lol [02:02] why isn't cobbler managing DHCP ? [02:02] this is the way the setup was sent to me [02:03] but, even if I change that ... i still have the problem that I cannot log int [02:03] s/int/in [02:03] well [02:03] so cloud-init may not have run [02:03] because its not got the right ip now [02:03] ok ... let's try to fix the DNS/DHCP issue first i guess.... [02:04] definitely [02:04] I have a dd-wrt router in the mix acting as DHCP ... let me pull that out of the equation...then I will need a bit of help fixing the DHCP in the maas server ... [02:04] brb ...breaking things :) [02:04] /etc/cobbler/settings [02:05] manage_dhcp: 1 [02:05] negronjl: the whole thing should be its own standalone network, with (I believe) a local archive mirror and everything [02:05] bigjools: checking /etc/cobbler/settings now [02:05] lifeless: i have the local mirror thing working. [02:05] negronjl: or at worst, with the maas server itself attached on one side to the maas cluster LAN,a nd on the other to your network. [02:06] again, AIUI, second hand knowledge ;) [02:06] oh wow ... according to /etc/cobbler/settings .... the maas server is managing both dns and dhcp ( manage_dhcp: 1 manage_dns: 1 ) [02:06] this doesn't seem right .... [02:09] check that your node has got an entry in the dhcpd config [02:09] is it using dnsmasq or isc? [02:10] it should map a mac to an IP [02:15] bigjools: ok .. let me check ... [02:15] let's start with this ... the maas server should have a static ip address [02:15] right now it doesn't [02:15] it is assumes that it will be 192.168.1.2 ( I am working with a 192.168.1.0/24 network ) [02:16] I should probably fix the maas server first [02:16] yup [02:16] if its dual attached, it can be dhcp on the external side [02:17] but the cluster side has to be pretty fixed ;) [02:18] lifeless: the maas server is a laptop with a broken screen :/ [02:20] and one interface only [02:20] no wifi? or wifi only ? [02:21] negronjl: it arrived broken? [02:22] lifeless: it did although I have a vague recollection of a conversation with mrussell ( Mark Russell ) that it may have been broken before shipment to my house. [02:22] maas server has a static ip now [02:22] now checking dhcpd ( /etc/dhcp3 ) configuration [02:23] cry for help sent [02:24] bigjools: in MaaS ... what handles the DHCP server ( I see maas-dhcp package installed but, don't really know where the config resides ) [02:24] its lacking all technical info about where we are at; wrong audience for that I think. [02:24] lifeless: thx ... well said .. cry for help [02:24] negronjl: cobbler does everything [02:24] maas is just driving it [02:25] bigjools: ok .. so earlier you had mentioned for me to make sure that I had something in dhcp config ... where should I be looking ? [02:25] dhcp config is written by cobbler [02:25] so we need to see if it wrote it right [02:25] so, just looking for that MAC to IP mapping so the node gets the IP address we expect [02:28] bigjools: I'll start a node to see how it behaves now [02:28] ok [02:30] hah ... i have to laugh because I don't want to cry ... the maas server just froze ... rebooting it [02:30] you are seriously unlucky [02:34] bigjools: what handles dhcp and dns here ... [02:34] negronjl: it's in settings [02:35] I'm really not trying to ask stupid questions ... I am just trying not to assume anything at this point [02:35] so you can tell cobbler to drive isc dhcp or dnsmasq [02:35] it's dnsmasq out the box IIRC [02:36] it appears to be dnsmasq [02:38] bigjools: where is the config for dnsmasq ? [02:39] /etc/dnsmasq.conf? [02:41] i may have to rebuild this environment .. ( maas-flush and all ). I think that having two dns/dhcp servers in the network ( cobbler and the router ) may have screwed the config on the nodes [02:41] very likely [02:44] rebuilding now ... it'll take a few minutes [02:54] bigjools: now pxe booting is not working .... TFTP open timeout [02:54] yay [02:54] bigjools: i have cobbler using the system's tftp ... should I change it so cobbler uses it's own instead ? [02:55] system's tfpt => tfptd-hpa ( /etc/init.d/tftpd-hpa ) [02:55] that should be ok [02:55] the system's ? it was working before so it seems to be a config issue somewhere in cobbler [02:55] you must have been use pxe on the router before [02:56] apparently [02:56] using* [02:56] I don't know what's best here, try flipping and see [02:56] ok [03:02] nope ... I think I was getting further with the router ... [03:02] maybe try having the router pass the pxe and dhcp and have the maas server handle dns [03:03] bigjools: ^^ [03:03] I don't know what's best, sorry [03:04] you need roaksoax perhaps [03:04] if he is up [03:05] negronjl: its what, 8pm for you ? [03:05] lifeless: it is [03:06] negronjl: ok; I'm going to break for a late lunch in a minute [03:06] negronjl: whats the network layout you have in use? [03:06] lifeless: flat network ( 192.168.1.0/24 ) [03:06] negronjl: e.g. nodes + maas server on one switch, where does your router come into it, and so on. [03:07] lifeless: router is 192.168.1.1 ( gw and dhcp ) [03:07] lifeless: 192.168.1.2 -> maas-server [03:07] negronjl: you haven't turned dhcp off ? [03:07] on the router that is [03:07] are all the necessary daemons running on the maas box? [03:07] lifeless: i did turn dhcp off on the router but, pxe stopped working [03:07] bigjools: they are [03:07] negronjl: ok, leave it off please :) [03:08] negronjl: you won't have your router at Structure :> [03:08] dnsmasq, dhcpd, bind, tftpd [03:08] and tgtd [03:08] lifeless: the router was shipped here with the rest of the stuff so, yes .. i could [03:08] bigjools: no dhcpd nor bind ... dnmasq instead [03:08] negronjl: oh, it was? See, bad assumptions R us :) [03:08] bigjools: tftpd-hpa for tftp [03:09] ok [03:09] so we need to work out why the node cannot see a pxe server (ie dhcpd) [03:09] firewalled? [03:09] bigjools: straight cable [03:09] negronjl: are the nodes plugged into the router, or a switch/hub ? [03:09] lifeless: switch [03:09] here is the setup [03:10] one switch where all of the nodes and maas-server are connected [03:10] maas server is a laptop running the above mentioned daemons [03:10] that is all [03:10] the router is plugged into the switch? [03:10] 9 nodes ( HP microservers ) one switch and one laptop. [03:10] can you get a pxe test client? [03:11] when I was using the router ... it was also plugged in the switch to provide dhcp/dns [03:11] ok, cool. [03:11] so, default switch environment is to forward everything, pxe should work fine there. [03:11] negronjl: got wireshark up? can you get the DCHP OFFER contents, that should have the pxe metadata in it. [03:11] lifeless: pxe was working fine with the router but, I believe that the nodes were getting their IPs from the DHCP but not updating the DNS. [03:12] I believe the DHCP was on the router but, DNS was on router AND maas server ... [03:12] not sure but, I really believe that there was a conflict between the router's dns/dhcp and the maas-server's dns/dhcp [03:13] sure, so we now have dhcp and dns both set to managed in cobbler [03:13] and wol is firing up the node [03:13] and its failing to PXE boot ? [03:13] execuse me, does anyone know the MaaSslave's default username and password ? [03:14] rick1: your ssh key should be copied into the ubuntu user for you by cloud-init. [03:14] lifeless: with all the wires running across my dining room table, I may have found something ... let me try again .. give me just a second [03:14] rick1: If I remember correctly. [03:14] lifeless: so, I can just login to it? [03:14] ssh yes [03:15] it won't have a password at all [03:15] lifeless: Thanks, I'll try [03:15] lifeless: pxe booting is now working off of the maas server ( no router involved ) [03:15] I am going to continue deploying to see if things are better now [03:15] negronjl: what was wrong? [03:16] negronjl: that machine has been broken for ages and that's the same one used at the ODS demo [03:16] rick1: the user is "ubuntu" [03:16] network loop [03:16] bigjools: oh, Okay [03:16] negronjl: use the dd-wrt router for DNS/DHCP [03:16] negronjl: don't use maas [03:17] roaksoax: ohhai!; I'll let you steer :) [03:17] roaksoax: i see now that by using the maas-server for dns/dhcp i can't spoof the archives ... [03:17] roaksoax: I'll switch it back ... give me just a second [03:17] roaksoax: do I need to turn dns/dhcp off in cobbler now that the router is back in the picture ?? [03:18] negronjl: yeah don't use maas-server as DHCP/DNS because you'll have to manually assign what IP addresses the clients will get throught DHCP [03:18] negronjl: yes you do [03:18] negronjl: the dd-wrt should already e confogured for DNS/DHCP so you shouldn't really have any issues there [03:18] lifeless: It shows Connection reset by peer, It must be something wrong with the PXE installation.... always shows commissioning on the WebUI [03:19] roaksoax: just to be sure .... manage_tftpd and manage_dns should both be set to 0 ( and not 1 ) on /etc/cobbler/settings ?? [03:19] negronjl: manage_dns manage_dhcp [03:19] negronjl: we *do* manage_tfptd [03:19] roaksoax: sorry yes .. those two [03:20] roaksoax: manage_dns and manage_dhcp [03:20] roaksoax: they are both set to 1 on /etc/cobbler/settings [03:21] negronjl: I can't recall (i'm zombie already), but you should be able to dpkg-reconfigure maas-dhcp and disable DNS/DHCP [03:21] roaksoax: negronjl: I'm going to take that lunch break. Will check in in ~40 [03:21] lifeless: thx [03:21] roaksoax: I'll do that .. [03:21] lifeless: provecho :) [03:22] roaksoax: did dpkg-reconfigure but, I don't get the option to disable it. [03:26] roaksoax: trying again ( i disabled it directly in /etc/cobbler/settings ) at this point, i am just trying things out [03:26] * roaksoax checks [03:26] negronjl: try purging maas-dhcp [03:27] roaksoax: will do [03:54] roaksoax: not sure if you are still around but, I have a question.... [03:54] negronjl: shoot [03:54] roaksoax: After the node is in "Ready" state ... can I then run "juju bootstrap" ? [03:55] roaksoax: or is there anything that I have to do between Ready and juju bootstrap [03:55] negronjl: ready state are ready to be used [03:55] roaksoax: so, juju bootstrap would be ok at that point ? [03:56] negronjl: yes, [03:57] roaksoax: thx ... I think I should be good for now ... thank you VERY much for all you patience and help during this. [03:57] no worries [03:57] negronjl: we've been there [03:57] :) [04:04] hows it looking guys? [04:09] Still stuck on "Commissioning" state [04:09] rick1: I don't know the states all *that* well, but I think that means 'trying to turn it on and install it' [04:10] lifeless: yes [04:12] rick1: are you using wake on lan or a power board or something? or just reaching down and pushing the power button ? [04:12] after maas slave PXE booting, scrolling of many screen of messages, stay on "cloud-init-nonet killed by TERM signal" for a while, then jump to login screen [04:13] lifeless: yes, I set to wol [04:13] bigjools: / roaksoax: ^ any thoughts? [04:13] no idea why that TERM happens, I think we went through all the usual suspects [04:14] so I'd have to defer to roaksoax or smoser who have more knowledge of cloud-init than I [04:16] I have had so many issues today that I may be able to help with this. [04:16] rick1: would you pastebin the contents of /etc/maas/maas_local_settings.py please ? [04:17] ok, wait a moment [04:17] rick1: I am looking for the value of: DEFAULT_MAAS_URL [04:18] rick1: In my case ( out of many ), cloud-init was not working correctly because I had an issue where DEFAULT_MAAS_URL was not properly set to the IP address ( not the hostname ) of the maas-server [04:18] negronjl: DEFAULT_MAAS_URL = "http://10.20.0.1/" [04:19] rick1: also make sure that the time on your nodes ( all of them ) as well as the maas-server is the same .. weird .. I know but, it helps [04:19] oauth FTL [04:23] bigjools: do I have to upload my SSH key to the maas server before i do anything with it ? [04:23] negronjl: oh, you are right, the time is different [04:23] bigjools: juju status ( after a successful juju bootstrap ) gives me Invalid SSH key [04:23] negronjl: no, unless you plan on starting nodes manually in the UI [04:23] rick1: put them all on the same time [04:24] bigjools: I get Invalid SSH key .... [04:24] negronjl: yeah that happens if it's still booting before its run the preseed that adds the user data [04:24] if the console is at the login prompt then it's not booted properly [04:25] wrong image or failure to get metadata [04:25] bigjools: hmmm... how do I avoid that ? Did I do something wrong here ? [04:26] negronjl: the clock problem? [04:26] perhaps [04:26] bigjools: nope ... all nodes are time synced [04:26] bigjools: maas-server as well [04:27] is it at login prompt? [04:27] bigjools: yup .. .and I can't log in to figure what may be wrong ... [04:27] ok is there anything in maas-server syslog from the node? [04:29] Test WP failed, assume Write Enabled [04:29] Asking for cache data failed [04:29] Assuming drive cache: write through [04:29] negronjl:now time is synced, but still the same problem. [04:29] Test WP failed ..... [04:30] rick1: .... 10.20.0.1 is your maas server right ? [04:30] negronjl: yes [04:30] negronjl: is juju bootstrap running for you now ? [04:31] lifeless: it does but, I cannot juju status ( invalid ssh key ) [04:31] check your environments config is right and so forth ? [04:31] rick1: dpkg-reconfigure maas ... make sure your maas server ip is correct ... [04:31] rick1: sudo cobbler sync [04:31] rick1: try again [04:32] negronjl: watch the console during boot, see if anything else looks bad [04:32] negronjl: ok [04:39] negronjl: still the same. and I found a bug report about this. https://bugs.launchpad.net/ubuntu/+source/maas/+bug/992075 [04:39] Ubuntu bug 992075 in maas (Ubuntu) "Commissioning status persists with cloud-init 0.6.3-0ubuntu1" [Undecided,Confirmed] [04:40] rick1: for me, it was a simple case of /etc/maas/maas_local_settings.py having the wrong DEFAULT_MAAS_URL [04:40] * bigjools adds that to the FAQ [04:40] negronjl: :), I'm go out and take a break, thank you VERY much [04:40] rick1: no worries .. just paying it forward ... I have been helped a LOT [04:43] please consider amending or adding to the https://answers.launchpad.net/maas/+faqs [05:06] bigjools, lifeless: good news: I am now able to ssh ubuntu@ [05:06] bigjools, lifeless: bad news: the dhcp address associated with the name does NOT match what MaaS thinks it should be [05:07] bigjools, lifeless: MaaS and Juju think that my node is 192.168.1.101 but, my node got an address of 192.168.1.127 ... I have to check the DHCP server to see what may be going on but, I will have to do that tomorrow ... I am calling it a night ... [05:17] negronjl: what is dhcp serving? [05:17] bigjools: checking that now ... 'cause as far as I know ...the router should be the _only_ DHCP running and it has static dhcp clients [05:18] negronjl: mac address mismatch then [05:18] check static config vs what maas has [05:18] bigjools: checking that too but, gotta be honest ... I'm pretty tired ... I'll be checking that and then calling it a night [05:19] what do you mean ... what maas has ? [05:19] maas stores macs for the nodes [05:19] and tells cobbler what to write [05:19] so if you're now doing your own static config then it's probably different [05:20] bigjools: do tell ... the router's DHCP/DNS seems to have all of the right MAC addresses ... [05:20] how do you know? [05:20] bigjools: can you tell me where to find what maas-server has ? [05:20] look in its UI [05:21] bigjools: that seems to be right as well [05:22] are you sure the DHCP server on the router is assigning the IP then? [05:25] bigjools: checking again [05:26] also, in some dhcp setups you need to explicitly say that pxe boots can have the same IPs as regular boots [05:39] done for tonight ... I'll fight some more with this tomorrow [05:39] thank you very much for the help all [05:44] sleep well [05:47] nn [06:03] life o/ [06:03] lifeless: o/ [06:04] * SpamapS wonders what happened to irssi there [06:04] o/ [06:04] you hit tab and typed too fast [06:04] and triggered its 'paste detection' [06:05] which, FTW, bypasses all character evaluation. [06:06] ahhh [06:07] probably the bursty ssh [06:07] actually I'd swear its only bursty when I'm in tmux === rick2 is now known as rick1125 [07:50] * czajkowski hugs jtv [08:26] czajkowski: all for that little email? *blush* [08:27] And hi to you too :) [08:27] jtv: yes you at least replied! [08:27] simple things in life realy [08:27] :) [13:19] roaksoax: have you seen the question from Brian G. Peterson on the maas-devel list? he's having problems with the avahi boot story [13:19] is the image supposed to be working? [13:20] is the error he's encountering the same than you did when you tested with matsubara? [13:27] i havent [13:27] i will address it in a bit [13:27] roaksoax: thanks! [15:05] Daviey: where on the TFTP path do the enlistment kernels live? [15:05] Or do we use the same as install/commissioning? [15:12] jtv: wherever makes sense to you. [15:12] enlistment kenel == install kernel [15:12] comissioning kernel is sperate [15:12] OK, I'll use the install kernel then. Thanks. [15:13] Daviey: do we know yet exactly what kernel options we need to append for enlistment? [15:15] jtv: I think it's the options of maas-enlist in /var/lib/tftpboot/pxelinux.cfg/default [15:15] jtv: am I right? [15:15] roaksoax: do you have those to hand? [15:16] rvba: not sure I understand you correctly. Are you referring to the maas-enlist profile in Cobbler? [15:17] jtv: no, I'm referring to the line " append initrd=..." next to the "LABEL maas-enlist" bit in the file /var/lib/tftpboot/pxelinux.cfg/default [15:18] Ah, thanks. [15:18] Unfortunately I don't seem to have that bit there. [15:18] jtv: http://paste.ubuntu.com/1049268/ [15:19] Whoa that's a lot bigger than what I've got [15:19] Daviey: the kernel options [15:19] Daviey: yeah those are appended automatically by cobbler [15:20] err [15:20] Weird stuff in there... multiple locale options [15:20] Yeah, talking to me, I know. :) [15:20] some of them are appended automatically [15:20] jtv: some of the ones appended in the default file come from maas-import-isos and some from the commissioning [15:21] From the commissioning? But this is for a node that isn't even ready for commissioning yet. [15:21] jtv: the default ones are: append initrd=/images/precise-i386/initrd.gz ksdevice=bootif lang= text hostname=precise-i386 domain=local.lan suite=precise [15:21] jtv: when you import an ISO it creates the file that appends a few kernel args [15:21] We need a way to organize generation of this stuff I guess. [15:21] jtv: when you run maas-import-isos, it appends other required kernel args [15:22] jtv: and when you run maas-import-ephemerals, it adds different kernel args [15:22] jtv: so we just need a default, and we need to be able to add and modify them as needed [15:22] That's all plumbing we don't have yet, I think. [15:23] jtv: the addition of new custom kernel args is done by both, maas-import-isos and maas-import-ephemerals. So we should be able to do similar way to add more kernel args. However, we should be able to also add kernel args per node and as needed [15:25] But we're driving this from the inside out, rather from the outside in as Cobbler does. [15:26] In Cobbler, the scripts make this stuff up because they need to create profiles and that's where this stuff gets stored. We're driving it from the database, independently from the import scripts. [15:28] jtv: right, so import a netboot or server ISO into cobbler and check what args are added by default :). [15:28] jtv: should be similar to http://pastebin.ubuntu.com/1049288/ [15:29] Have to organize this in some way that makes it reusable. [15:29] roaksoax: another question for you: right now, a node, once the install is done, calls "wget "http://$http_server:$http_port/cblr/svc/op/nopxe/system/$system_name" -O /dev/null". We've already created the equivalent method (to disable PXE boot) on the metadata API. The trouble is that the call to this method needs to be oauth-authenticated. I'm afraid I'm going to have to expose that method anonymously. Do [15:29] you have a better idea? [15:31] roaksoax: I'm asking you because I'm not sure what the limitations are exactly… could we use a tool with oauth support instead of wget? [15:33] We have some code that we were hoping to extract into a library at some point… [15:33] Based on the juju code. [15:33] src/apiclient/maas_client.py [15:33] rvba: can't we just do similar way to how cloud init does it? [15:34] roaksoax: what would that mean exactly? [15:34] rvba: hold on, have an idea [15:35] rvba: ok, so for cloud-init, MAAS gives cobbler the MAAS_PRESEED which is a base64 blob right? [15:36] roaksoax: yes [15:37] roaksoax: in the cobbler-less world, this is not base64 encoded any more, but obviously the same data is present in the preseed context when it gets rendered. [15:38] rvba: ok, so even better then, in the template, we can just get the auth credentials and hand them to curl [15:39]