[00:00] hack the api source that deals with maas-enlist [00:00] log the request there [00:00] bigjools: this is not maas-enlist btw [00:00] this is maas0singla [00:00] maas-signal [00:00] yeah [00:00] that hits metadata service IIRC? [00:01] yeah === freeflying_away is now known as freeflying [00:29] roaksoax: any luck? [00:31] Any ideas why tags aren't working? http://pastebin.ubuntu.com/6207405/ === freeflying is now known as freeflying_away [00:32] bigjools: nope [00:32] bigjools: i gave up [00:32] roaksoax: urgh [00:32] roaksoax: I'll see if we can extend that logging to the metadata api [00:33] roaksoax: but surely a logger.info() hack will show you the data we need [00:33] bigjools: i uploaded the paclage to experimwntal ppa [00:33] that includes the [00:33] moonshotbstudd [00:33] stuff [00:33] anyway i gotta go [00:33] will be back later [00:33] roaksoax: ok [00:34] kurt_: are you sure the system id is right [00:34] I'll pastebin [00:37] bigjools: good call. I'm working with two different test systems. Thanks [00:37] im still having issues getting the maas instance to power off/start a node via virsh [00:38] ive tested that i can virsh --connect qemu+ssh://ubuntu@10.x.x.x/system from the maas instance and can see the nodes [00:38] and starting/stopping from there works [00:39] ssh keys are deployed from maas instance to host as well [00:39] keys are added to my preferences [00:42] stokachu, is your problem that maas can't power the nodes on? [00:42] smoser: yea [00:42] are yo usure ? [00:42] smoser: i click start node from the UI and nothing happens [00:42] if i use virsh from maas instance i can start it there [00:42] ive looked through all the logs and don't actually see where it attempts to power the node on [00:43] you can do that as the 'maas' user ? [00:43] smoser: yea running virsh --connect qemu+ssh://ubuntu@10.x.x.x/system from the maas instance works [00:43] no [00:43] as the maas user [00:44] sudo -Hu maas virsh --connect ... [00:44] lemme try [00:44] in saucy the virsh.template is /etc/maas/templates/power/virsh.template [00:44] it might be elsewhere on raring. [00:44] it prompts for a password so i dont have ssh keys copied for that user [00:45] right. so 2 options then. [00:45] a.) get the maas user able to connect without prompt [00:46] b.) change that template file in issue_virsh_command() to do sudo -Hu someuser [00:46] and make someuser able to connect [00:46] ok, and where in the logs could i find errors like this? [00:46] like where it seems to be waiting on input [00:47] is that a celery thing? [00:47] i dont know where you'd see logs liek this. probably would be in cluster-controller. [00:47] it may be that its just still blocked int hat script [00:47] and there *is* no long [00:47] no log [00:48] fwiw, http://bazaar.launchpad.net/~virtual-maasers/charms/precise/virtual-maas/trunk/view/head:/scripts/setup-maas [00:48] see line 230 there. thats how we hacked it. 229 -> 243 [00:49] ah ok [00:50] but you shoudl prbably be able to get the maas user able to ssh itself like that [00:50] smoser: yea i need to set a shell for the maas user and all that [00:50] right. [00:51] ok lemme do that now and see [00:56] smoser: http://askubuntu.com/questions/292061/how-to-configure-maas-to-be-able-to-boot-virtual-machines found this as well [00:56] maybe we should add that into the official docs for maas on libvirt [00:57] that is a well written answer. [00:58] thats the way i just did it and it worked [02:06] roaksoax, bigjools ... is there a way to create a non-admin user non-interactively ? [02:06] i guess i can just create admin and the drop staff or something. [02:07] smoser: don't think there is [02:07] no idea [03:36] bigjools: any luck? [03:36] roaksoax: well I was waiting for you to implement my suggestion [03:37] bigjools: lol what's your suggestion again? [03:37] logger.info() in the relevant API call [03:37] dump the request [03:37] either that or you just tcpdump [03:39] bigjools: well unfortunately i lost access to the environment [03:39] so i guess nothing will get done today [03:39] damn [03:39] well we can reproduce elsewhere [03:39] bigjools: yeah [03:39] bigjools: one with ipmi [03:39] at least if it's not reproducible it's a little less urgent [03:39] smoser: do we have anything available? [03:57] bigjools: do we have somewhere to reproduce this? [03:57] roaksoax: qa lab? [03:57] garage maas? [03:57] bigjools: garage maas is a no [04:00] bigjools: i have an idea [04:00] will try to reproduce locally faking ipmi discovery [04:05] roaksoax: we need an SRU for python-tx-tftp (0.1~bzr31-0ubuntu8) [04:05] for precise [04:05] bigjools: cloud archive [04:05] is it in there already? [04:05] should be yes [04:05] and is this the excuse for not doing SRUs now? :) [04:06] bigjools: not, but the reason of the cloud-archive is we want to avoid the headaches of sru's right? [04:07] I thought it was to get around changes that were not SRUable [04:07] that said, I don't know if that tftp one is or not [04:08] bigjools: probably [04:08] well let's think about it post release [04:12] yeah, better time :) [04:26] bigjools: why can't maas-cli support multiple value pairs for tags? i.e. maas-cli maas tag nodes cntrlr-node, quantum-node OR cntrl-node quantum-node etc. [04:27] bigjools: reproduced locally [04:27] bigjools: so how can I see logs again? [04:27] roaksoax: good, yet bad :( [04:27] roaksoax: edit the maas-signal function to log the request [04:30] bigjools: payload = geturl(url, creds=creds, headers=headers, data=data) > [04:30] ? [04:30] kurt_: because it wasn't implemented that way I guess [04:30] doesn't it make sense to do that? [04:30] roaksoax: hang on [04:30] kurt_: probably, but it is what it is [04:31] too much to ask for an enhancement? [04:31] feel free to file a bug [04:31] ok dokey [04:31] not sure if/when it would get looked at [04:31] but the source is open.... ;) [04:31] i'm almost certain there are bigger fish [04:31] :) [04:32] I can agree there [04:33] about your question about how to detect VMWare, I was going to respond in email. But since you are here…how is kvm detected? [04:33] bigjools: i think we could log where we receive the mparticular request [04:34] on the metadata server side [04:34] roaksoax: I am making you a patch [04:34] ok [04:34] bigjools: never mind. I just added info to the bug. [04:36] roaksoax: untested, but: http://pastebin.com/zVdmRAPD [04:41] bigjools: nothing logged :/ [04:41] ! [04:41] restarted apache? [04:41] did that [04:41] restarting machine [04:41] wtf [04:52] roaksoax: https://bugs.launchpad.net/maas/+bug/1232154 [04:52] Ubuntu bug 1232154 in MAAS "Logs seem incomplete" [Critical,Triaged] [04:52] something is screwed [04:52] try using print() instead of logger [04:52] yeah is saw a few MP's related to loggers [04:54] bigjools: ok so tomorrow i'll confirm this bug is in trunk without the moonshot stuff [04:54] bigjools: but this really shouldn't be affected by mnoonshot [04:54] bigjools: lp:~andreserl/maas/moonshot_enablement [04:54] yeah [04:54] I agree [04:55] roaksoax: having said that, the daily tests in the QA lab would have been blowing up [04:56] bigjools: yeah/ I'm wondering if it is because enlistment works, then commissioning for some reason keeps the same credentials [04:56] i'll debug this tomorrow though [04:59] roaksoax: it's possible that is exactly what is happening [04:59] a bug was fixed where something was getting overwritten unnecessarily IIRC [04:59] maybe that's related [04:59] we'll see [05:04] bigjools: so it seems the mparameters are not being passed [05:04] raise MAASAPIBadRequest("power_type %s, power_params %s" % (power_type, power_parameters)) [05:04] MAASAPIBadRequest: power_type ipmi, power_params None [05:05] roaksoax: where is that code? [05:06] bigjools: api.py [05:06] store_node_xyz [05:06] maasserver/api.py [05:06] store_node_power_parameters [05:06] so maas-signal is broken? [05:07] bigjools: maybe! i'll debug this tomorrow against the latest trunk (without my enablement branch) [05:08] ok [05:09] bigjools: ok [05:09] i know what the issue is [05:09] * roaksoax is blind [05:09] roaksoax: params["power_parameters"] = json.dumps(power_parms) [05:09] that's what was missing :( [05:09] roaksoax: missing where? [05:09] bigjools: in my enablement branch :) [05:09] roaksoax: lol [05:09] sorry for laughing :) [05:10] roaksoax: TESTS! [05:10] bigjools: yeah I haven't gotten to that part yet [05:10] can you see why we are anal about tests? :) [05:12] bigjools: lol [05:12] bigjools: anyway, sorry for the noise [05:12] * roaksoax bed [05:12] roaksoax: sleep well! [05:12] 2 more months till vacation [05:12] yaaaaaaaay [05:14] :) === CyberJacob|Away is now known as CyberJacob === rbasak_ is now known as rbasak [10:30] how does the maas api versioning work? [10:30] bug 1236734 mentiones that ip_addresses doesn't exist in the raring maas, can I not check that based on maas version somehow? [10:30] bug 1236734 in juju-core "juju 1.15.1 polls maas API continually" [Undecided,New] https://launchpad.net/bugs/1236734 [10:34] be great if someone could take a look at bug 1236786 [10:34] bug 1236786 in MAAS "ISO fails installing maas-dhcp" [Undecided,New] https://launchpad.net/bugs/1236786 [10:34] it breaks the install CD currently === freeflying_away is now known as freeflying [10:47] roaksoax: hi, could you please have a look at bug 1236786? [10:47] bug 1236786 in MAAS "ISO fails installing maas-dhcp" [Undecided,New] https://launchpad.net/bugs/1236786 === freeflying is now known as freeflying_away === freeflying_away is now known as freeflying [13:19] rvba: there's really no information for me to figure out what's wrong on that bug, I'd need logs [13:20] smoser: bug #1236786 [13:20] bug 1236786 in MAAS "ISO fails installing maas-dhcp" [Undecided,New] https://launchpad.net/bugs/1236786 [13:20] bbcmicrocomputer: ^^ [13:21] roaksoax: thanks for looking into it. === freeflying is now known as freeflying_away [13:58] smoser: ok so the fix for the apache2 thing works in saucy [13:58] i have not tested packaging [13:58] in precise [13:58] but saucy works [14:14] rvba: around? [14:15] roaksoax: yes [14:15] rvba: so any updates on the maas-import-ephemerals stuff? [14:16] roaksoax: well, yes; making progress but it's not yet done. Julian wanted Jeroen to make a few changes to the migration stuff… sorry I can't be more precise than that. [14:17] rvba: so there's no way we will have something for today ig uess then? [14:17] roaksoax: no, not today. Hopefully tomorrow. [14:17] rvba: alright.. tomorrow is really our last day to upload stuff [14:17] smoser: ^^ [14:46] roaksoax: added syslog with debug mode to bug 1236786 [14:46] bug 1236786 in MAAS "ISO fails installing maas-dhcp" [Undecided,New] https://launchpad.net/bugs/1236786 [14:51] bbcmicrocomputer: what's the workaround you had? [14:53] roaksoax: I was mounting securityfs on /target/sys/kernel/security, then installing, but in the preseed/late_command option [14:53] roaksoax: although you can't use that here [14:53] bbcmicrocomputer: right cause judging from the logs... it doesn't really tell me what's failing exactly [14:54] bbcmicrocomputer: can you try this though? [14:54] bbcmicrocomputer: http://paste.ubuntu.com/6209648/ [14:55] bbcmicrocomputer: that should probably deal with it [14:55] since invoke-rc.d is denied execution (which is expected) maybe that's causing the failure [14:55] yeah it's not obvious, 'Warning: unable to find a suitable fs in /proc/mounts, is it mounted?' comes from apparmor_parser inside maas-dhcp postinst I believe [14:55] which returns exit status of 1 [14:56] bbcmicrocomputer: righ [14:56] then we would just run preseed/late_command as 'in-target sh -c \"mount -t securityfs none /sys/kernel/security; apt-get -y install maas-dhcp"' [14:57] and that would work [14:57] bbcmicrocomputer: right, so have you looked at any other package that can be installed during the installed that uses apparmor_parser? [14:58] roaksoax: nope [14:58] bbcmicrocomputer: because i'm pretty sure there are other packages that are installed during the installer that install an apparmor profiel [14:58] so i don't think that would be a cause of issues [14:58] ok [14:59] bbcmicrocomputer: let me see what's happening in that postinst though [14:59] ok, thanks [15:10] bbcmicrocomputer: taking for ever to download the latest ISO [15:10] bbcmicrocomputer: is it possible for you to quickly edit the postinst script [15:10] bbcmicrocomputer: before it gets run and add a set -x ? [15:10] so it outputs what's actually doing? [15:11] roaksoax: sure, I figure something [15:11] bbcmicrocomputer: thank you! [15:34] bbcmicrocomputer: so I'm pretty sure this would avoid the issue on the cd install: http://paste.ubuntu.com/6209795/ [15:35] bbcmicrocomputer: but dunno what the effect would be [15:36] roaksoax: I can try, I'm just collecting the previous output :) [15:37] roaksoax: the install cycle takes 10 mins on my machine :( [15:38] boomer [15:42] bbcmicrocomputer: that seems to be the correct fix. bind9 for example does this: debian/bind9.postinst: apparmor_parser -r "$APP_PROFILE" || true [15:42] bbcmicrocomputer: so adding the|| true would allow this to work on CD isntall [15:44] roaksoax: yep [15:44] roaksoax: I'll try [15:44] just confirmed the theory with jdstrand [15:44] so I'll make the fix [15:45] roaksoax: I've added the output to the ticket [15:45] debug output, which gives up on that line, yes [15:49] bbcmicrocomputer: ok, i just tested it [15:49] bbcmicrocomputer: it works [15:49] roaksoax: cool [15:54] roaksoax: how long will it take for that fixed package to make it into the archive? [15:55] bbcmicrocomputer: tomorrow [15:55] bbcmicrocomputer: i'm not gonna upload anything until the final changes to trunk land [15:55] roaksoax: ok, cool, thanks [16:09] does maas utilize fast-path installation yet? [16:09] i see python-curtin [16:09] but there aren't any preseed template defined for it [16:15] i see curtin and curtin_userdata should i just extend userdata to contain post_Scripts? === teknico1 is now known as teknico [18:51] has curtin/xinstall been tested with preseed_xinstall+generic? [18:51] so far its not picking up the preseeds [19:17] stokachu: yes it has [19:18] roaksoax: i see where curtin_userdata has a spot for late_commands [19:18] but i dont see how preseed_xinstall is coming into play [19:18] stokachu: it does, i haven't test it [19:18] stokachu: preseed_xinstall is only a "preview" [19:18] so whats the format of curtin_userdata if i wanted to add multiple commands [19:19] stokachu: you'd need to check with smoser :) I haven't got that far just yet [19:19] roaksoax: http://paste.ubuntu.com/6210689/ [19:19] ok [19:23] stokachu: but judging for the pb, i'd say something like: http://pastebin.ubuntu.com/6210707/ [19:24] roaksoax: gotcha trying that now [19:26] hmm tried something like http://paste.ubuntu.com/6210716/ and it didn't work.. ill just wait for smoser to come back [19:27] stokachu, here. [19:27] :) [19:27] you rad too far [19:27] preseed_xinstall is not invovled period. [19:27] that was from the docs on maas.ubuntu.com [19:27] http://maas.ubuntu.com/docs/development/preseeds.html [19:28] k. docs suck. [19:28] smoser: so ive been messing with curtin_userdata [19:28] http://paste.ubuntu.com/6210716/ trying to get late_commands to work [19:28] fastpath installs and it's definately fast [19:28] 12 seconds [19:29] right. so curtin_userdata is what you'd modify. [19:29] what do you need to do [19:29] http://paste.ubuntu.com/6210716/ check that snippet [19:29] just want to download and run a shells cript in the late command [19:32] i was using http://bazaar.launchpad.net/~curtin-dev/curtin/trunk/view/head:/doc/topics/overview.rst as a reference [19:33] stokachu, right. thts not unreasonable. [19:33] smoser: is my config correct with the late_commands? [19:35] stokachu, yeah, its not unreasonable. [19:35] http://paste.ubuntu.com/6210737/ [19:35] might work also [19:36] smoser: nice let me try that [19:36] you're just replacing the builtin "configure network" command [19:36] which is [19:36] 'network_commands': {'builtin': ['curtin', 'net-meta', 'auto']} [19:36] stokachu, are you getting installs ? [19:37] your 12 seconds doesn't seem reasonble. [19:37] smoser: so it installs and i can ssh to it [19:37] and i create a file to make sure it is destroyed on new node installs and its gone [19:37] wow. [19:37] well thats fast! [19:37] the logs indicate that its pulling the curtin images [19:38] dude its super fast... 25minutes using the old way, 12 seconds using curtin [19:38] keep in mind this is in a kvm though [19:38] yeah. [19:38] so bare metal may be different [19:38] and with unsafeio also [19:38] yea [19:38] still i hadn't seen that fast. [19:38] :) [19:39] trying the install now to see if the network gets picked up [19:39] the goal is for when maas knows about networking is for it to do basically what i told you to do there. [19:40] smoser: http://imgur.com/ytjNstd [19:40] and it didn't land in saucy, but the goal was/is also to have a 'user-install-data' snippet. [19:40] that can be provided to maas. [19:40] ah ok [19:40] and it goes in as a config to curtin [19:40] so you'd no thave to be able to modify the preseed, and could change your mind on every install. [19:40] thats awesome [19:40] the install didn't pick up the vlansetup script i have though :( [19:41] you should configure console output on your vms. [19:41] hm.. [19:42] serial console output log to a file [19:42] is what i meant. [19:42] http://paste.ubuntu.com/6210772/ thats my script [19:42] so you dont have ot deal with crappy vnc for watching [19:42] smoser: yea i am going to set that up today as well [19:42] oh. well, fwiw, what i gave you above was not to be executed. [19:42] just to be downloaded. [19:42] ah [19:43] that would be what you have in /etc/network/interfaces [19:43] :) [19:43] ahh [19:43] is $OUTPUT_INTERFACES defined or should i change that to /etc/network/interfaces? [19:44] OUTPUT_INTERFACES should be defined by curtin when it executes the command. [19:44] ok [19:46] to execute, something like: [19:46] mine: ['sh', '-c', 'wget "$1" -O - | sh', '--', 'http://192.168.122.206/vlansetup'] [19:46] smoser: thanks ill try that [19:46] but you'll have to change your script so that it writes to $OUTPUT_INTERFACES [19:46] or [19:47] yeah, you'll have ot do that reasonably. writing to /target/etc/network/interfaces wont work [19:47] for 2 reasons [19:47] a.) i dont think it will be mounted at target [19:47] b.) the i think it'd get copied over. (or might fail later if there i sno OUTPUT_INTERFACES) [19:48] ok thats cool, ill mess around with it some more today and see what i can come up with [19:48] at least this gets me in the right spot for curtin [19:48] smoser: thanks :) [19:48] no, clearly your devcycle is pretty fast here. [19:48] now. [19:49] but if you want, you can use curtin trunk there is a 'launch' that will boot kvm install to a disk and stop [19:49] smoser: niceee that would be handy [19:49] but it doesn't reboot yet :) [19:50] thats ok it would still easier [19:50] see doc/devel/README.txt [19:50] and feel free to add a '--then-boot-it-dummy' flag [19:51] smoser: when are we going to have saucy images? [19:51] cool ill check that out today too [19:51] roaksoax, saucy images ephemeral ? [19:51] they're there. [19:51] smoser: released? [19:52] smoser: or daily's? [19:53] http://paste.ubuntu.com/6210816/ [19:53] smoser: cool thanks [19:53] smoser: btw... so we need to decide what to do with the moonshot and maas-enlist and ipmitool [20:08] in my /etc/network/interfaces file after a node start i see ## This file is generated by cloud-initramfs-dyn-netconf [20:08] could that be overriding my customizations? [20:20] ooh dynamic network confic [20:20] smoser: ^ is that new ? [20:29] lifeless, /etc/network/interfaces.d is new in saucy. [20:29] that config there is for 'curtin' which is the "fast path installer" in saucy. [20:32] https://git.openstack.org/cgit/openstack/diskimage-builder/tree/elements/dhcp-all-interfaces [20:32] is a related thing we did [20:33] yeah, curtin does that too. [20:33] its kind of garbage though [20:33] as dhcp eth0 [20:33] then wait some length of time [20:33] which might be to short [20:33] kinda sucks. [20:34] yeah [20:34] http://bazaar.launchpad.net/~curtin-dev/curtin/trunk/view/head:/curtin/net/__init__.py [20:34] 'is_connected' at least is sane. [20:40] stokachu, progress ? [20:41] smoser: everytime i boot the node it just seems like /etc/network/interfaces is unchanged [20:41] i changed the script to output a interfaces file [20:41] and i added a config_hook [20:41] hm.. [20:41] i'll play some [20:41] http://paste.ubuntu.com/6211001/ [20:41] thats what i have so far [20:42] http://paste.ubuntu.com/6211003/ thats my interfaces file [20:44] hm.. [20:44] am i supposed to copy the /etc/network/interfaces file in the config_hook? [20:45] docs suggest that unless told so it just keeps the interfaces file in memory [20:46] curtin shoudl copy OUTPUT_INTERFACES to the target /etc/network/interfaces [20:46] automatically right? [20:46] right. [20:46] hmmm [20:47] smoser: could it be and then getting overwritten by cloud-initramfs-dyn-netconf? [20:49]