/srv/irclogs.ubuntu.com/2012/10/03/#maas.txt

roaksoaxsmoser: around already?>00:19
smoserhere, roaksoax00:29
roaksoaxsmoser: so we are gonna need to pass the "config" to my script, instead of harcoding it00:30
roaksoaxsmoser: so I'm guessing I should pass the config to maas-signal, which will pass it to maas-autodetect-ipmi?00:31
smoserwhat do you mean pass00:31
roaksoaxsmoser: as it tell it where's the template / config00:31
roaksoaxas in*00:31
roaksoaxsmoser: https://pastebin.canonical.com/75758/00:31
roaksoaxsmoser: that one, we need to tell the location of that template00:32
roaksoaxtemplate/config00:32
smoseri'm sorry. i'm still being dense.00:32
roaksoaxsmoser: bmc-config --commit --filename /location/of/cofig/file.ipmi00:33
bigjoolshello guys00:34
roaksoaxbigjools: good morning00:34
bigjoolsI might have done something bad to my debconf database, I am getting this trying to install the maas package after previously purging it: http://paste.ubuntu.com/1255593/00:35
bigjools(I need to sh -x the postinst to get that output, otherwise it's fails silently)00:35
roaksoaxbigjools: what new package are you trying to install? the latest I made available in experimental PPA?00:35
bigjoolss/it's/it/00:35
bigjoolsroaksoax: one I made myself yesterday00:36
roaksoaxbigjools: that's fixed then :)00:36
bigjoolsit's not the same problem as the one you fixed00:36
bigjoolssince I'd already found that and tried to fix it and then got held up by this :/00:36
roaksoaxbigjools: are you referring that this doesn't fix it? http://bazaar.launchpad.net/~maas-maintainers/maas/packaging/revision/11300:37
roaksoaxsee the postinst00:38
smoserdaily ppa has maas-dhcp installable again00:39
bigjoolsroaksoax: it's hard to see what the effect of the changes are - I was assuming you were referring to the fix for "service isc_dhcp_server stop"00:39
smoserbigjools, just curious, what does "fix committed" / "fix released" mean in maas ?00:39
smoserbigjools, he was referring to the celery postinst00:40
smoseri think00:40
bigjoolssmoser: not a a lot.  Tarmac changes bugs to fix-committed but we don't distinguish that with fix-released on trunk00:40
smoserhttps://bugs.launchpad.net/ubuntu/+source/maas/+bug/105052300:40
ubot5Ubuntu bug 1050523 in maas (Ubuntu) "maas kernel cmdline must include iscsi_initiator" [High,Triaged]00:40
smoserso you marked that "fix released"00:40
smoseri'm just curios, so i can at least attempt to be consistent.00:41
bigjoolssmoser: yes, they are all fix released if they are in trunk, I am not using fix-committed00:41
bigjoolsthen they disappear off bug listings00:41
roaksoaxbigjools: check the changes made in maas-region-controller.postinst, I removed and if statement00:42
roaksoaxan*00:42
bigjoolsroaksoax: aha!00:42
bigjoolsnice one00:43
roaksoaxbigjools: there's a related issue though that I can't find a solution for00:43
roaksoaxbigjools: if you remove (not purge) and install again, it will show the same error00:43
roaksoaxbigjools: and I already know how to fix the maas-celery thing, i'll work on it in a bit though00:44
bigjoolsroaksoax: ok - jtv landed a change so we can pass uids I think00:44
bigjoolsbug 106011400:45
ubot5Launchpad bug 1060114 in MAAS "DELETE operations are not idempotent" [Medium,Invalid] https://launchpad.net/bugs/106011400:45
bigjoolsheh00:45
smoserbigjools, https://bugs.launchpad.net/maas/+bug/105813700:45
ubot5Ubuntu bug 1058137 in MAAS "no way to get first user's first set of api creds" [High,Triaged]00:45
smoserif i just put that in comment 4 into a python script and run it00:45
smoseri stack trace00:45
bigjoolsmatsubara: that python script works for you?  --^00:46
smoserhttp://paste.ubuntu.com/1257137/00:46
bigjoolssmoser: export DJANGO_SETTINGS_MODULE=path/to/settings.py00:47
bigjools/etc/maas/whatever00:47
matsubarabigjools, yes, I think I did some path hackery to get it to work00:48
bigjoolsgrumble grumble, roaksoax removed my changelog attribution :)00:53
roaksoaxbigjools: no i didn't :)00:54
bigjools -- Andres Rodriguez <andreserl@ubuntu.com>  Tue, 02 Oct 2012 13:39:47 -040000:54
bigjoolsyou did :)00:54
roaksoaxbigjools: yeah I made changes to the changelog00:54
roaksoaxdch -i changes that :)00:54
* bigjools shakes fist at dch00:55
bigjools-u surely?00:55
bigjoolsor -e?00:55
roaksoaxbigjools: btw... an assumption I am working with if the fact that it doens't matter how many releases we release in PPA00:55
roaksoaxbigjools: the packaging should remain UNRELEASED00:55
bigjoolsok00:55
roaksoaxand only change when we release to archives00:55
roaksoaxbigjools: but since the changelog is being pilling up I think is ok00:56
bigjoolsroaksoax: well dch -i doesn't change the release it adds a new one, so something changed the attribution I added00:56
roaksoaxbigjools: because entries of releases that haven't been uploaded to the archives will appear in the changelog even though we didn't00:56
bigjoolsyup00:57
bigjoolssoyuz is great :)00:57
roaksoaxbigjools: dch -e -DUNRELEASED works?00:57
roaksoaxerrerr dch -e -Dquantal00:57
bigjoolsroaksoax: in r113 the change line was updated, what dch option does that?00:59
smoseri tihnk in thie change log there we need to ditch the double bzr branch00:59
smoserwe get one "for free" from the daily ppa00:59
smoserbiuld00:59
smoserso we should either be pedantic about getting that updated in the code on each commit (or tooling) or just remove the bzr revno from there.01:00
roaksoaxbigjools: dch -i does it for me01:00
bigjoolstrue, but it's needed to build locally from recipe isn't it?01:00
bigjoolsroaksoax: -i adds a new one for me01:00
smoserbigjools, well we can just add a script that does the build locally and inserts it.01:01
bigjoolssmoser: just need to fix the debian/rules target I expect01:01
bigjoolsroaksoax: bzr diff -c 113 debian/changelog01:01
roaksoaxbigjools: dch -e --nomainttrailer01:05
bigjoolsroaksoax: aha01:06
bigjoolsroaksoax: useful, it saves me having to use -k when recipe building :)01:06
roaksoaxindeed :)01:06
roaksoaxbigjools: btw.. i already told allenap... no scaping from Peruvian Pisco is allowed at UDS01:07
roaksoaxsmoser: ^^01:07
bigjools?!01:07
roaksoaxescaping*01:07
roaksoax:)01:07
bigjoolsdoes that involve naked bodies and mud?01:07
bigjoolsotherwise I am totally not up for it01:07
roaksoaxlol01:08
bigjoolsroaksoax: getting this on the experimental package: http://paste.ubuntu.com/125716701:09
roaksoaxbigjools: as an error?01:13
smoserok. this is silly.01:13
roaksoaxbigjools: can you pastebin a full log?01:13
smoserhttp://paste.ubuntu.com/1257171/01:13
bigjoolsmatsubara: can you paste a full hackery for smoser please?01:14
roaksoaxsmoser: do you think i should allow sending several config files rather than just one?01:14
bigjoolsroaksoax: http://paste.ubuntu.com/125717201:15
smoserroaksoax, do you have a reason for doing so01:15
smoser?01:15
roaksoaxsmoser: so if someone sets custom ipmi configs they can be sourced automatically?01:15
roaksoaxbigjools: yeah that's dbconfig being an asshole i think01:15
smoserit doesn't sound unreasonable.01:15
bigjoolsroaksoax: yay01:15
bigjoolsroaksoax: I'll re-purge01:16
roaksoaxbigjools: yes, just to be sure, remove dbconfig-common too and make sure /etc/dbconfig-common is empty01:16
roaksoaxbigjools: i discovered that maas.conf under dbconfig-common is preserverd, and we also need to remove that01:16
roaksoaxbigjools: i'll take care of that along the celery stuff01:16
roaksoaxalong with*01:16
matsubarabigjools, smoser: something like this should work: https://pastebin.canonical.com/75761/01:16
smosermatsubara, well, thats basically what i did01:18
bigjoolsroaksoax: /etc/dbconfig-common/maas-region-controller.conf exists01:18
bigjoolsweird I'll remove it01:18
bigjoolsroaksoax: on re-install (after removing it) I now get a debconf page telling me it was currently iunstalled and locally mnodified01:19
bigjoolsinstalled*01:19
* bigjools installs package version01:20
matsubaraok, guys, gotta go. talk to you tomorrow.01:21
matsubarahave a good night/day01:21
=== matsubara is now known as matsubara-afk
roaksoaxbigjools: say yes01:22
bigjools:)01:22
bigjoolssmoser, roaksoax: how long are you in CPH?01:23
roaksoaxbigjools: 2 weeks01:23
bigjoolsjust UDS?01:23
bigjoolsah01:23
smosersame as roaksoax01:23
smosertoo long01:23
roaksoaxsmoser: yay!!01:23
roaksoaxlol01:23
bigjoolsok plenty of time to meet up for an evening food/drink01:24
bigjoolsyeah too long01:24
bigjoolsmy wife is going to self destruct looking after our twins on her own01:24
jtvHi folks01:25
bigjoolshi jtv01:25
smoseri'm really sorry for being a dolt. but some help would be greatly appreciated.01:26
smoserhttp://paste.ubuntu.com/1257179/01:26
jtvUser isn't a MAAS thing.  It comes with Django.01:29
jtvdjango.contrib.auth.models.User01:30
jtvHow it got to be in contrib I may learn someday.  How it got to stay there I just won't believe no matter what you tell me.01:30
bigjoolsjtv: should still work01:30
bigjoolshowever I get this:01:31
bigjools    from django.db import utils01:31
bigjoolsImportError: cannot import name utils01:31
bigjoolsat the bottom of the trace01:31
smoserhttps://bugs.launchpad.net/maas/+bug/1058137/comments/501:31
ubot5Ubuntu bug 1058137 in MAAS "no way to get first user's first set of api creds" [High,Triaged]01:31
roaksoaxsmoser: http://paste.ubuntu.com/1257181/ with http://paste.ubuntu.com/1257183/ --> what do you think?01:31
smosergot it01:31
bigjoolssmoser: phew :)01:32
smoserthank you bigjools jtv01:32
bigjoolsnp01:32
roaksoaxbigjools: btw... I think we need to be able to pass kernel arguments01:33
roaksoaxbigjools: and be able to edit that01:33
bigjoolspxe template?01:33
roaksoaxbigjools: console=ttyS001:34
roaksoax bigjools we need to be able to edit things like that01:34
roaksoaxbigjools: so yes, it should be added to the pxe template01:34
roaksoaxbigjools: but should probably be per node01:34
roaksoaxif one wishes to change that from a node, or set of nodes01:34
bigjoolsroaksoax: how important is this?01:35
bigjoolsshould be easy enough to do, but I need to prioritise against the thousand other bugs01:36
roaksoaxbigjools: ver important i'd say, even sabdlf mention we should be able to edit it. For example, console configuration needs to be done manually directly in the BIOS for ipmi cards01:36
bigjoolseugh01:36
roaksoaxbigjools: so they configure serial to be ttyS2001:36
roaksoaxso maas, as it stands right now, sends console=ttyS101:37
bigjoolsok, Node needs a custom "kernel_params" field then, and we can add that to the pxe template01:37
roaksoaxbigjools: indeed, so we don't really need it to be displayed in the WebUI01:38
smoserroaksoax, bigjools well, per-node kernel settings are not all that important really.01:38
smoseri opened that bug01:38
smoserand i do think they need to be editable01:38
roaksoaxsmoser: right, so we can set global settings, but if the user wants to change them, per node, he needs to01:38
smoserbut realistically, in a large scale, you're going to have a couple groups of systems01:38
bigjoolsmy question is, important for *this release*01:38
smoserso per-node would suck anyway.01:38
smoseri cant think of them being a hard requirement for this release.01:38
roaksoaxsmoser: right but for example, in sabdlf's cluster I had to manually edit what console to output in maas code, because it was configured differently on the BIOS01:39
smoserbut the ability to append cmdline arguments without editing python source would be nice.01:39
bigjoolssmoser: I am thinking that the cluster controller would want to set the parameters when it uploads node definitions (in the future)01:39
smoserbigjools, ok. heres the whole sucky problem.01:39
bigjoolsall this stuff will/should be automated01:39
smosera.) before a node is enlisted, you know nothing about it. so you can't really customize stuff per-node at that point01:39
smoserb.) after that point, you could have some custom settings01:40
bigjoolsroaksoax: https://code.launchpad.net/~julian-edwards/maas/broken-dns-install-bug-1060549/+merge/12762101:40
smoserbut *really*01:40
smoserc.) in a perfect world, give some knowledge of the setup and the node (hw and the like) you could perfectly generate all that anyway.01:40
bigjoolsI disagree with (a) since the cluster controller may know all about its hardware already01:40
smoserin which case you're to 'c'01:40
smoserso you dont need them01:40
smoser:)01:40
smoserroaksoax, i think you'd have been fine with the ailbiyt to append cmdline settings.01:42
smoserright?01:42
smoserand i would have in all my testing been ok to do the same.01:42
smoserthat would releieve a lot of pain01:42
roaksoaxbigjools: looks good, make sure +    add_user_group01:42
roaksoax49+01:42
roaksoaxis ident'd correctly01:42
roaksoaxsmoser: huh?01:42
bigjoolsroaksoax: GRARGH.  The rest of the file has got TABS in it01:43
bigjoolsI used 4 spaces01:43
bigjoolsTABS ARE EVIL01:43
roaksoaxsmoser: btw... i'm just gonna pass maas-signal with an argument with the json stuff, such as maas-signal --config --power-setings <json format>01:43
* roaksoax uses tabas for shell scripts and spaces for python code01:43
bigjoolsTABS ARE EVIL01:44
bigjools:)01:44
roaksoaxxD01:44
bigjoolsbecause you end up with what you see on the MP01:44
bigjoolsdo you care if I s/\t/    /g ?01:45
roaksoaxbigjools: different schooling I guess.. I was always a TAB user01:45
roaksoaxbigjools: go for it01:45
bigjoolswe're very particular about our code formatting in the Launchpa^WCloud Engineering team01:45
roaksoaxsmoser: yeah I think we need to be able to append that01:45
roaksoaxsmoser: adding the per-node isn't really that big of a deal, we don't even need to display it in the UI, just be able to modify it per node from maascli01:46
bigjoolsroaksoax: can you approve the MP please01:47
roaksoaxbigjools: done!01:48
roaksoaxsmoser: ok I think it is done i just need to test it01:48
* roaksoax heads home01:49
roaksoaxsmoser: i'll go home and test it on the HP mini servers01:49
bigjoolscheers01:49
bigjoolsjtv: it looks like you didn't fix the upstart conf to expect a fork, or am I missing something?01:56
jtvbigjools: I didn't get around to looking into that; I assumed it already expected a fork but got the wrong one.01:57
bigjoolsjtv: ok I 'll sort it out01:58
jtvBased on what you said, I thought the packaging branch already expected a fork.01:58
bigjoolsjtv: I tried it but it hung, I'll try it again01:58
jtv:(01:59
bigjoolsnow there *is* a fork :)01:59
bigjoolsit should work01:59
jtvThere was a fork before as well.01:59
jtvThat hasn't changed.01:59
jtvBut I removed the spurious fork that preceded it.02:00
bigjoolshmm02:00
bigjoolsjtv: something else is forking02:02
bigjoolsjtv: http://paste.ubuntu.com/125720202:02
jtvWell it can fork right off.02:03
* bigjools straces it02:03
bigjoolsARGH02:04
bigjoolsit's the sodding wrapper script02:04
bigjoolsjtv: when starting a python script, straces shows that it forks off this lot: "id, sh, ldconfig, sh, ldconfig, sh, uname"02:11
bigjoolswtf02:12
jtvwtf indeed02:12
jtvSo celeryd is a wrapper script?02:13
bigjoolsno02:13
bigjoolsmaas-provision02:13
bigjoolsthe one installed by the package that checks for uid==002:13
jtvGrrr02:14
bigjoolsso basically python is forking a load of stuff off before it starts your actual code02:14
bigjoolsunless it's hiding in maas-provision?02:14
jtvI bet it's import side effects.  :(02:15
jtvI tried this and found no forks: $ strace python -c 'print' 2>&1 | grep fork02:15
bigjoolsI am doing this:02:16
bigjoolssudo PYTHONPATH="/usr/share/maas${PYTHONPATH:+:}${PYTHONPATH}" CELERY_CONFIG_MODULE="celeryconfig_cluster" strace -o /tmp/strace.log -fFv /usr/bin/python -m provisioningserver start-cluster-controller http://maas.internal.example.com/02:16
bigjools(ignoring the wrapper for now)02:16
bigjoolsoh god02:17
bigjoolseven if we get this right it won't work jtv02:17
jtvYes, my son?02:17
bigjoolsbecause the fork can happen ages later02:17
bigjoolsso the upstart script will always hang until the admin accepts the cluster02:17
jtvYes02:18
* bigjools gives up02:18
jtvThen maybe we should kill two birds with one stone.02:18
jtvMove the polling into the child process, and instead of exec'ing, just have it hang around as basically a wrapper to celeryd.02:18
bigjoolsnot sure it'll work ATM anyway02:19
bigjoolsbecause of these 6 spurious forks02:19
jtvThat way, start-cluster-controller will hang around for as long as celeryd is running.02:19
bigjoolsI'd rather not do that02:19
bigjoolsbut let;s see what roaksoax thinks02:20
* bigjools lols at jtv's continued wrong merge targets :D02:21
jtvYes, it's become part of the process.  Why change it now?02:21
jtvbigjools, roaksoax: I think upstart can manage a process that just never exits.  If so, I think it would simplify our problem a lot if we could just keep the maas-provision process alive, instead of forking & exiting.02:30
smoserhttp://paste.ubuntu.com/1257227/02:30
smoserthoughts?02:30
bigjoolsis node-group-interfaces valid?02:31
smoserdifferent than http://paste.ubuntu.com/1257229/02:32
bigjoolsok02:32
* bigjools looks at code02:33
bigjoolssmoser: nodegroup called "master" does not exist02:33
jtvWait...  did I seriously just catch a snippet of advertising for the miraculous innovation of...  coffee without sugar!?02:33
bigjoolswell, uuid of master I mean02:34
smoserso what is 'master' there?02:34
bigjoolsmust be the uuid of an existing nodegroup02:35
bigjoolsmaas-cli api localmaas node-groups list02:35
smoserit would seem like no02:36
smoserhttp://paste.ubuntu.com/1257235/02:36
smosergot 'master' as you said02:37
smoserhm..02:37
bigjoolsif you list existing NGIs does it show?02:39
bigjoolshey, this is not a maas bug, where should be be retargeted? https://bugs.launchpad.net/maas/+bug/106019402:39
ubot5Ubuntu bug 1060194 in MAAS "cannot find "bnx2-mips-09-6.2.1a.fw" when booting from maas" [Undecided,New]02:39
smoserwoohoo.02:40
smoserbigjools, initramfs-tools maybe02:42
bigjoolsok02:42
smoseri actually think i saw this.02:42
bigjoolsthanks02:42
smosererr... jsut reading code thought "how strange, it brings up networking without looking waiting for udev"02:42
bigjoolssmoser: or cloud-initramfs-tools?02:42
smoserno02:42
smoserinitramfs-tools02:42
bigjoolsok done02:42
smoserincidently, it is probably fixed now02:42
smoserbecause cloud-initramfs-tools code goes looking for NICs and waits for udev02:43
smoser:)02:43
roaksoaxsmoser: i'm gonna have to test the cloud-init stuff tomorrow... the mini servers are gonna take forever to install02:43
roaksoaxsmoser: err upgrade02:44
smoserupgrade?02:44
roaksoaxsmoser: my maas mini server needs to be upgraded :)02:44
smoserbigjools, ok. so my goal in the last escapade is to configure dns02:46
smoserhttp://paste.ubuntu.com/1257248/02:46
smoserthat gets me a valid response02:47
smoserbut my goal is to get dhcpd functioning02:47
bigjoolscool02:47
smoserand i still have no /etc/maas/dhcpd.conf02:47
jtvHi roaksoax!  Did you see our problem with the fork-tracking for the cluster controller?02:47
bigjoolssmoser: check the celery log02:47
roaksoaxjtv: howdy. no I didn't02:48
roaksoaxjtv: any lp02:48
roaksoaxjtv: any lp bug?02:48
bigjoolssmoser: 1. has it received secrets from the region, 2. did it get any DNS tasks?02:48
jtvroaksoax: bug 1059453.  Turns out maas-provision does a lot of forks before it gets to the one we'd want upstart to track.02:48
ubot5Launchpad bug 1059453 in maas (Ubuntu) "The celery cluster worker is not properly stopped" [Critical,Triaged] https://launchpad.net/bugs/105945302:49
bigjoolssmoser: is https://bugs.launchpad.net/bugs/1060331 fixed now?02:49
ubot5Ubuntu bug 1060331 in MAAS "daily ppa install fails fully automated" [Undecided,New]02:49
smoserbigjools, sure.02:50
smoseri tihnk it was the isc_service_stop thing02:50
bigjoolsyeah02:50
roaksoaxjtv: i think that's a question for james hunt02:50
roaksoaxjtv: if not, we are probably have to run it as we used to02:50
bigjoolsroaksoax: oh FWIW there's a trick to conf file removal02:50
bigjoolsI asked colin about it02:50
roaksoaxbigjools: what did he say?02:51
bigjoolsroaksoax: "man dh_installdeb, man dpkg-maintscript-helper" :)02:51
bigjoolsand search for "conffile removal"02:51
roaksoaxbigjools: ah yeah, i know that :)02:51
smosercan i turn of header output from the cli ?02:52
bigjoolsso basically upstart confs are not treated specially02:52
bigjoolssmoser: not yet :(02:52
roaksoaxbigjools: nope, so we can use rm_conffile02:52
jtvThanks roaksoax -- I'll see if I can reach JH later.02:52
bigjoolsroaksoax: exactly02:52
roaksoaxbigjools: we need to do the same for maas.conf under /etc/dbconfig-common02:52
roaksoaxbigjools: i'll take care of it :)02:52
bigjoolsroaksoax: ok thanks!02:52
smoserbigjools, http://paste.ubuntu.com/1257255/02:55
smoserthat would seem the relevant bit of celery log02:55
bigjoolsyay02:57
bigjoolssmoser: anything for dhcp?02:58
smoserhttp://paste.ubuntu.com/1257259/02:58
smoserfull log02:58
bigjoolsis maas-dns installed?03:00
smoserno03:02
bigjoolsthat might help :)_03:02
bigjoolsalso maas-dhcp?03:02
bigjoolswell the former will ensure the latter03:02
smosermaas-dhcp, yes.03:03
smoseryou're saying dns ensures dhcp?03:04
bigjoolsyes03:04
smoserk.03:04
bigjoolswe can't do dns without dhcp03:04
bigjoolsI suspect a signal is missing to push out the dhcp config when NGI changes03:04
bigjoolshmm no that's wired03:06
smoserbigjools, so am i at least going down the correct path here?03:11
bigjoolssmoser: yes03:11
smoseris there some expected way that someone would set this up?03:11
bigjoolsthis is fine so far, if it's not working it's a bug03:11
roaksoaxsmoser: ok so the script itself seems to work03:14
roaksoaxsmoser: just need to test in a deployment03:15
smoserroaksoax, nice work.03:16
roaksoaxi'll try to get that tonight03:19
roaksoaxi just realized that i need to reconfigure the network :(03:19
roaksoaxmy home network :(03:20
smoserSetting up maas-dns (0.1+bzr1134+dfsg-0+1136+113~ppa0~quantal1) ...03:20
smoserchown: invalid user: `maas:root'03:20
smoserdpkg: error processing maas-dns (--configure):03:20
smoser subprocess installed post-installation script returned error exit status 103:20
roaksoaxsmoser: that's already been fixed by bigjools03:20
bigjoolssmoser: there's a fix just landed03:20
bigjoolsneeds rebuilding03:20
* bigjools hits the daily rebuild button03:20
bigjoolsshould be ready (published) in 10-15 mins03:21
bigjoolsI <3 recipes03:23
* bigjools grabs food03:29
smoserbigjools, ok03:33
smoserhttp://paste.ubuntu.com/1257282/03:33
smoser^ results in http://paste.ubuntu.com/1257283/03:34
smoserand no /etc/maas/*dhcp*03:35
smoserwell, i have to go to bed.03:44
smosertell me what stupid thing i've done there.03:48
smoserhttp://paste.ubuntu.com/1257288/ is some node group info.03:48
smoserit seems my operation to update my node worked.03:48
smoserok. bed.03:50
smosergood night.03:51
bigjoolssmoser: hmmm looks like a bug.  I'll get rvb to look, unless jtv wants to03:51
bigjoolsnn smoser03:51
jtvnn smoser03:51
jtvI'm looking up the pastes03:51
smoseroh. one thing to note. there are things in /etc/bind/maas03:52
smoser$ ls /etc/bind/maas/03:52
smosernamed.conf.maas       rndc.conf.maas                zone.master03:52
smosernamed.conf.rndc.maas  zone.77.168.192.in-addr.arpa03:52
bigjoolsthat looks expected03:52
smoserright03:52
smoserbut no dhcp03:52
bigjoolsyeah dhcp is differnet tasks and they';re not getting fired03:52
* jtv inserts lame joke about doing the same thing for us03:53
bigjoolsone of us will try to reproduce03:53
smoserok. both of you can get to ubuntu@10.55.60.13003:54
smoserto poke around03:54
smoserdo whatever you want.03:54
jtvThanks.  That'll tell us if it's a missing sudo or an apparmor thing.03:54
smoser(hoping through chinstrap03:54
smoserlater.03:55
jtvnn03:56
jtvpermission denied :(03:56
jtvbigjools, can you get into that system?03:57
bigjoolsit's probably on the VPN03:57
smoserubuntu@03:57
jtvYes I did03:57
jtvThis is a canonistack instance, not the VPN AFAICS03:58
smoserright03:58
smoserhttp://paste.ubuntu.com/1257300/03:58
bigjoolsoh that needs spethial setup too IIRC03:58
smoseronly hoping ghrough chinstrap03:58
smoseris all it needs03:58
jtvWell I did do it through chinstrap of course03:58
smoserHost 10.55.58.* 10.55.60.* 10.55.62.* 10.55.63.* *.canonistack *.canonistack203:59
smoser    ProxyCommand ssh chinstrap.canonical.com nc -q0 %h %p03:59
smoserthats all the config you should need.03:59
jtvWell, my key is in there.04:00
jtvAnd ssh into canonistack instances normally works for me.04:00
jtvAnd I'm doing the ubuntu@04:00
jtvAnd I'm going through chinstrap.04:00
jtvBut it denies me public-key auth.04:00
roaksoaxsmoser: so there's various maas regions now?04:00
roaksoaxerr04:01
roaksoaxcanonistafck regions*04:01
smoser204:01
roaksoaxah cool04:01
smoserwell, password auth is now enabled there04:03
jtvAhhhh this needs my canonistack key, not my regular ssh key04:03
jtvSo I need to register my canonistack keys as ssh keys.04:03
smoserand password is "maas-is-fun"04:03
smoserjtv, you dont have to do that.04:03
jtvYou sick, sick bastard04:03
jtvWell it doesn't seem to work without!  My ssh config for canonistack instances is set up to use my canonistack key as IdentityFile.04:04
smoseri imported your credentials from launchpad. those should forward through the ssh ProxyCommand above.04:04
smoseryeah, i'd recommend ditching that crappy canonistack key04:04
jtvBut every time you do this, it denies me login.04:04
smoserand only ever using your real keys04:04
jtvOh, that's not needed!?04:05
smosereuca-import-keypair --public-key-file ~/.ssh/id_rsa.pub jtf04:05
smoserthen launch instances with '--keypair jtf'04:05
smoser(spelled jtv wrong twice)04:05
jtvWhat are the odds04:05
jtvAnyway, still not working without that IdentityFile line in my ssh config.  :(04:06
roaksoaxsmoser: http://paste.ubuntu.com/1257310/ alright, we just need to figure out a way how to run it04:06
roaksoaxsmoser: unless you want to run it as a script?04:06
smoserwell, since it isn't really a script, it doesn't make sense.04:08
smoseri'd just add something that runs that script and calls maas signal04:09
smoserroaksoax, i can look more tomorrow.04:09
roaksoaxsmoser: yeah, either way I won't test it until tomorrow04:09
* roaksoax is off04:12
roaksoaxnight all04:12
jtvnn roaksoax04:27
bigjoolssmoser: if you are still there, I know what's wrong04:36
bigjoolswe never believe you when you say you're going to bed :)04:37
jtvHe must be asleep for real.  He wouldn't have been able to resist this one.05:08
bigjoolssmoser: adding a new nodegroup requires a corresponding celeryd to be running, listening on the right queue.  maas-provision start_cluster_controller does that in packaging but you can just run "celeryd -Q <uuid>"05:13
Fajkowskyhello I have problem with http://askubuntu.com/questions/195115/nodes-cant-connect-to-server-after-bootstrap08:51
Fajkowskycan someone help me with it?08:51
mgzwhat is the lint thing jtv will cry if I don't run?09:12
bigjoolsmake lint!09:13
mgzta.09:13
mgzsounds counter productive though...09:13
bigjoolsFajkowsky: answered your Q in #ubuntu-server but better to talk about it here09:13
mgzunmake lint!09:13
bigjoolsmgz: hoho!09:13
FajkowskyI have question about maas, I want to check if performence will be grow if I add more nodes to one service. I want to check on minecraft server because I think it's make big diffrence in performance if I add another node, I am right?09:17
bigjoolsFajkowsky: this is not a maas-specific question, but yes you can scale out easily with juju and maas09:18
Fajkowskywhoops , you are right09:19
jambigjools: so we will have the tags getting processed in the async workers eventually, how do we do integration testing of that?09:22
jamATM, we only seem to have unit tests09:22
jamthat mock out the actual POST/GET requests.09:22
bigjoolsjam: the test harness will run the tasks synchronously09:24
bigjoolsso leave everything unmocked and test for the desired end result09:25
bigjoolsprovided you're running with the right TestCase, anyway.  There are diffrerent ones and they are all called TestCase, which is something that needs to be fixed before I strangle a cat in frustration.09:25
bigjoolsand you need the celery test resource loaded, theres plenty of examples in existing tests09:26
rvbajam: just be aware of the fact that, in the tests, task.delay() calls the task in a synchronous fashion.09:27
jambigjools: so do all nodegroups have celery workers in the test suite, then?09:27
jam(right now the apis are restricted so that only the associated worker is allowed to call them)09:28
jamsince otherwise you could get around the privacy stuff09:28
bigjoolsjam: not sure how that'd work out, you might need to patch stuff in that case.  You just need to know that apply_async() or delay() just calls the Task func there and then.09:29
bigjoolsobviously there's no queue in this case09:29
jambigjools: well part of this is that I want to have at least one test that things work in a real integration setting (multiple nodegroups, and real celery workers talking via the apis.)09:29
bigjoolsok09:30
bigjoolsmmmm that might be hard09:30
jamI can also update some apis so they can be called by superusers09:30
bigjoolswe've always mocked things out in existing tests09:30
jambigjools: well, it is either that, or all the testing falls onto matsubara09:30
rvbaThe only thing you can do in tests is make sure that each task is routed to the right queue.09:30
bigjoolsyou can test either side of the Task easily09:30
bigjoolsand what rvba just said09:31
jamrvba: the problem with mocking it out on the celery side, is that you can easily get skew between what you think the API takes and returns, and what reality says.09:31
bigjoolsbut a complete integration test is very tricky as we don't start up celeryd09:31
bigjoolsjam: the API should be static anyway09:32
bigjoolsit's versioned09:32
jambigjools: well, it is just being implemented now... :)09:32
bigjoolswell, it's versioned in that we reserved a space for a version :)09:32
jamso the testing needs to be done manually for now?09:32
bigjoolsjam: if you patch out the task you can check the params and fail if they change at all09:32
rvbajam: you should be able to use the API for real (without mocking it) in a celery task.09:32
jamrvba: except for the permissions bits? or how do we work around those?09:33
jam(I do know that I had one accidental success if you 'yield' in an api call.09:33
jamapi stubbing returns the generator, and 'assertItemsEqual' passes just fine)09:33
bigjoolsjam: you should be able to test end-to-end provided you do something about the queue checking, perhaps mock it, I'd need to see it09:33
jambut in reality, the HTML level stuff turns the generator into a string09:33
jam"<generator ...>"09:34
bigjoolsbut the queue given gets passed to the task so it's easily checked09:34
bigjoolsrvb can help you, I gotta EOD now09:34
jambigjools: np, have a good evening09:34
jamrvba: I actually need to go pick up my son now, but I'll be back to pick your brain some more later.09:35
bigjoolscheers09:35
rvbajam: by permissions you mean the API credentials?09:35
rvbajam: sure, no pb.09:35
jamrvba: right, so I added apis that let the nodegroup worker get access to all information about nodes in the nodegroup09:35
jambut we can't make that public because that gets around the Node VIEW permission stuff09:35
jamso it is only allowed by the oauth key associated with the nodegroup worker.09:36
rvbaMakes sense.09:36
jamwhich isn't the 'client' that is running the 'create a tag'09:36
jamanyway, really gone09:36
jtvAny reviewers in the house?  https://code.launchpad.net/~jtv/maas/bug-1059453/+merge/12768009:41
mgzwhat is this netifaces python package the provisionserver wants, why do I not have it, and where can I get it...09:41
rvbajam: I think the only thing you need to do is patch the credentials used by the worker to connect to the API (i.e. simulate what refresh_secrets would do)09:42
mgzah, I bet it's because it's been added to required-packages since I last pulled locally...09:42
jtvmgz: "make install-dependencies"09:42
mgzI should fix my deployment script.09:42
jtvAnd pull regularly.09:42
jtvYou don't want your branch to fall out of date or you'll be building merge conflicts into branches right from the start.09:43
mgzI should change it to use bzr cat lp:maas rather than peeking at my stale branch on this box09:43
jtvWe've got a pretty high ratio of changes to code right now.09:43
mgzthe copy I'm working with is up to date, but what I'm telling cloud-init to install is based off a copy of maas I'm not using09:44
jtvAh09:44
mgzhence forget to keep up to date.09:45
mgzstill my fault though :)09:45
jtvallenap: want to talk cli?09:47
allenapjtv: Sure.09:47
allenapjtv: Shall I call you?09:47
jtvCall?09:48
allenapjtv: It's probably quicker, but here is okay I guess.09:49
jtvI mean, do we do a hangout?09:49
jtvI'm starting one09:50
jtvallenap: https://plus.google.com/hangouts/_/aaedfd9ecc51ae502006d3c55aa21e6680992c86?authuser=0&hl=en-GB09:50
mgzjam: the maas lander really hates you...09:52
mgzit's not done that "additional revisions which have not been approved" thing to me once...09:52
mgzlet's see if I've jinxed myself...09:53
mgzhm, is there some way I get get django up on localhost without all the rest of the junk for make run? celery is unhappy now...10:02
mgzbecause there's no 'maas' user... what should have created that?10:05
rvbajtv: what mgz is experiencing looks like a fallout from what you're doing to fix the start-cluster script…10:11
jtvotp10:12
rvbajtv: I'm just guessing here but if we try to use the maas user to run celeryd on a dev instance, that will be a problem.10:12
jtvmgz: try bin/maas-provision start-cluster-controller http://localhost:5240/ -u `pwd` -g `pwd`10:13
jtvrvba: that's right, so don't do that.  Use your real identity.10:13
jtvmgz: sorry, not pwd10:13
jtvthe other one10:13
jtvwhoami10:13
jtvand yes I'm conflating your user with your group of the same name and just guessing that there is one10:14
mgzubuntu:ubuntu so yup.10:14
rvbaservices/cluster-worker/run will probably need to be fixed then.10:14
jtvDoes services/cluster-worker/run use start-cluster-controller already?10:14
rvbaYep, since last week.10:14
jtvallenap had a cute idea: default not to maas but to the _current_ user, except when that's root.10:14
mgzfor the record: <http://pastebin.ubuntu.com/1257666/>10:15
jtvUnless & until we do that, pass explicit user/group.10:15
jtvYes, that's what happens if you leave it to default to "maas" on a system that doesn't have a maas user.10:16
jtvrvba: I'll update the "run" script.  Thanks for updating that btw.10:16
jtvI mean, thanks for the update _before_ the one I'm about to make.  :)10:16
rvbajtv: great.10:16
mgzI take it the early errors in logs/webapp/current are inconsequential as it's repeated until it works?10:16
mgzokay, this is all quite nice10:37
jtvrvba: aigh.  One problem with updating the cluster services file: start-cluster-controller will no longer exit!  Do we have some utility at hand to wrap it in a daemon?10:47
jtvstart-stop-daemon -b?10:48
rvbajtv: not sure I follow, you've removed start-cluster-controller?10:49
jtvNo, but it will no longer exit.10:50
jtvIt'll keep running forever.10:50
jtvWell, waiting.  :)10:50
jtvAnd I'm guessing that our services machinery will want it to run in the background.10:50
rvbaQuite the opposite I think, hence the usage of 'exec' in these files.10:51
jtvAh, that makes sense!10:51
* jtv gives it a whirl10:52
jtvI'm ditching the fghack as well10:52
jtvrvba: this is almost funny.  With my change, "make run" seems to add a new celery (with 5 processes) every 10 seconds!11:07
rvbajtv: the supervise stuff is probably unable to detect that the process is running ok, so it relaunches one every x seconds.11:07
jtvYeah I guess.  :(11:08
rvbajtv: IIRC that's precisely what happens when the process launched by the 'run' script gets daemonized.11:08
jtvI wonder if re-introducing fghack helps then...11:10
jtvThe region worker otoh isn't shutting down properly.11:10
jtvNow breaks my wooden shoe.11:11
jtv(The literal equivalent of which is a Dutch expression for "WTF????")11:12
jtvfghack seems to fix it.11:12
jtvOh well.  Don't question the oracle.11:13
jtvI'm out.  allenap, could you have a look at my updated MP?11:35
mgzjtv: lgtm, can I land it while you sleep?11:44
jammgz: well, I often do a quick 'cleanup' patch, and then try to submit it.11:52
jamix11:52
jammgz: however, gavin mentioned that you have to wait for the mp to update before you mark it approved. However, I waited until it saw 'Unmerged revisions' but not before the diff was updated.11:55
jamit is possible that you need to sing and dance and sacrifice a chicken to get things to go any faster.11:57
jamI haven't quite worked out the exact syntax yet.11:57
mgzhm, that may be it, I tend to not mark approved instantly after pushing tweaks12:00
jammgz: well 'instantly', I do the tweaks and want to submit it before I forget about it and it sits for a day. I know I need to wait 'a little' but trying to figure out when I can push it out of my mental context is difficult.12:01
mgzwrite a local launchpad script that sleeps for five mins then flips the status :D12:01
jammgz: at least with feed-pqm /pqm-submit we would check that the branch tip was up to date, and not have to wait for the N async processes that go from branch tip changed, to MP noticing, to approve state.12:02
allenapmgz: I'll land jtv's branch in his absence.12:07
jamjelmer: both of my api branches have landed12:07
jelmerjam: \o/12:07
mgzallenap: he seemed to have remained awake just long enough to re-mark it approved :)12:08
mgzah, there is one of his pending still though12:09
allenapI think we're concerned about different branches :)12:09
mgzthe scary celery exec change I wouldn't touch :)12:10
jammgz: hows the search stuff looking?12:18
mgzI have html edited, need to wire everything up12:19
jammgz: did you know about huw's branch?12:19
jammgz: https://code.launchpad.net/~huwshimi/maas/hardware-search12:19
jam(It was on the kanban board, but I realize you don't actually see that stuff)12:19
mgzjam: no...12:20
mgzI shall get that and examine12:20
mgzokay, that seems all reasonably straight forward12:23
jammgz: I also should have some mockups for what it should look like12:24
jamlet me dig them up12:24
mgzin lynx? :)12:24
mgzI'll socks proxy so I can see the graphical view :D12:25
jelmermgz: have you tried using w3m or links? They should be able to do graphics if you have a framebuffer..12:25
jammgz: https://docs.google.com/a/canonical.com/drawings/d/1AH_8gCyTYG6LfbjzYYjJQowpT-m8wq_P3oV5kL2XrmY/edit12:25
jamand: https://docs.google.com/a/canonical.com/drawings/d/1NnBi_3bzpFjhbC6X8KYbh40qbj-FfxzlPJ40TbmXTGE/edit?pli=112:25
mgzta12:25
jamthe idea is that the search bar at the top should be everywhere, but when you get to the nodes/ page itself, it moves down into a larger view inside the page12:26
jamin *my* head the main change is for that page to take an optional 'constraints'  (search?) parameter12:26
jamwhich then gets parsed from text form, into a dict, and essentially handed off to the filter code we just landed12:26
jammgz: note that some of that view will not be present in 12.10, some of it is 'future work'. Like if we don't have pagination yet, etc.12:27
smoserjtv, did you sort the dhcpd.conf issue i was hitting yesterday?12:28
jamand the Node list/search should use the same: 'macaddress (hostname)' that we currently have, rather than the 2 columns that is on that page.12:28
smoserah. i see. reading up.12:28
jamrvba: so I can easily test that the helper function returns an indication that the task needs to be retried. Is there a way to test that the retry is done?12:42
jamOr do you just patch out retry and assert that it is called?12:42
rvbajam: no, you can test the retry for real, see test_rndc_command_is_retried_a_limited_number_of_times in src/provisioningserver/tests/test_tasks.py12:44
rvbajam: that's why we've created MultiFakeMethod :)12:44
jamrvba: so that test would succeed if the code just raised the exception on the first try12:48
jamI don't see it asserting that it is actually retrying12:48
jamI see that it is asserting it does eventually stop retrying12:48
jamah, I guess that is 'can_be_retried'12:48
rvbajam: indeed, but you're right it's a bit implicit.12:49
jamrvba: so how often is the refresh_secrets called?12:50
jamrndc only waits 20s before failing12:50
jam(10 tries at 2s each)12:50
rvbajam: the main reason for retrying the rndc command is that bind sometimes takes some time to start up.12:51
jamrvba: so the issue seems to be that it is possible for a queue to exist, but not have credentials to actually talk to the mass server yet12:51
jam(mass_url may not be set, you may not have creds, you may not have a nodegroup.uuid yet)12:52
rvbajam: I assume s/queue/nodegroup/12:52
rvbaQueues are simply created on demand.12:53
jamrvba: so I was told to do "async_queue(queue=nodegroup.uuid)"12:54
jamin order to get them started on the right provisioning_server12:54
jaminstead of running it on the master12:54
jam(each nodegroup's controller is meant to refresh the tag matching for the nodes under its control)12:55
jam(so that master doesn't have to update 100,000 linearly, but each cluster can update their <10k nodes in a row.)12:55
rvbaMakes sense.12:55
jamrvba: I would have thought that you need a queue around to make sure it is running on the right machine/worker/something12:55
rvbajam: yeah, but queues are autocreated when the region sends tasks to it or when a cluster connects to a queue.12:56
rvbaCalling task.apply_async('my queue') will create 'my queue' and send task to it.12:57
rvbaThen the task just sits there until a celery worker feeds from that queue.12:57
rvbaAll I'm saying is that you don't need to create the queues.12:58
jamrvba: ok, but that means we need to call refresh_secrets before we call the task we want to do?13:01
rvbajam: indeed.  refresh_secrets gives to the cluster worker the credentials it needs to access the API.13:02
rvbajam: glancing at the code, that is only called when the cluster controller starts up.13:02
jamrvba: so the question is still open whether we can trust that the controller has creds or not. The other api has tests that assert it 'does nothing if it doesn't have creds'13:03
jamand we *need* the work to be done13:03
jamrvba: I got DC'd for abit.13:05
roaksoaxmorning13:05
rvbajam: you're right.  Right now we trust that refresh_secrets was called when the cluster got started.13:05
jamso the idea of putting in retry is that we need it to run once we get creds, or we can trust that we always have creds, or we always push creds out before we ask for the work to be done.13:05
rvbaI understand.  This is a tricky problem.13:08
jamrvba: if we could get an error message back we could go with that13:10
jamor we can create our own 'queue' of work that is remaining to be done13:10
jamin the db tables13:10
jamand then create another async process that makes sure the queue is being worked on.13:10
jam(since we can assert the transactional nature of the primary db, but not really the rabbit queues)13:10
rvbaThat is exactly what celery-django does for you :)13:10
rvba(A package we don't use atm)13:11
roaksoaxsmoser: around already?13:11
jamrvba: so should we try to do that, or should we just live with 'things may get out of date and we should add a "manuallly trigger a refresh"' in case stuff ever gets dropped. ?13:12
smoserroaksoax, here.13:12
roaksoaxsmoser: where do you tell commissioning to install packages?13:12
roaksoaxor, extra packages13:12
roaksoaxsmoser: we need freeipmi-tools13:12
jamwithout maas_url, there isn't even a place to put an anonymous 'something failed' handler.13:12
jamthat the celery worker can inform us it needs to be retried.13:12
rvbaIndeed, chicken and egg problem.13:13
jamrvba: right, so either we write it as 'assume something has failed until the worker has said it succeeded' or ?13:15
jamand if we do that, what task sits around making sure it gets retried?13:15
jamyou can poke at the API to get workers refreshed13:15
jamand we can just call that immediately before we try to do work13:15
smoserroaksoax, looking13:16
smoserroaksoax, also13:16
smoserhttps://bugs.launchpad.net/maas/+bug/106094213:16
ubot5Ubuntu bug 1060942 in MAAS "maas-cluster-celery job dies" [Undecided,New]13:16
rvbaYeah, that sounds like a good way to do that quickly.  Long term, it would be good to use celery-django for that kind of stuff.13:16
rvbaIf celery-django gives us that possibility that is.  I haven't looked into it really, but I think that's what it does: track the status of the tasks within the db.13:17
smoserroaksoax, sorry for the slow response.13:20
smoseryou have to do it in that script there (./etc/maas/commissioning-user-data)13:20
smoserthat script gets sent to cloud-init verbatum as user-data13:20
smoser(which it executes because of '#!')13:21
smoserso you'll have to 'apt-get update' there13:21
roaksoaxsmoser: ok cool thanks13:22
smoserroaksoax, i'm kind of ton on this.13:22
smoseron one side i feel ike i should put more into the imgaes13:22
smoser(like maas-enlist and freeipmi-tools)13:23
smoserbut on the other, we have to be able to deliver updated versions of those *anyway*13:23
roaksoaxyeah13:23
smoserso 'apt-get update && apt-get install' is pretty much required anyway13:23
roaksoaxputting them would indeed speed up the process, but we depend on they being updated13:24
roaksoaxat least maas-enlist13:24
smoserright.13:24
smoserso it wouldn't really speed it up13:24
smoserin the end13:24
smoserwell, the deps would help if they're extensive13:24
smoser(but they're not i dont think)13:24
smoserrvba, roaksoax do you have thoughts on https://bugs.launchpad.net/maas/+bug/106094213:25
ubot5Ubuntu bug 1060942 in MAAS "maas-cluster-celery job dies" [Undecided,New]13:25
smoserit looks to me like we've tried twice now to do this and both have failed13:25
smoseri'm not exactly sure why celery can't setgid when it starts as root13:25
roaksoaxsmoser: i'm looking at it now13:26
roaksoaxsmoser: the upstart jobs used to set the gid/uid before, when running celerd directly13:27
roaksoaxceleryd*13:27
rvbaBut now there is communication phase before we can start the celer worker for the cluster controller.13:27
roaksoaxsmoser: which is what maas-region-controller does13:28
rvbasmoser: could it be caused by a limitation in upstart somewhere?13:29
roaksoaxrvba: so upstart stats the script as root, the root check in the wrapper passes just fine13:29
jammgz: so I'm heading out for the day, feel free to push up work in progress if you want me to take a look at it tomorrow before you wake up.13:30
jamor if there is anything that is blocking you now?13:30
jamrvba: nodegroup-ui branch was a merge conflict, not a test failing (from what I can tell)13:31
jamrvba: but of course, I read it wrong.13:32
jamI agree, the build says 'SUCCESS' at the end.13:32
mgzjam: thanks, will do13:32
mgznothing blocking atm13:32
roaksoaxrvba: what log does cluster-controller creates? celery-cluster.log?13:32
jammgz: we have roughly 1 more day to land it, so I'm trying to make sure everything is as smooth as possible.13:33
rvbajam: I don't see the merge conflict but I'll merge trunk anyway, just to be sure :).13:33
rvbaroaksoax: /var/log/maas/celery.log13:34
jamrvba: well, be careful, since otherwise maas lander will reject it because the branch changed after it was approved :)13:34
smoseri am confused on whats doing this13:34
smoseri added13:34
smoserprint("user=%s group=%s uid=%s gid=%s curuid=%s curgid=%s" % (user, group, uid, gid, os.getuid(), os.getgid())13:34
jamI think I just misread the error message ,which looks to be 100% bogus as you mentioned.13:34
smoserright before the fork/seuid/gid13:34
smoserand i see13:34
mgzjam: I was counting thursday and friday as two, but we want to be all done by tomorrow?13:35
jammgz: we need some time to get it into packaging,etc.13:35
smoseruser=maas group=maas uid=113 gid=120 curuid=0 curgid=013:35
jamand jelmer is gone on friday regardless.13:35
smoserso at that point i'm running as 0:0 and trying to go to 113:12013:35
jamso I would say "1-ish"13:35
mgzokay, 1-ish it is.13:36
roaksoaxsmoser: yeah that seems to be the issue13:36
smoser?13:36
smoseri'm saying it looks right13:36
roaksoaxsmoser: setgid13:36
smosercuruid, curgid = (0,0)13:37
jammgz: did you get hardware_details for sampledata ? (not yet, I believe we deprioritized it, vs tag data and having search implemented)13:37
smoseri'm just trying to drop gid to 120.13:37
smoserwhy would i not be able to do that?13:37
roaksoaxsmoser: i commented out the setgid and the process starts just fine13:37
smoserhttp://stackoverflow.com/questions/4692720/operation-not-permitted-while-dropping-privileges-using-setuid-function13:37
mgzjam: nope, but could trivially do that (something for mem and cpu_count may also be useful)13:38
smoserwait. i'm confused by that though.13:38
roaksoaxsmoser: [2012-10-03 09:39:14,571: WARNING/Beat] DBAccessError: (13, 'Permission denied')13:39
smoserah13:39
smoserduh13:39
smoserstuipd13:39
roaksoaxsmoser: that's besides the gid/uid thing13:39
jammgz: so if you get search up and nobody reviews it, then look at the sampledata stuff :)13:41
jamanyway, I'm off, have a good evening13:41
mgzlater!13:42
smoserroaksoax, https://code.launchpad.net/~smoser/maas/lp1060942/+merge/12776713:44
smoserroaksoax, where did you see your error ?13:48
roaksoaxsmoser: it is approaved13:49
rvbasmoser: if that really fixes the problem, you might want to change to comment also :)13:49
roaksoaxsmoser: in /var/log/celery.log13:49
roaksoaxrvba: where does celery stores its db?13:49
roaksoaxor beat?13:49
smoseryeah, i'm seeing that now13:49
rvbaroaksoax: don't know what the default is… hang on.13:50
roaksoaxhttp://comments.gmane.org/gmane.comp.python.amqp.celery.user/237513:51
roaksoaxsmoser: ^^13:51
rvbaroaksoax: looks like the default is to use /var/run/celerybeat-schedule13:52
roaksoaxrvba: so how can we do this: http://comments.gmane.org/gmane.comp.python.amqp.celery.user/237513:53
rvbaroaksoax: and the region is told explicitly to use /var/lib/maas/celerybeat-region-schedule in the upstart script.13:53
rvbaroaksoax: are you seing that error?13:54
smoserso what is it supposed to be?13:54
roaksoaxrvba: yeah I'll test/fix13:55
roaksoaxsmoser: it works13:55
roaksoaxrvba: it works :)13:55
smoserwhat do we need to pass?13:55
smoserwhats the value we need to pass ?13:55
roaksoaxsmoser: http://paste.ubuntu.com/1258008/13:56
smoserare we easily able to pass that in ?13:57
smoseras an argument13:57
roaksoaxsmoser: yes, we pass that in the upstart job13:58
roaksoaxrvba: should be just pass "/var/lib/maas/celerybeat-region-schedule" or should that be detected automatically13:59
roaksoaxrvba: it guess it would break make run?13:59
rvbaroaksoax: you're right, we can make it use a param that would be different in the local config.14:00
rvbaroaksoax: I can put together a branch to do that.14:00
roaksoaxrvba: awesome! that'd be great14:01
smoserrvba, that just gets us to the next error14:01
rvbasmoser: what's the next error?14:01
smoserhttp://paste.ubuntu.com/1258023/14:01
rvbasmoser: the error in provisioningserver.tasks.upload_dhcp_leases ?14:02
smoserbut at least the celery seems up at that point14:02
smoserit does suck that it dies permenently on that error14:02
smoserhm..14:02
rvbaYeah, there is a task failing, we know about this one.14:02
smoseryeah, this is really fragile14:02
rvbaWe just need to make the task code deal with the fact that the lease file might not be there.14:02
smoserbut i'm not sure why upstart isn't re-starting it14:02
rvbaWe're aware of this one.14:02
rvbajtv: ^ ;)14:02
jtv?14:03
rvbaThe dhcp task failing because the lease file is not there :).14:03
jtvThe cluster controller upstart job can't really respawn because the typical reason for failure is that the cluster controller has been rejected.14:04
rvbaNot urgent but if would be good to tweak the task so that it could cope elegantly with the situation.14:04
jtvI agree.14:04
rvbaGot task from broker: provisioningserver.tasks.refresh_secrets14:04
rvbaTask provisioningserver.tasks.refresh_secrets[d8c820a8-d7cd-453b-9816-747ee2556927] succeeded in 0.201004981995s14:04
rvbaContact with the region controller was ok.14:04
smoserjtv, i dont understand the comment above.14:12
smoserupstart will handle that most likely14:12
smoserif the job dies 5 times in 5 seconds, upstart will give up on it14:13
jtvAh good14:13
smoserif it doesn't hit that threshold, then, who cares.14:13
jtvJust so long as Right14:13
jtvAhem.14:13
jtvRight.14:13
jtvJust so long as it doesn't loop wildly.14:13
smoserwho cares if a daemon keeps spawning that shouldnt14:13
smoserit was configured wrong by the admin explicitly14:13
jtvIn this case we do care, because it makes API requests.14:13
smoserin this case.14:13
smoserso the way you could do this14:14
smoseris a pre-start job14:14
smoserthat does the check for "am i accepted"14:14
smoserand exit 0 if "no"14:14
smoser( i think that sthe right symantics)14:14
smoserbut then that would only run once14:14
jtvWe considered something like that, but decided it had to be this way.14:14
smoserwell then you need to make it re-spawn14:15
jtvIt needs to repeat in some cases.14:15
smoseror its just going to die all the time.14:15
smoserupstart respawns daemons for good reason. its generally considered the right thing to do.14:15
jtvWell like I said, it's fine here too -- as long as it's not a tight endless loop.14:16
smosersee 'man 5 init' look for 'respawn'14:16
smoserhow did you disable this ?14:16
jtvI didn't.14:17
smoseris it exiting success on stack trace?14:17
smoseri'm confused why it isn't getting respawned14:17
jtvI don't have the answer.14:17
jtvBut Julian told me it wasn't set up to respawn.14:18
rvbaroaksoax: https://code.launchpad.net/~rvb/maas/explicit-beat-schedule/+merge/12777514:21
smoserroaksoax, do you have ideas?14:22
allenapsmoser: respawn is not in maas-cluster-controller.maas-cluster-celery.upstart?14:23
allenapsmoser: Also, that setuid/setgid branch slipped in; I guess Tarmac got it too quickly.14:23
smoserallenap, bah.14:23
smoseri will do a revert branch14:24
allenapsmoser: If I get time I'll write a test for this ordering.14:24
allenapsmoser: In fact, I'll do the revert if you want, and add a test at the same time?14:25
roaksoaxsmoser: why isn't what respawned? maas-cluster-celery due to the leases stack trace?14:25
roaksoaxok I need to test the IPMI so gonna setup my network properly14:26
smoserhttps://code.launchpad.net/~smoser/maas/revert-1151/+merge/12777814:26
smoserroaksoax, thats what i'm confused by14:26
smoserallenap, well, there is the revert.14:26
smoserit doesn't respawn.14:27
roaksoaxsmoser: why are oyou confused by it?14:27
smoserwhy does it not respawn14:27
smoserhttp://paste.ubuntu.com/1258070/14:27
smoserdefault is to respawn14:27
smoser(but i was explicit and added it there)14:28
roaksoaxsmoser: hold on, you want it to respawn due to the failurs of the leases file?14:28
smoserit should respawn14:30
smoserits an upstart job14:30
smoserand if it dies, it should not be fatal14:31
roaksoaxsmoser: it is not fatal14:31
roaksoaxsmoser: i see that error over and over14:31
roaksoaxsmoser: but celery doesn't die14:31
roaksoaxsmoser: it keeps running14:31
roaksoaxit only error's out14:31
smoseri dont think so.14:31
smoserit dies14:31
rvbaYeah, it's just a failed task.14:31
smoserthe job dies at least.14:31
rvbaNothing critical for celeryd.14:31
smosersudo status maas-cluster-celery stop/waiting14:32
roaksoaxsmoser: did you psas the --schedule?14:32
smoserhttp://paste.ubuntu.com/1258075/14:33
roaksoaxsmoser: and removed the old pyc files?14:33
smoserit dies now because of the dhcp14:33
smoserroaksoax, wouldn't you expect to see that pid change there?14:35
roaksoaxsmoser: oh i thnk what it is14:36
roaksoaxsmoser: jtv reported and error on which upstart couldn't track all the instances of celeryd14:36
roaksoaxsmoser: that might be related14:36
jtvIt wasn't quite that.14:36
jtvUpstart couldn't track the fork.  As it turned out, there were several forks before the one that actually spawned the celeryd.14:37
jtvSo upstart couldn't track any instances at all.14:37
smoserok. i'll open a bug here.14:38
smoserbut this is fairly severe.14:38
roaksoaxsmoser: https://pastebin.canonical.com/75796/14:38
smoseror am i incorrect14:38
roaksoaxsmoser: i think i know why14:38
jtvI'm getting bits and pieces of the conversation... what is the problem?14:39
smosermaas-cluster-celery is fragile14:39
roaksoaxsmoser: becuase maas-provision start-cluster-controller simply tells it to start celeryd and then it returns and exits14:39
smoserin that if anything goes wrong it will die and never start14:39
smoserrequiring a human14:39
roaksoaxsmoser: so maas-provision start-cluster-controller is not really a daemon14:39
jtvroaksoax: it no longer does that14:39
jtvIt keeps running for as long as celeryd does.14:39
jtvWell, it execs celeryd once it's got its approval etc. from the server.14:40
roaksoaxjtv: right, but for some reason the way it is started seems to not be tracked by upstart, and i think it is because it is not a daemon14:42
jtvStill not being tracked?  :(14:42
jtvAnd it's so simple now!14:42
jtvAs it stands, "maas-provision start-cluster-controller" should only exit if (a) the cluster controller has been rejected, or (b) celeryd exits.14:44
jtvWow, that's a pretty lame slogan: "Stiebel Eltron.  Originally German."14:45
smoserhm..14:45
jtvIt's like saying "Pizza Hut.  One of us saw a pizza once and liked it so much we named the company after it."14:46
smoseri must be misunderstanding something of upstart14:46
smoserroaksoax, well.14:56
smoserit doesn't respawn because you need the 'respawn' if you want it to14:57
smoserie, you need both:14:57
smoserrespawn14:57
smoserrespawn limit 5 6014:57
roaksoaxack14:59
smosersuck. why didn't i realize that.14:59
smoseranyway14:59
roaksoaxsmoser: why did you revert the setgui/setuid thing?15:00
smoserunless allenap was going to do it.15:00
smoserallenap, roaksoax just ack https://code.launchpad.net/~smoser/maas/revert-1151/+merge/12777815:01
smoserto get my stupid change out15:01
allenapsmoser: +1 and approved. I'm writing a test for that code; I've noticed another bug at the same time.15:02
smoserjtv, so you're working on something to make the dhcp work?15:02
smoserrvba, do you know anything about mirrors in maas?15:11
rvbasmoser: not really, I just know that we have a config option named 'keep_mirror_list_uptodate' that is not used anywhere yet :/15:12
smoserright. and i saw a list of mirrors in the ui15:13
smoserthat contained 'archive.ubuntu.com'15:13
smoserand could not be selected or changed15:13
smoser:)15:13
smoseri'm thinking we're kind of SOL on local mirrors at the moment.15:13
rvbasmoser: that can be changed and updated. See the 'update from' dropdown.15:15
rvbasmoser: but the 'Default distro series used for deployment' thing is positioned right in the middle of it.  And that's confusing.15:15
smoserhm.. i was just probably remembering incorrectly15:16
roaksoaxDaviey: do you happen to know how can I force IPMI to obtian a new IP address? (it is set to DHCP but doesn't get address)15:38
roaksoaxDaviey: nevermind i got it15:46
Davieyroaksoax: how did you do it?15:47
roaksoaxDaviey: I set it to static, and then back to dhcp and it renewed it15:47
Davieyhah, that is how i did it :)15:48
roaksoaxDaviey: :)15:48
roaksoaxsmoser: http://paste.ubuntu.com/1258243/ so look at the first part15:53
roaksoaxadding the apt stuff that way?15:53
smoserhttp://paste.ubuntu.com/1258248/15:55
smoserroaksoax, yes. basically right, but the above is safer15:55
smoser(and, fwiw, the '-q' to apt doesn't really mean 'quiet' it means "produces output suitable for logging"15:56
smoserwhich is what we'd want)15:56
roaksoaxsmoser: ok cools15:59
mgzyup, basically not using \r to print the same line multiple times with progress updates15:59
roaksoaxsmoser: alright, so it looks something like: http://paste.ubuntu.com/1258286/16:15
smoseryou have to quote fargs at 4216:17
smoseras it probably contains spaces, right?16:17
smoserand you didnt patch signal all the way, right?16:18
roaksoaxsmoser: right :)16:25
smoserdid you want help with that16:26
roaksoaxsmoser: btw... --power-settings is json16:31
roaksoaxsmoser: it reutrns this: ('IPMI', '{"power_address": "192.168.2.111", "power_pass": "8ruHtjzpGdU", "power_user": "maas"}')16:32
smoserright.16:32
roaksoaxsmoser: so like this? signal "$fargs" OK "power-settings sent [$power_settings]"16:35
smoserrbasak, how easily could you test something16:38
smoserspecifically https://code.launchpad.net/~smoser/maas/preserve-sources-list/+merge/12782516:39
smoserroaksoax, let me look.16:41
smoserallenap, or rvba, i'd appreciate your input on that merge ^16:41
smoseri'm sure the test could be cleaned up16:41
rbasaksmoser: my test node is being shipped to the MAAS QA lab now!16:42
rbasaksmoser: I need to switch my testing to another node, which shouldn't take long, but I have a hard stop soon. I can test it for you in the morning though? Normally it'd be about two commands!16:43
smoserhm..16:43
smoserrbasak, how were you telling cloud-init preserve_source_list off16:43
smosererr... on16:43
* rbasak looks for the patch16:44
roaksoaxallenap: smoser how do you debug? something seems to have failed :(16:44
rbasaksmoser: in contrib/preseeds_v2/enlist16:45
rbasaksmoser: +apt_preserve_sources_list: true16:45
rbasaksmoser: that's all. And after deployment I fix manually. I presume this still breaks juju though.16:45
smoserah.16:45
smoserwhat breaks juju?16:46
smoserthis would fix it.16:46
rbasaksources.list being wrong after install16:46
rbasaksince it doesn't use that file on boot16:46
rbasakYour fix is a proper fix I presume.16:46
smoserright. my patch wuld make juju work.16:46
smoserright.16:46
smoserbut, actually, doe not fix enlistment16:46
smoseror commissioning16:47
smoser(but those are sort of fixed for you already)16:47
smoserby the ephemeral nodes having newer cloud-init16:47
roaksoaxsmoser: https://pastebin.canonical.com/75814/16:47
smosercarp16:48
smoseryou're out of memory16:48
smoserhow much memory do you have there?16:48
roaksoaxsmoser: memory as in ram?16:48
roaksoaxah right16:48
roaksoaxdah16:48
smoseras in ram16:48
smoser:)16:48
roaksoax25616:48
smoseryeah.16:48
smoserdo you want to try to work around this?16:49
roaksoaxsmoser: it is a VM16:49
roaksoaxsmoser: so i'm just increasing memory16:49
smoseroh. ok.16:49
smoseri thought it was one of your servers16:49
smoserbut your vm wont have ipmi16:49
roaksoaxsmoser: nope, just testing whether the scripts are running properly16:50
smoserright16:50
roaksoaxsmoser: btw.. you happen to have the link for the pastebin you got yesterday on how to send power settings back to MAAS?16:50
smoserroaksoax, i dont remember pastein16:51
smoserfrom you?16:51
roaksoaxsmoser: from matsubara i thnk16:51
roaksoaxmatsubara: ^^ do you know how to send power settings to maas?16:51
smoseroh. i think that was something else.16:51
matsubara maas-cli api maas node update node-7d828c3e-0902-11e2-8461-00e081ddd1cf power_type=ipmi power_parameters_power_address=192.168.22.33 power_parameters_power_user=root power_parameters_power_pass=ubuntu16:53
matsubararoaksoax, ^16:53
roaksoaxmatsubara: thanks16:53
matsubaranp16:53
smoserroaksoax, but that doens't help us.16:54
roaksoaxsmoser: i know :(16:55
roaksoaxsmoser: shouldn't we just send it as the other data you are sending?16:59
smoserroaksoax, right. your patch looks good. we just have to tell that 'maas-signal' python script how to send the ppower settings.17:01
roaksoaxyeah17:02
roaksoaxsmoser: we are still going to do the enlistment setting temporary ipmi credentials right?17:18
smoserwell, we dont have a way to post those at the moment.17:19
roaksoaxallenap: ^^17:20
roaksoaxsmoser: can i ssh to the commissioning images?17:23
smoseri wouldnt think so.17:25
smoserroaksoax, what i'd suggest is preparing the ephemeral image to let you in.17:26
roaksoaxsmoser: will do17:26
roaksoaxsmoser: so something is wrong and never returns from commissioning, but I think i know why17:26
roaksoaxsmoser: and don't have console nor a monitor :(17:27
smoserwhat system?17:27
smoserphysical system?17:27
smoserdo those hp microservers ipmi have remote serial console ?17:27
Davieyseemed not.17:29
roaksoaxsmoser: and the problem is that there's an issue with the kernel not correctly recognizing ipmi port and stuff17:33
smoseroh joy.17:33
Davieyyou have to modprobe with custom stuff17:35
Davieymodprobe ipmi_si type=kcs ports=0xca217:36
DavieyBut even so, with that - i didn't think you got console?17:36
roaksoaxDaviey: yeah i added that but doesn't seem to be working,17:43
roaksoaxsmoser: ok it says that module ipmi_si does not exist on /etc/modules18:07
roaksoaxsmoser: which basically tells us that we can't access the bmc :)18:08
smoser?18:08
smoserwhere would that module come from?18:08
smoserhttps://code.launchpad.net/~smoser/maas/preserve-sources-list/+merge/12782518:08
smoser^ comments nplease!18:08
roaksoaxsmoser: i'd say it is in the kernel18:09
smosersmart @$!$18:11
smoserwhat says that, roaksoax18:11
roaksoaxsmoser: rmmod but i just realizd i probably dont even need it if the modufle hasn;t been loaded18:13
smoserso modprobe it?18:13
smoserfeel free to do that in the commissioning script18:13
roaksoaxsmoser: my default?18:14
roaksoaxsmoser: i alreayd did that but was removing it first, before loading it18:14
roaksoaxthe thing is if the module is loaded already, then you need to remove it to load it again18:14
roaksoaxso the parameters take effect18:14
smoseryes.18:16
smoseryou do.18:16
smoserright.18:16
smoserso that sucks.18:16
roaksoaxsmoser: so it commissioned this time but the it didn't seem to have done the ipmi stuff18:17
roaksoaxsmoser: so how can I enable ssh access? besides of course ssh-import-id18:17
smoserhttp://bazaar.launchpad.net/~maas-maintainers/maas/trunk/revision/108918:18
smoserroaksoax, what i'd do is just add another user18:18
roaksoaxoh wait i guess i can just install openssh-server18:18
smosermount loopback, add user, add ssh keys umount18:18
smoserits installed,18:18
smoserbut by default you wont have access as there is no passowrd and no user and no ssh keys installed18:18
smoser:)18:18
roaksoaxsmoser: right, so can't i create a user from the commissioning script?18:19
smoserah. yeah, you could easily enough.18:19
roaksoaxsmoser: does it run as 'ubuntu' user?18:19
smoserthat runs as root18:19
roaksoaxack!18:20
roaksoaxsmoser: and how to prevent it from powering  off?18:22
roaksoaxtrying to avoid having to mount it :)18:24
smoserroaksoax, sleep!18:27
roaksoaxsmoser: yeah figured :)18:27
smoseri think that script is what powers off, right?18:27
roaksoaxnope18:27
smoseryeah it does18:28
smoserjust disable the call to 'poweroff'18:28
smosererr.. 'shutdown'18:28
roaksoaxah lol18:28
smoser(it calls that via 'trap' on exit)18:28
smoserso just make that function return true18:28
smoseror false18:29
smoserdoesn't actually matter18:29
smoser:)18:29
roaksoax:)18:31
roaksoaxsmoser: i think what's happening is that it is not detecting IPMI so it continues normally18:31
roaksoaxsmoser: yeah so it seems it is an issue with ipmi detection18:37
roaksoaxsmoser: err modules18:37
roaksoaxsmoser: ok so what i'm gonna do is simply enable the modules from the script, and look for errors i guess18:40
smoserwell, load the module in the script18:41
smoserand generally ignore errors form that.18:41
roaksoaxsmoser: like "modprobe XYZ || true"18:41
roaksoaxsmoser: right so I was thinking, however, that what if the module is loaded (ipmi_si and needs to be reloaded?)18:42
roaksoaxdah18:42
roaksoaxduhh18:42
roaksoaxi guess i can do the same18:42
smoserroaksoax, the script isprobably not set -e18:45
smoseri dont generally do that.18:45
roaksoaxsmoser: so do you think it is safe to wait 10 seconds for an IP address?18:59
smosersafe as in too long or too short?18:59
roaksoaxsmoser: enough time18:59
roaksoaxyes18:59
smoseri think you  might as well give it 60 at least.18:59
roaksoaxsmoser: right, so I will give it 60 only if we had to change from Static to DHCP19:00
roaksoaxmakes better sense that way?19:00
smoserwell, if you're dhcping in general19:02
smoserwhy would you want to risk giving up early19:02
roaksoaxsmoser: right, but if the machine turns on and starts doing comissioning, then it would be safe to assume that IPMI has an IP address already19:03
smoserroaksoax, so then its dhcp request shoudl come back quickly19:07
smoseror dhcp was the wrong setting19:08
roaksoaxsmoser: right but for example, what happens if the image doesn't get an IP address quickly?19:11
roaksoaxsmoser: it sits and waits for one right?19:11
smoseran image?19:11
smoseras in what19:11
smosernode booting ephemeral image ?19:12
roaksoaxsmoser: yes19:12
smoserpxeboot annoying will time out19:12
roaksoaxsmoser: right, but ok, we pass pxeboot, the image also request the IP address right?19:13
smoserwell,the kernel does. but only for the interface that got pxe booted19:15
smoser(unless there is a bug)19:15
smoserand i guess its not really the kernel19:15
smoserits the initramfs19:15
smoserroaksoax, you know how to access the maas db?19:29
roaksoaxsmoser: yes, what do you need?19:30
smoseri ended up getting in with sudo -Hu maas psql maasdb19:36
roaksoaxsmoser: sudo maas shell sshoudl do19:36
smoseri needed db19:38
smosernot shell19:38
smoserhow do i put it into debug ?19:40
smoserthe api server19:40
roaksoaxsmoser: ah lol :) but you can modify the db from the shell19:40
roaksoaxsmoser: that i don't know19:40
roaksoaxsmoser: btw... how can we log all the stuff that the commissioning script does?19:40
smoserwell, the stuff it calls gets logged back to the server19:41
smoserwhich is what i'm testing)19:41
smoserand why i was asking such silly things19:41
roaksoaxah!! cool19:44
roaksoaxsmoser: btw.. i think overtime it would be a good idea to make the commissioning/enlistment use the proxy if available19:44
smoserproxy?19:45
smoserroaksoax, http://paste.ubuntu.com/1258781/20:01
smoser./maas-signal --config my.config --post power_type=ipmi --post "power_parameters={'blob': 'foo'}" WORKING "credsasdasdfasdfasdf"20:02
smoserif you call the pastebin there, like that. it will post power params successfully.20:03
roaksoaxsmoser: awesome thanks20:07
roaksoaxsmoser: i'm trying to debug why it is not executing the script automatically but it is doing what it should if I run it manually on the ephemeral image20:07
roaksoaxsmoser: any help would be appreciated20:08
roaksoaxhttp://paste.ubuntu.com/1258793/20:08
smoserroaksoax, http://paste.ubuntu.com/1258807/20:14
smoserreplace the one there with this paste ^20:14
smoserand then call it like20:14
smoser./maas-signal --config my.config --power-type=ipmi '--power-parameters={}' WORKING "credsasdasdfasdfasdf"20:15
roaksoaxsmoser: ok cool20:15
smoserroaksoax, how did you think it was going to run that20:17
smoserin your paste i dont see how it would run anything20:17
roaksoaxsmoser: yeah just noticed :)20:17
smoseroh.20:17
smoserah20:17
smoseradd_bin != add_script20:17
roaksoaxsmoser: yeah, so, how do you think it should be called?20:18
roaksoaxsmoser: should we do it from maas-signal directly?20:18
smoserhm..20:21
smoserroaksoax, i think i'd just have main in that top level script handle this specifically.20:24
smoserie, add_bin to add it20:25
smoserand then from main just invoke your script into a temp output20:25
smoserthen call maas-signal .... --power-par...20:25
roaksoaxsmoser: so maas-signal is only called once right?20:45
smoserno.20:45
smoserit gets called before and after each script with WORKING20:45
smoserand a status [1/X]20:45
smoserand then once at the end20:45
smoserit posts all the files20:45
smoserwith OK20:45
smoseryou can either add the params to the "OK" call20:46
smoseror make a WORKING call separate20:46
smoseri'm pretty sure the WORKING no longer do anything20:46
smoser:)20:46
smoserother than return "OK"20:46
smoserroaksoax, does that all make sense?20:56
roaksoaxsmoser: http://pastebin.ubuntu.com/1258893/21:02
smoserroaksoax,21:06
smosera.) uotes are bad21:06
smoserb.) you have to say "WORKING"21:07
smoser(and you have to say that before the one runs finished.21:07
smoserwell, quotes are bad is what it really amounts to.21:07
smoserthat make sense?21:08
smoseri have to run for a few hours21:09
roaksoaxsmoser: ok it works... but the settngs are not set in the server :S21:35
roaksoaxq21:35
smoserroaksoax turn debug on on the server21:37
smoserand you might see21:37
smoserhttp://paste.ubuntu.com/1258943/21:37
smoserand look in the DB21:37
smoserhttp://paste.ubuntu.com/1258944/21:38
smoserthat above is after i did:21:38
smoser ./maas-signal --config my.config --power-type=ipmi '--power-parameters={"a":"b"}' WORKING "credsasdasdfasdfasdf"21:38
smoserit insists that it be valid json21:38
smoserbut thats about it really21:38
roaksoaxsmoser: ah yes... i wonder why ...21:45
smoserroaksoax, any thing furthre?22:38
smoserwe're pretty close. definitely the cmdline tool there can now post data.22:39
smoserand it seems like you have a pretty good handle on getting it from the ipmi card22:39
smoserif you'd like we can add some data manipulation into the 'maas-signal' if that is easier to work with rather than you feeding it a json blob22:40
roaksoaxsmoser: yeah that could be even easier22:44
roaksoaxsmoser: and no i dind't get any further22:45
smoserok.22:45
smoseri have to run, and do not intend on being back tonight.22:45
roaksoaxsmoser: ack!22:45
smoseryou figure out something that makes sense to you as a format22:45
smoserhave your tool dump that to a file22:45
roaksoaxsmoser: i was just planning on "ip_address,user,pass"22:46
smoserand i can write something to read and post it back22:46
roaksoaxsmoser: as an argument22:46
smosersure. thats easy.22:46
roaksoaxand then split it in python and make it json22:46
smoseryep.22:46
smoserk. good night.22:46
roaksoaxnight22:49
bigjoolsmorning23:15
roaksoaxbigjools: howdy23:20
roaksoaxbigjools: the meta-data api stuff for the power parameters doesn't seem to be saving23:40
bigjoolsroaksoax: show me your client code23:41
roaksoaxbigjools: http://paste.ubuntu.com/1259090/23:41
roaksoaxbigjools: check the part of power_parms in maas-signal23:42
roaksoaxbigjools: I see things like : MAASAPIBadRequest: Bad power_type 'ipmis'23:42
roaksoaxif I sent a bad power type23:42
bigjoolspower_parms23:42
roaksoaxor bad parameters23:42
bigjoolsthere's your prob23:42
bigjoolsoh wait23:42
roaksoaxbigjools: power_parms is converted into json, then passed to the server, we know it gets there/... but doesn't save23:43
roaksoaxbigjools: here's the whole thing: http://paste.ubuntu.com/1259093/23:44
bigjoolsroaksoax: `are you putting it in POST data?23:45
roaksoaxbigjools: yes, it is exaclt the same as sending stuff the other commissioning stuff23:46
roaksoaxbigjools: so for example, if the power_params are not json, MAAS complains23:46
bigjoolsok so it gets that far then23:46
bigjoolshow do you know it's not saving ?23:46
roaksoaxbigjools: http://paste.ubuntu.com/1258943/23:47
roaksoaxbigjools: becuase they are not displayed on the UI for the commissioned node23:47
roaksoaxbigjools: so it doesn't auto start the node23:47
bigjoolsdoes everything else sent in the signal() get saved?23:48
bigjoolsfiles, basically23:48
roaksoaxbigjools: yes23:49
bigjoolswell that's odd then23:49
bigjoolscan you check the database itself23:49
bigjoolsrather than UI23:49
roaksoaxbigjools: >>> node.power_type23:50
roaksoaxu''23:50
bigjoolsargh23:50
bigjoolsI see the bug23:50
bigjoolsis node.power_parameters set?23:50
roaksoax>>> node.power_type23:51
roaksoaxu''23:51
roaksoax>>> node.power_parameters23:51
roaksoax{u'power_address': u'192.168.2.121', u'power_pass': u'KP0eOSy9Q9X4RZ', u'power_user': u'maas'23:51
roaksoaxit seems like it is23:51
bigjoolsthe code forgot to set the power_type23:51
bigjoolsyou can hack it locally to continue testing23:51
bigjoolsI'll land a  fix23:51
bigjoolssorry!23:51
roaksoaxbigjools: hehe no worries :)23:52
roaksoax bigjools now it workds :D23:53
roaksoaxyuaaaay23:53
bigjools\o/23:53
bigjoolsFWIW you should be able to call that any time to update power params23:53
roaksoaxbigjools: ok cool, but it is authenticated right?23:54
bigjoolsall metadata access is authed23:54
bigjoolswell, 99%23:54
roaksoaxbigjools: we might need that unauth for enlistment though :(23:54
bigjoolsroaksoax: didn't we have this discussion already? :)23:55
bigjoolsthe answer was that you don't, the oauth key should be passed to enlisting nodes23:55
roaksoaxbigjools: yeah but smoser had another idea that we discussed this morning23:55
roaksoaxbigjools: it is basically to only set a temporary user/password for IPMI and send it back in enlistent23:55
bigjoolsok23:56
bigjoolslet me think about this later23:56
roaksoaxsure, talk to smoser though :)23:56
* roaksoax has been all day working on this thing23:57
roaksoaxso happy it works now23:57
roaksoaxlol23:57
bigjoolsheh23:57
bigjoolsthis is why we are developers23:57
bigjoolsbetter than drugs23:57
roaksoaxlol23:59

Generated by irclog2html.py 2.7 by Marius Gedminas - find it at mg.pov.lt!