roaksoax | smoser: around already?> | 00:19 |
---|---|---|
smoser | here, roaksoax | 00:29 |
roaksoax | smoser: so we are gonna need to pass the "config" to my script, instead of harcoding it | 00:30 |
roaksoax | smoser: so I'm guessing I should pass the config to maas-signal, which will pass it to maas-autodetect-ipmi? | 00:31 |
smoser | what do you mean pass | 00:31 |
roaksoax | smoser: as it tell it where's the template / config | 00:31 |
roaksoax | as in* | 00:31 |
roaksoax | smoser: https://pastebin.canonical.com/75758/ | 00:31 |
roaksoax | smoser: that one, we need to tell the location of that template | 00:32 |
roaksoax | template/config | 00:32 |
smoser | i'm sorry. i'm still being dense. | 00:32 |
roaksoax | smoser: bmc-config --commit --filename /location/of/cofig/file.ipmi | 00:33 |
bigjools | hello guys | 00:34 |
roaksoax | bigjools: good morning | 00:34 |
bigjools | I might have done something bad to my debconf database, I am getting this trying to install the maas package after previously purging it: http://paste.ubuntu.com/1255593/ | 00:35 |
bigjools | (I need to sh -x the postinst to get that output, otherwise it's fails silently) | 00:35 |
roaksoax | bigjools: what new package are you trying to install? the latest I made available in experimental PPA? | 00:35 |
bigjools | s/it's/it/ | 00:35 |
bigjools | roaksoax: one I made myself yesterday | 00:36 |
roaksoax | bigjools: that's fixed then :) | 00:36 |
bigjools | it's not the same problem as the one you fixed | 00:36 |
bigjools | since I'd already found that and tried to fix it and then got held up by this :/ | 00:36 |
roaksoax | bigjools: are you referring that this doesn't fix it? http://bazaar.launchpad.net/~maas-maintainers/maas/packaging/revision/113 | 00:37 |
roaksoax | see the postinst | 00:38 |
smoser | daily ppa has maas-dhcp installable again | 00:39 |
bigjools | roaksoax: it's hard to see what the effect of the changes are - I was assuming you were referring to the fix for "service isc_dhcp_server stop" | 00:39 |
smoser | bigjools, just curious, what does "fix committed" / "fix released" mean in maas ? | 00:39 |
smoser | bigjools, he was referring to the celery postinst | 00:40 |
smoser | i think | 00:40 |
bigjools | smoser: not a a lot. Tarmac changes bugs to fix-committed but we don't distinguish that with fix-released on trunk | 00:40 |
smoser | https://bugs.launchpad.net/ubuntu/+source/maas/+bug/1050523 | 00:40 |
ubot5 | Ubuntu bug 1050523 in maas (Ubuntu) "maas kernel cmdline must include iscsi_initiator" [High,Triaged] | 00:40 |
smoser | so you marked that "fix released" | 00:40 |
smoser | i'm just curios, so i can at least attempt to be consistent. | 00:41 |
bigjools | smoser: yes, they are all fix released if they are in trunk, I am not using fix-committed | 00:41 |
bigjools | then they disappear off bug listings | 00:41 |
roaksoax | bigjools: check the changes made in maas-region-controller.postinst, I removed and if statement | 00:42 |
roaksoax | an* | 00:42 |
bigjools | roaksoax: aha! | 00:42 |
bigjools | nice one | 00:43 |
roaksoax | bigjools: there's a related issue though that I can't find a solution for | 00:43 |
roaksoax | bigjools: if you remove (not purge) and install again, it will show the same error | 00:43 |
roaksoax | bigjools: and I already know how to fix the maas-celery thing, i'll work on it in a bit though | 00:44 |
bigjools | roaksoax: ok - jtv landed a change so we can pass uids I think | 00:44 |
bigjools | bug 1060114 | 00:45 |
ubot5 | Launchpad bug 1060114 in MAAS "DELETE operations are not idempotent" [Medium,Invalid] https://launchpad.net/bugs/1060114 | 00:45 |
bigjools | heh | 00:45 |
smoser | bigjools, https://bugs.launchpad.net/maas/+bug/1058137 | 00:45 |
ubot5 | Ubuntu bug 1058137 in MAAS "no way to get first user's first set of api creds" [High,Triaged] | 00:45 |
smoser | if i just put that in comment 4 into a python script and run it | 00:45 |
smoser | i stack trace | 00:45 |
bigjools | matsubara: that python script works for you? --^ | 00:46 |
smoser | http://paste.ubuntu.com/1257137/ | 00:46 |
bigjools | smoser: export DJANGO_SETTINGS_MODULE=path/to/settings.py | 00:47 |
bigjools | /etc/maas/whatever | 00:47 |
matsubara | bigjools, yes, I think I did some path hackery to get it to work | 00:48 |
bigjools | grumble grumble, roaksoax removed my changelog attribution :) | 00:53 |
roaksoax | bigjools: no i didn't :) | 00:54 |
bigjools | -- Andres Rodriguez <andreserl@ubuntu.com> Tue, 02 Oct 2012 13:39:47 -0400 | 00:54 |
bigjools | you did :) | 00:54 |
roaksoax | bigjools: yeah I made changes to the changelog | 00:54 |
roaksoax | dch -i changes that :) | 00:54 |
* bigjools shakes fist at dch | 00:55 | |
bigjools | -u surely? | 00:55 |
bigjools | or -e? | 00:55 |
roaksoax | bigjools: btw... an assumption I am working with if the fact that it doens't matter how many releases we release in PPA | 00:55 |
roaksoax | bigjools: the packaging should remain UNRELEASED | 00:55 |
bigjools | ok | 00:55 |
roaksoax | and only change when we release to archives | 00:55 |
roaksoax | bigjools: but since the changelog is being pilling up I think is ok | 00:56 |
bigjools | roaksoax: well dch -i doesn't change the release it adds a new one, so something changed the attribution I added | 00:56 |
roaksoax | bigjools: because entries of releases that haven't been uploaded to the archives will appear in the changelog even though we didn't | 00:56 |
bigjools | yup | 00:57 |
bigjools | soyuz is great :) | 00:57 |
roaksoax | bigjools: dch -e -DUNRELEASED works? | 00:57 |
roaksoax | errerr dch -e -Dquantal | 00:57 |
bigjools | roaksoax: in r113 the change line was updated, what dch option does that? | 00:59 |
smoser | i tihnk in thie change log there we need to ditch the double bzr branch | 00:59 |
smoser | we get one "for free" from the daily ppa | 00:59 |
smoser | biuld | 00:59 |
smoser | so we should either be pedantic about getting that updated in the code on each commit (or tooling) or just remove the bzr revno from there. | 01:00 |
roaksoax | bigjools: dch -i does it for me | 01:00 |
bigjools | true, but it's needed to build locally from recipe isn't it? | 01:00 |
bigjools | roaksoax: -i adds a new one for me | 01:00 |
smoser | bigjools, well we can just add a script that does the build locally and inserts it. | 01:01 |
bigjools | smoser: just need to fix the debian/rules target I expect | 01:01 |
bigjools | roaksoax: bzr diff -c 113 debian/changelog | 01:01 |
roaksoax | bigjools: dch -e --nomainttrailer | 01:05 |
bigjools | roaksoax: aha | 01:06 |
bigjools | roaksoax: useful, it saves me having to use -k when recipe building :) | 01:06 |
roaksoax | indeed :) | 01:06 |
roaksoax | bigjools: btw.. i already told allenap... no scaping from Peruvian Pisco is allowed at UDS | 01:07 |
roaksoax | smoser: ^^ | 01:07 |
bigjools | ?! | 01:07 |
roaksoax | escaping* | 01:07 |
roaksoax | :) | 01:07 |
bigjools | does that involve naked bodies and mud? | 01:07 |
bigjools | otherwise I am totally not up for it | 01:07 |
roaksoax | lol | 01:08 |
bigjools | roaksoax: getting this on the experimental package: http://paste.ubuntu.com/1257167 | 01:09 |
roaksoax | bigjools: as an error? | 01:13 |
smoser | ok. this is silly. | 01:13 |
roaksoax | bigjools: can you pastebin a full log? | 01:13 |
smoser | http://paste.ubuntu.com/1257171/ | 01:13 |
bigjools | matsubara: can you paste a full hackery for smoser please? | 01:14 |
roaksoax | smoser: do you think i should allow sending several config files rather than just one? | 01:14 |
bigjools | roaksoax: http://paste.ubuntu.com/1257172 | 01:15 |
smoser | roaksoax, do you have a reason for doing so | 01:15 |
smoser | ? | 01:15 |
roaksoax | smoser: so if someone sets custom ipmi configs they can be sourced automatically? | 01:15 |
roaksoax | bigjools: yeah that's dbconfig being an asshole i think | 01:15 |
smoser | it doesn't sound unreasonable. | 01:15 |
bigjools | roaksoax: yay | 01:15 |
bigjools | roaksoax: I'll re-purge | 01:16 |
roaksoax | bigjools: yes, just to be sure, remove dbconfig-common too and make sure /etc/dbconfig-common is empty | 01:16 |
roaksoax | bigjools: i discovered that maas.conf under dbconfig-common is preserverd, and we also need to remove that | 01:16 |
roaksoax | bigjools: i'll take care of that along the celery stuff | 01:16 |
roaksoax | along with* | 01:16 |
matsubara | bigjools, smoser: something like this should work: https://pastebin.canonical.com/75761/ | 01:16 |
smoser | matsubara, well, thats basically what i did | 01:18 |
bigjools | roaksoax: /etc/dbconfig-common/maas-region-controller.conf exists | 01:18 |
bigjools | weird I'll remove it | 01:18 |
bigjools | roaksoax: on re-install (after removing it) I now get a debconf page telling me it was currently iunstalled and locally mnodified | 01:19 |
bigjools | installed* | 01:19 |
* bigjools installs package version | 01:20 | |
matsubara | ok, guys, gotta go. talk to you tomorrow. | 01:21 |
matsubara | have a good night/day | 01:21 |
=== matsubara is now known as matsubara-afk | ||
roaksoax | bigjools: say yes | 01:22 |
bigjools | :) | 01:22 |
bigjools | smoser, roaksoax: how long are you in CPH? | 01:23 |
roaksoax | bigjools: 2 weeks | 01:23 |
bigjools | just UDS? | 01:23 |
bigjools | ah | 01:23 |
smoser | same as roaksoax | 01:23 |
smoser | too long | 01:23 |
roaksoax | smoser: yay!! | 01:23 |
roaksoax | lol | 01:23 |
bigjools | ok plenty of time to meet up for an evening food/drink | 01:24 |
bigjools | yeah too long | 01:24 |
bigjools | my wife is going to self destruct looking after our twins on her own | 01:24 |
jtv | Hi folks | 01:25 |
bigjools | hi jtv | 01:25 |
smoser | i'm really sorry for being a dolt. but some help would be greatly appreciated. | 01:26 |
smoser | http://paste.ubuntu.com/1257179/ | 01:26 |
jtv | User isn't a MAAS thing. It comes with Django. | 01:29 |
jtv | django.contrib.auth.models.User | 01:30 |
jtv | How it got to be in contrib I may learn someday. How it got to stay there I just won't believe no matter what you tell me. | 01:30 |
bigjools | jtv: should still work | 01:30 |
bigjools | however I get this: | 01:31 |
bigjools | from django.db import utils | 01:31 |
bigjools | ImportError: cannot import name utils | 01:31 |
bigjools | at the bottom of the trace | 01:31 |
smoser | https://bugs.launchpad.net/maas/+bug/1058137/comments/5 | 01:31 |
ubot5 | Ubuntu bug 1058137 in MAAS "no way to get first user's first set of api creds" [High,Triaged] | 01:31 |
roaksoax | smoser: http://paste.ubuntu.com/1257181/ with http://paste.ubuntu.com/1257183/ --> what do you think? | 01:31 |
smoser | got it | 01:31 |
bigjools | smoser: phew :) | 01:32 |
smoser | thank you bigjools jtv | 01:32 |
bigjools | np | 01:32 |
roaksoax | bigjools: btw... I think we need to be able to pass kernel arguments | 01:33 |
roaksoax | bigjools: and be able to edit that | 01:33 |
bigjools | pxe template? | 01:33 |
roaksoax | bigjools: console=ttyS0 | 01:34 |
roaksoax | bigjools we need to be able to edit things like that | 01:34 |
roaksoax | bigjools: so yes, it should be added to the pxe template | 01:34 |
roaksoax | bigjools: but should probably be per node | 01:34 |
roaksoax | if one wishes to change that from a node, or set of nodes | 01:34 |
bigjools | roaksoax: how important is this? | 01:35 |
bigjools | should be easy enough to do, but I need to prioritise against the thousand other bugs | 01:36 |
roaksoax | bigjools: ver important i'd say, even sabdlf mention we should be able to edit it. For example, console configuration needs to be done manually directly in the BIOS for ipmi cards | 01:36 |
bigjools | eugh | 01:36 |
roaksoax | bigjools: so they configure serial to be ttyS20 | 01:36 |
roaksoax | so maas, as it stands right now, sends console=ttyS1 | 01:37 |
bigjools | ok, Node needs a custom "kernel_params" field then, and we can add that to the pxe template | 01:37 |
roaksoax | bigjools: indeed, so we don't really need it to be displayed in the WebUI | 01:38 |
smoser | roaksoax, bigjools well, per-node kernel settings are not all that important really. | 01:38 |
smoser | i opened that bug | 01:38 |
smoser | and i do think they need to be editable | 01:38 |
roaksoax | smoser: right, so we can set global settings, but if the user wants to change them, per node, he needs to | 01:38 |
smoser | but realistically, in a large scale, you're going to have a couple groups of systems | 01:38 |
bigjools | my question is, important for *this release* | 01:38 |
smoser | so per-node would suck anyway. | 01:38 |
smoser | i cant think of them being a hard requirement for this release. | 01:38 |
roaksoax | smoser: right but for example, in sabdlf's cluster I had to manually edit what console to output in maas code, because it was configured differently on the BIOS | 01:39 |
smoser | but the ability to append cmdline arguments without editing python source would be nice. | 01:39 |
bigjools | smoser: I am thinking that the cluster controller would want to set the parameters when it uploads node definitions (in the future) | 01:39 |
smoser | bigjools, ok. heres the whole sucky problem. | 01:39 |
bigjools | all this stuff will/should be automated | 01:39 |
smoser | a.) before a node is enlisted, you know nothing about it. so you can't really customize stuff per-node at that point | 01:39 |
smoser | b.) after that point, you could have some custom settings | 01:40 |
bigjools | roaksoax: https://code.launchpad.net/~julian-edwards/maas/broken-dns-install-bug-1060549/+merge/127621 | 01:40 |
smoser | but *really* | 01:40 |
smoser | c.) in a perfect world, give some knowledge of the setup and the node (hw and the like) you could perfectly generate all that anyway. | 01:40 |
bigjools | I disagree with (a) since the cluster controller may know all about its hardware already | 01:40 |
smoser | in which case you're to 'c' | 01:40 |
smoser | so you dont need them | 01:40 |
smoser | :) | 01:40 |
smoser | roaksoax, i think you'd have been fine with the ailbiyt to append cmdline settings. | 01:42 |
smoser | right? | 01:42 |
smoser | and i would have in all my testing been ok to do the same. | 01:42 |
smoser | that would releieve a lot of pain | 01:42 |
roaksoax | bigjools: looks good, make sure + add_user_group | 01:42 |
roaksoax | 49+ | 01:42 |
roaksoax | is ident'd correctly | 01:42 |
roaksoax | smoser: huh? | 01:42 |
bigjools | roaksoax: GRARGH. The rest of the file has got TABS in it | 01:43 |
bigjools | I used 4 spaces | 01:43 |
bigjools | TABS ARE EVIL | 01:43 |
roaksoax | smoser: btw... i'm just gonna pass maas-signal with an argument with the json stuff, such as maas-signal --config --power-setings <json format> | 01:43 |
* roaksoax uses tabas for shell scripts and spaces for python code | 01:43 | |
bigjools | TABS ARE EVIL | 01:44 |
bigjools | :) | 01:44 |
roaksoax | xD | 01:44 |
bigjools | because you end up with what you see on the MP | 01:44 |
bigjools | do you care if I s/\t/ /g ? | 01:45 |
roaksoax | bigjools: different schooling I guess.. I was always a TAB user | 01:45 |
roaksoax | bigjools: go for it | 01:45 |
bigjools | we're very particular about our code formatting in the Launchpa^WCloud Engineering team | 01:45 |
roaksoax | smoser: yeah I think we need to be able to append that | 01:45 |
roaksoax | smoser: adding the per-node isn't really that big of a deal, we don't even need to display it in the UI, just be able to modify it per node from maascli | 01:46 |
bigjools | roaksoax: can you approve the MP please | 01:47 |
roaksoax | bigjools: done! | 01:48 |
roaksoax | smoser: ok I think it is done i just need to test it | 01:48 |
* roaksoax heads home | 01:49 | |
roaksoax | smoser: i'll go home and test it on the HP mini servers | 01:49 |
bigjools | cheers | 01:49 |
bigjools | jtv: it looks like you didn't fix the upstart conf to expect a fork, or am I missing something? | 01:56 |
jtv | bigjools: I didn't get around to looking into that; I assumed it already expected a fork but got the wrong one. | 01:57 |
bigjools | jtv: ok I 'll sort it out | 01:58 |
jtv | Based on what you said, I thought the packaging branch already expected a fork. | 01:58 |
bigjools | jtv: I tried it but it hung, I'll try it again | 01:58 |
jtv | :( | 01:59 |
bigjools | now there *is* a fork :) | 01:59 |
bigjools | it should work | 01:59 |
jtv | There was a fork before as well. | 01:59 |
jtv | That hasn't changed. | 01:59 |
jtv | But I removed the spurious fork that preceded it. | 02:00 |
bigjools | hmm | 02:00 |
bigjools | jtv: something else is forking | 02:02 |
bigjools | jtv: http://paste.ubuntu.com/1257202 | 02:02 |
jtv | Well it can fork right off. | 02:03 |
* bigjools straces it | 02:03 | |
bigjools | ARGH | 02:04 |
bigjools | it's the sodding wrapper script | 02:04 |
bigjools | jtv: when starting a python script, straces shows that it forks off this lot: "id, sh, ldconfig, sh, ldconfig, sh, uname" | 02:11 |
bigjools | wtf | 02:12 |
jtv | wtf indeed | 02:12 |
jtv | So celeryd is a wrapper script? | 02:13 |
bigjools | no | 02:13 |
bigjools | maas-provision | 02:13 |
bigjools | the one installed by the package that checks for uid==0 | 02:13 |
jtv | Grrr | 02:14 |
bigjools | so basically python is forking a load of stuff off before it starts your actual code | 02:14 |
bigjools | unless it's hiding in maas-provision? | 02:14 |
jtv | I bet it's import side effects. :( | 02:15 |
jtv | I tried this and found no forks: $ strace python -c 'print' 2>&1 | grep fork | 02:15 |
bigjools | I am doing this: | 02:16 |
bigjools | sudo PYTHONPATH="/usr/share/maas${PYTHONPATH:+:}${PYTHONPATH}" CELERY_CONFIG_MODULE="celeryconfig_cluster" strace -o /tmp/strace.log -fFv /usr/bin/python -m provisioningserver start-cluster-controller http://maas.internal.example.com/ | 02:16 |
bigjools | (ignoring the wrapper for now) | 02:16 |
bigjools | oh god | 02:17 |
bigjools | even if we get this right it won't work jtv | 02:17 |
jtv | Yes, my son? | 02:17 |
bigjools | because the fork can happen ages later | 02:17 |
bigjools | so the upstart script will always hang until the admin accepts the cluster | 02:17 |
jtv | Yes | 02:18 |
* bigjools gives up | 02:18 | |
jtv | Then maybe we should kill two birds with one stone. | 02:18 |
jtv | Move the polling into the child process, and instead of exec'ing, just have it hang around as basically a wrapper to celeryd. | 02:18 |
bigjools | not sure it'll work ATM anyway | 02:19 |
bigjools | because of these 6 spurious forks | 02:19 |
jtv | That way, start-cluster-controller will hang around for as long as celeryd is running. | 02:19 |
bigjools | I'd rather not do that | 02:19 |
bigjools | but let;s see what roaksoax thinks | 02:20 |
* bigjools lols at jtv's continued wrong merge targets :D | 02:21 | |
jtv | Yes, it's become part of the process. Why change it now? | 02:21 |
jtv | bigjools, roaksoax: I think upstart can manage a process that just never exits. If so, I think it would simplify our problem a lot if we could just keep the maas-provision process alive, instead of forking & exiting. | 02:30 |
smoser | http://paste.ubuntu.com/1257227/ | 02:30 |
smoser | thoughts? | 02:30 |
bigjools | is node-group-interfaces valid? | 02:31 |
smoser | different than http://paste.ubuntu.com/1257229/ | 02:32 |
bigjools | ok | 02:32 |
* bigjools looks at code | 02:33 | |
bigjools | smoser: nodegroup called "master" does not exist | 02:33 |
jtv | Wait... did I seriously just catch a snippet of advertising for the miraculous innovation of... coffee without sugar!? | 02:33 |
bigjools | well, uuid of master I mean | 02:34 |
smoser | so what is 'master' there? | 02:34 |
bigjools | must be the uuid of an existing nodegroup | 02:35 |
bigjools | maas-cli api localmaas node-groups list | 02:35 |
smoser | it would seem like no | 02:36 |
smoser | http://paste.ubuntu.com/1257235/ | 02:36 |
smoser | got 'master' as you said | 02:37 |
smoser | hm.. | 02:37 |
bigjools | if you list existing NGIs does it show? | 02:39 |
bigjools | hey, this is not a maas bug, where should be be retargeted? https://bugs.launchpad.net/maas/+bug/1060194 | 02:39 |
ubot5 | Ubuntu bug 1060194 in MAAS "cannot find "bnx2-mips-09-6.2.1a.fw" when booting from maas" [Undecided,New] | 02:39 |
smoser | woohoo. | 02:40 |
smoser | bigjools, initramfs-tools maybe | 02:42 |
bigjools | ok | 02:42 |
smoser | i actually think i saw this. | 02:42 |
bigjools | thanks | 02:42 |
smoser | err... jsut reading code thought "how strange, it brings up networking without looking waiting for udev" | 02:42 |
bigjools | smoser: or cloud-initramfs-tools? | 02:42 |
smoser | no | 02:42 |
smoser | initramfs-tools | 02:42 |
bigjools | ok done | 02:42 |
smoser | incidently, it is probably fixed now | 02:42 |
smoser | because cloud-initramfs-tools code goes looking for NICs and waits for udev | 02:43 |
smoser | :) | 02:43 |
roaksoax | smoser: i'm gonna have to test the cloud-init stuff tomorrow... the mini servers are gonna take forever to install | 02:43 |
roaksoax | smoser: err upgrade | 02:44 |
smoser | upgrade? | 02:44 |
roaksoax | smoser: my maas mini server needs to be upgraded :) | 02:44 |
smoser | bigjools, ok. so my goal in the last escapade is to configure dns | 02:46 |
smoser | http://paste.ubuntu.com/1257248/ | 02:46 |
smoser | that gets me a valid response | 02:47 |
smoser | but my goal is to get dhcpd functioning | 02:47 |
bigjools | cool | 02:47 |
smoser | and i still have no /etc/maas/dhcpd.conf | 02:47 |
jtv | Hi roaksoax! Did you see our problem with the fork-tracking for the cluster controller? | 02:47 |
bigjools | smoser: check the celery log | 02:47 |
roaksoax | jtv: howdy. no I didn't | 02:48 |
roaksoax | jtv: any lp | 02:48 |
roaksoax | jtv: any lp bug? | 02:48 |
bigjools | smoser: 1. has it received secrets from the region, 2. did it get any DNS tasks? | 02:48 |
jtv | roaksoax: bug 1059453. Turns out maas-provision does a lot of forks before it gets to the one we'd want upstart to track. | 02:48 |
ubot5 | Launchpad bug 1059453 in maas (Ubuntu) "The celery cluster worker is not properly stopped" [Critical,Triaged] https://launchpad.net/bugs/1059453 | 02:49 |
bigjools | smoser: is https://bugs.launchpad.net/bugs/1060331 fixed now? | 02:49 |
ubot5 | Ubuntu bug 1060331 in MAAS "daily ppa install fails fully automated" [Undecided,New] | 02:49 |
smoser | bigjools, sure. | 02:50 |
smoser | i tihnk it was the isc_service_stop thing | 02:50 |
bigjools | yeah | 02:50 |
roaksoax | jtv: i think that's a question for james hunt | 02:50 |
roaksoax | jtv: if not, we are probably have to run it as we used to | 02:50 |
bigjools | roaksoax: oh FWIW there's a trick to conf file removal | 02:50 |
bigjools | I asked colin about it | 02:50 |
roaksoax | bigjools: what did he say? | 02:51 |
bigjools | roaksoax: "man dh_installdeb, man dpkg-maintscript-helper" :) | 02:51 |
bigjools | and search for "conffile removal" | 02:51 |
roaksoax | bigjools: ah yeah, i know that :) | 02:51 |
smoser | can i turn of header output from the cli ? | 02:52 |
bigjools | so basically upstart confs are not treated specially | 02:52 |
bigjools | smoser: not yet :( | 02:52 |
roaksoax | bigjools: nope, so we can use rm_conffile | 02:52 |
jtv | Thanks roaksoax -- I'll see if I can reach JH later. | 02:52 |
bigjools | roaksoax: exactly | 02:52 |
roaksoax | bigjools: we need to do the same for maas.conf under /etc/dbconfig-common | 02:52 |
roaksoax | bigjools: i'll take care of it :) | 02:52 |
bigjools | roaksoax: ok thanks! | 02:52 |
smoser | bigjools, http://paste.ubuntu.com/1257255/ | 02:55 |
smoser | that would seem the relevant bit of celery log | 02:55 |
bigjools | yay | 02:57 |
bigjools | smoser: anything for dhcp? | 02:58 |
smoser | http://paste.ubuntu.com/1257259/ | 02:58 |
smoser | full log | 02:58 |
bigjools | is maas-dns installed? | 03:00 |
smoser | no | 03:02 |
bigjools | that might help :)_ | 03:02 |
bigjools | also maas-dhcp? | 03:02 |
bigjools | well the former will ensure the latter | 03:02 |
smoser | maas-dhcp, yes. | 03:03 |
smoser | you're saying dns ensures dhcp? | 03:04 |
bigjools | yes | 03:04 |
smoser | k. | 03:04 |
bigjools | we can't do dns without dhcp | 03:04 |
bigjools | I suspect a signal is missing to push out the dhcp config when NGI changes | 03:04 |
bigjools | hmm no that's wired | 03:06 |
smoser | bigjools, so am i at least going down the correct path here? | 03:11 |
bigjools | smoser: yes | 03:11 |
smoser | is there some expected way that someone would set this up? | 03:11 |
bigjools | this is fine so far, if it's not working it's a bug | 03:11 |
roaksoax | smoser: ok so the script itself seems to work | 03:14 |
roaksoax | smoser: just need to test in a deployment | 03:15 |
smoser | roaksoax, nice work. | 03:16 |
roaksoax | i'll try to get that tonight | 03:19 |
roaksoax | i just realized that i need to reconfigure the network :( | 03:19 |
roaksoax | my home network :( | 03:20 |
smoser | Setting up maas-dns (0.1+bzr1134+dfsg-0+1136+113~ppa0~quantal1) ... | 03:20 |
smoser | chown: invalid user: `maas:root' | 03:20 |
smoser | dpkg: error processing maas-dns (--configure): | 03:20 |
smoser | subprocess installed post-installation script returned error exit status 1 | 03:20 |
roaksoax | smoser: that's already been fixed by bigjools | 03:20 |
bigjools | smoser: there's a fix just landed | 03:20 |
bigjools | needs rebuilding | 03:20 |
* bigjools hits the daily rebuild button | 03:20 | |
bigjools | should be ready (published) in 10-15 mins | 03:21 |
bigjools | I <3 recipes | 03:23 |
* bigjools grabs food | 03:29 | |
smoser | bigjools, ok | 03:33 |
smoser | http://paste.ubuntu.com/1257282/ | 03:33 |
smoser | ^ results in http://paste.ubuntu.com/1257283/ | 03:34 |
smoser | and no /etc/maas/*dhcp* | 03:35 |
smoser | well, i have to go to bed. | 03:44 |
smoser | tell me what stupid thing i've done there. | 03:48 |
smoser | http://paste.ubuntu.com/1257288/ is some node group info. | 03:48 |
smoser | it seems my operation to update my node worked. | 03:48 |
smoser | ok. bed. | 03:50 |
smoser | good night. | 03:51 |
bigjools | smoser: hmmm looks like a bug. I'll get rvb to look, unless jtv wants to | 03:51 |
bigjools | nn smoser | 03:51 |
jtv | nn smoser | 03:51 |
jtv | I'm looking up the pastes | 03:51 |
smoser | oh. one thing to note. there are things in /etc/bind/maas | 03:52 |
smoser | $ ls /etc/bind/maas/ | 03:52 |
smoser | named.conf.maas rndc.conf.maas zone.master | 03:52 |
smoser | named.conf.rndc.maas zone.77.168.192.in-addr.arpa | 03:52 |
bigjools | that looks expected | 03:52 |
smoser | right | 03:52 |
smoser | but no dhcp | 03:52 |
bigjools | yeah dhcp is differnet tasks and they';re not getting fired | 03:52 |
* jtv inserts lame joke about doing the same thing for us | 03:53 | |
bigjools | one of us will try to reproduce | 03:53 |
smoser | ok. both of you can get to ubuntu@10.55.60.130 | 03:54 |
smoser | to poke around | 03:54 |
smoser | do whatever you want. | 03:54 |
jtv | Thanks. That'll tell us if it's a missing sudo or an apparmor thing. | 03:54 |
smoser | (hoping through chinstrap | 03:54 |
smoser | later. | 03:55 |
jtv | nn | 03:56 |
jtv | permission denied :( | 03:56 |
jtv | bigjools, can you get into that system? | 03:57 |
bigjools | it's probably on the VPN | 03:57 |
smoser | ubuntu@ | 03:57 |
jtv | Yes I did | 03:57 |
jtv | This is a canonistack instance, not the VPN AFAICS | 03:58 |
smoser | right | 03:58 |
smoser | http://paste.ubuntu.com/1257300/ | 03:58 |
bigjools | oh that needs spethial setup too IIRC | 03:58 |
smoser | only hoping ghrough chinstrap | 03:58 |
smoser | is all it needs | 03:58 |
jtv | Well I did do it through chinstrap of course | 03:58 |
smoser | Host 10.55.58.* 10.55.60.* 10.55.62.* 10.55.63.* *.canonistack *.canonistack2 | 03:59 |
smoser | ProxyCommand ssh chinstrap.canonical.com nc -q0 %h %p | 03:59 |
smoser | thats all the config you should need. | 03:59 |
jtv | Well, my key is in there. | 04:00 |
jtv | And ssh into canonistack instances normally works for me. | 04:00 |
jtv | And I'm doing the ubuntu@ | 04:00 |
jtv | And I'm going through chinstrap. | 04:00 |
jtv | But it denies me public-key auth. | 04:00 |
roaksoax | smoser: so there's various maas regions now? | 04:00 |
roaksoax | err | 04:01 |
roaksoax | canonistafck regions* | 04:01 |
smoser | 2 | 04:01 |
roaksoax | ah cool | 04:01 |
smoser | well, password auth is now enabled there | 04:03 |
jtv | Ahhhh this needs my canonistack key, not my regular ssh key | 04:03 |
jtv | So I need to register my canonistack keys as ssh keys. | 04:03 |
smoser | and password is "maas-is-fun" | 04:03 |
smoser | jtv, you dont have to do that. | 04:03 |
jtv | You sick, sick bastard | 04:03 |
jtv | Well it doesn't seem to work without! My ssh config for canonistack instances is set up to use my canonistack key as IdentityFile. | 04:04 |
smoser | i imported your credentials from launchpad. those should forward through the ssh ProxyCommand above. | 04:04 |
smoser | yeah, i'd recommend ditching that crappy canonistack key | 04:04 |
jtv | But every time you do this, it denies me login. | 04:04 |
smoser | and only ever using your real keys | 04:04 |
jtv | Oh, that's not needed!? | 04:05 |
smoser | euca-import-keypair --public-key-file ~/.ssh/id_rsa.pub jtf | 04:05 |
smoser | then launch instances with '--keypair jtf' | 04:05 |
smoser | (spelled jtv wrong twice) | 04:05 |
jtv | What are the odds | 04:05 |
jtv | Anyway, still not working without that IdentityFile line in my ssh config. :( | 04:06 |
roaksoax | smoser: http://paste.ubuntu.com/1257310/ alright, we just need to figure out a way how to run it | 04:06 |
roaksoax | smoser: unless you want to run it as a script? | 04:06 |
smoser | well, since it isn't really a script, it doesn't make sense. | 04:08 |
smoser | i'd just add something that runs that script and calls maas signal | 04:09 |
smoser | roaksoax, i can look more tomorrow. | 04:09 |
roaksoax | smoser: yeah, either way I won't test it until tomorrow | 04:09 |
* roaksoax is off | 04:12 | |
roaksoax | night all | 04:12 |
jtv | nn roaksoax | 04:27 |
bigjools | smoser: if you are still there, I know what's wrong | 04:36 |
bigjools | we never believe you when you say you're going to bed :) | 04:37 |
jtv | He must be asleep for real. He wouldn't have been able to resist this one. | 05:08 |
bigjools | smoser: adding a new nodegroup requires a corresponding celeryd to be running, listening on the right queue. maas-provision start_cluster_controller does that in packaging but you can just run "celeryd -Q <uuid>" | 05:13 |
Fajkowsky | hello I have problem with http://askubuntu.com/questions/195115/nodes-cant-connect-to-server-after-bootstrap | 08:51 |
Fajkowsky | can someone help me with it? | 08:51 |
mgz | what is the lint thing jtv will cry if I don't run? | 09:12 |
bigjools | make lint! | 09:13 |
mgz | ta. | 09:13 |
mgz | sounds counter productive though... | 09:13 |
bigjools | Fajkowsky: answered your Q in #ubuntu-server but better to talk about it here | 09:13 |
mgz | unmake lint! | 09:13 |
bigjools | mgz: hoho! | 09:13 |
Fajkowsky | I have question about maas, I want to check if performence will be grow if I add more nodes to one service. I want to check on minecraft server because I think it's make big diffrence in performance if I add another node, I am right? | 09:17 |
bigjools | Fajkowsky: this is not a maas-specific question, but yes you can scale out easily with juju and maas | 09:18 |
Fajkowsky | whoops , you are right | 09:19 |
jam | bigjools: so we will have the tags getting processed in the async workers eventually, how do we do integration testing of that? | 09:22 |
jam | ATM, we only seem to have unit tests | 09:22 |
jam | that mock out the actual POST/GET requests. | 09:22 |
bigjools | jam: the test harness will run the tasks synchronously | 09:24 |
bigjools | so leave everything unmocked and test for the desired end result | 09:25 |
bigjools | provided you're running with the right TestCase, anyway. There are diffrerent ones and they are all called TestCase, which is something that needs to be fixed before I strangle a cat in frustration. | 09:25 |
bigjools | and you need the celery test resource loaded, theres plenty of examples in existing tests | 09:26 |
rvba | jam: just be aware of the fact that, in the tests, task.delay() calls the task in a synchronous fashion. | 09:27 |
jam | bigjools: so do all nodegroups have celery workers in the test suite, then? | 09:27 |
jam | (right now the apis are restricted so that only the associated worker is allowed to call them) | 09:28 |
jam | since otherwise you could get around the privacy stuff | 09:28 |
bigjools | jam: not sure how that'd work out, you might need to patch stuff in that case. You just need to know that apply_async() or delay() just calls the Task func there and then. | 09:29 |
bigjools | obviously there's no queue in this case | 09:29 |
jam | bigjools: well part of this is that I want to have at least one test that things work in a real integration setting (multiple nodegroups, and real celery workers talking via the apis.) | 09:29 |
bigjools | ok | 09:30 |
bigjools | mmmm that might be hard | 09:30 |
jam | I can also update some apis so they can be called by superusers | 09:30 |
bigjools | we've always mocked things out in existing tests | 09:30 |
jam | bigjools: well, it is either that, or all the testing falls onto matsubara | 09:30 |
rvba | The only thing you can do in tests is make sure that each task is routed to the right queue. | 09:30 |
bigjools | you can test either side of the Task easily | 09:30 |
bigjools | and what rvba just said | 09:31 |
jam | rvba: the problem with mocking it out on the celery side, is that you can easily get skew between what you think the API takes and returns, and what reality says. | 09:31 |
bigjools | but a complete integration test is very tricky as we don't start up celeryd | 09:31 |
bigjools | jam: the API should be static anyway | 09:32 |
bigjools | it's versioned | 09:32 |
jam | bigjools: well, it is just being implemented now... :) | 09:32 |
bigjools | well, it's versioned in that we reserved a space for a version :) | 09:32 |
jam | so the testing needs to be done manually for now? | 09:32 |
bigjools | jam: if you patch out the task you can check the params and fail if they change at all | 09:32 |
rvba | jam: you should be able to use the API for real (without mocking it) in a celery task. | 09:32 |
jam | rvba: except for the permissions bits? or how do we work around those? | 09:33 |
jam | (I do know that I had one accidental success if you 'yield' in an api call. | 09:33 |
jam | api stubbing returns the generator, and 'assertItemsEqual' passes just fine) | 09:33 |
bigjools | jam: you should be able to test end-to-end provided you do something about the queue checking, perhaps mock it, I'd need to see it | 09:33 |
jam | but in reality, the HTML level stuff turns the generator into a string | 09:33 |
jam | "<generator ...>" | 09:34 |
bigjools | but the queue given gets passed to the task so it's easily checked | 09:34 |
bigjools | rvb can help you, I gotta EOD now | 09:34 |
jam | bigjools: np, have a good evening | 09:34 |
jam | rvba: I actually need to go pick up my son now, but I'll be back to pick your brain some more later. | 09:35 |
bigjools | cheers | 09:35 |
rvba | jam: by permissions you mean the API credentials? | 09:35 |
rvba | jam: sure, no pb. | 09:35 |
jam | rvba: right, so I added apis that let the nodegroup worker get access to all information about nodes in the nodegroup | 09:35 |
jam | but we can't make that public because that gets around the Node VIEW permission stuff | 09:35 |
jam | so it is only allowed by the oauth key associated with the nodegroup worker. | 09:36 |
rvba | Makes sense. | 09:36 |
jam | which isn't the 'client' that is running the 'create a tag' | 09:36 |
jam | anyway, really gone | 09:36 |
jtv | Any reviewers in the house? https://code.launchpad.net/~jtv/maas/bug-1059453/+merge/127680 | 09:41 |
mgz | what is this netifaces python package the provisionserver wants, why do I not have it, and where can I get it... | 09:41 |
rvba | jam: I think the only thing you need to do is patch the credentials used by the worker to connect to the API (i.e. simulate what refresh_secrets would do) | 09:42 |
mgz | ah, I bet it's because it's been added to required-packages since I last pulled locally... | 09:42 |
jtv | mgz: "make install-dependencies" | 09:42 |
mgz | I should fix my deployment script. | 09:42 |
jtv | And pull regularly. | 09:42 |
jtv | You don't want your branch to fall out of date or you'll be building merge conflicts into branches right from the start. | 09:43 |
mgz | I should change it to use bzr cat lp:maas rather than peeking at my stale branch on this box | 09:43 |
jtv | We've got a pretty high ratio of changes to code right now. | 09:43 |
mgz | the copy I'm working with is up to date, but what I'm telling cloud-init to install is based off a copy of maas I'm not using | 09:44 |
jtv | Ah | 09:44 |
mgz | hence forget to keep up to date. | 09:45 |
mgz | still my fault though :) | 09:45 |
jtv | allenap: want to talk cli? | 09:47 |
allenap | jtv: Sure. | 09:47 |
allenap | jtv: Shall I call you? | 09:47 |
jtv | Call? | 09:48 |
allenap | jtv: It's probably quicker, but here is okay I guess. | 09:49 |
jtv | I mean, do we do a hangout? | 09:49 |
jtv | I'm starting one | 09:50 |
jtv | allenap: https://plus.google.com/hangouts/_/aaedfd9ecc51ae502006d3c55aa21e6680992c86?authuser=0&hl=en-GB | 09:50 |
mgz | jam: the maas lander really hates you... | 09:52 |
mgz | it's not done that "additional revisions which have not been approved" thing to me once... | 09:52 |
mgz | let's see if I've jinxed myself... | 09:53 |
mgz | hm, is there some way I get get django up on localhost without all the rest of the junk for make run? celery is unhappy now... | 10:02 |
mgz | because there's no 'maas' user... what should have created that? | 10:05 |
rvba | jtv: what mgz is experiencing looks like a fallout from what you're doing to fix the start-cluster script… | 10:11 |
jtv | otp | 10:12 |
rvba | jtv: I'm just guessing here but if we try to use the maas user to run celeryd on a dev instance, that will be a problem. | 10:12 |
jtv | mgz: try bin/maas-provision start-cluster-controller http://localhost:5240/ -u `pwd` -g `pwd` | 10:13 |
jtv | rvba: that's right, so don't do that. Use your real identity. | 10:13 |
jtv | mgz: sorry, not pwd | 10:13 |
jtv | the other one | 10:13 |
jtv | whoami | 10:13 |
jtv | and yes I'm conflating your user with your group of the same name and just guessing that there is one | 10:14 |
mgz | ubuntu:ubuntu so yup. | 10:14 |
rvba | services/cluster-worker/run will probably need to be fixed then. | 10:14 |
jtv | Does services/cluster-worker/run use start-cluster-controller already? | 10:14 |
rvba | Yep, since last week. | 10:14 |
jtv | allenap had a cute idea: default not to maas but to the _current_ user, except when that's root. | 10:14 |
mgz | for the record: <http://pastebin.ubuntu.com/1257666/> | 10:15 |
jtv | Unless & until we do that, pass explicit user/group. | 10:15 |
jtv | Yes, that's what happens if you leave it to default to "maas" on a system that doesn't have a maas user. | 10:16 |
jtv | rvba: I'll update the "run" script. Thanks for updating that btw. | 10:16 |
jtv | I mean, thanks for the update _before_ the one I'm about to make. :) | 10:16 |
rvba | jtv: great. | 10:16 |
mgz | I take it the early errors in logs/webapp/current are inconsequential as it's repeated until it works? | 10:16 |
mgz | okay, this is all quite nice | 10:37 |
jtv | rvba: aigh. One problem with updating the cluster services file: start-cluster-controller will no longer exit! Do we have some utility at hand to wrap it in a daemon? | 10:47 |
jtv | start-stop-daemon -b? | 10:48 |
rvba | jtv: not sure I follow, you've removed start-cluster-controller? | 10:49 |
jtv | No, but it will no longer exit. | 10:50 |
jtv | It'll keep running forever. | 10:50 |
jtv | Well, waiting. :) | 10:50 |
jtv | And I'm guessing that our services machinery will want it to run in the background. | 10:50 |
rvba | Quite the opposite I think, hence the usage of 'exec' in these files. | 10:51 |
jtv | Ah, that makes sense! | 10:51 |
* jtv gives it a whirl | 10:52 | |
jtv | I'm ditching the fghack as well | 10:52 |
jtv | rvba: this is almost funny. With my change, "make run" seems to add a new celery (with 5 processes) every 10 seconds! | 11:07 |
rvba | jtv: the supervise stuff is probably unable to detect that the process is running ok, so it relaunches one every x seconds. | 11:07 |
jtv | Yeah I guess. :( | 11:08 |
rvba | jtv: IIRC that's precisely what happens when the process launched by the 'run' script gets daemonized. | 11:08 |
jtv | I wonder if re-introducing fghack helps then... | 11:10 |
jtv | The region worker otoh isn't shutting down properly. | 11:10 |
jtv | Now breaks my wooden shoe. | 11:11 |
jtv | (The literal equivalent of which is a Dutch expression for "WTF????") | 11:12 |
jtv | fghack seems to fix it. | 11:12 |
jtv | Oh well. Don't question the oracle. | 11:13 |
jtv | I'm out. allenap, could you have a look at my updated MP? | 11:35 |
mgz | jtv: lgtm, can I land it while you sleep? | 11:44 |
jam | mgz: well, I often do a quick 'cleanup' patch, and then try to submit it. | 11:52 |
jam | ix | 11:52 |
jam | mgz: however, gavin mentioned that you have to wait for the mp to update before you mark it approved. However, I waited until it saw 'Unmerged revisions' but not before the diff was updated. | 11:55 |
jam | it is possible that you need to sing and dance and sacrifice a chicken to get things to go any faster. | 11:57 |
jam | I haven't quite worked out the exact syntax yet. | 11:57 |
mgz | hm, that may be it, I tend to not mark approved instantly after pushing tweaks | 12:00 |
jam | mgz: well 'instantly', I do the tweaks and want to submit it before I forget about it and it sits for a day. I know I need to wait 'a little' but trying to figure out when I can push it out of my mental context is difficult. | 12:01 |
mgz | write a local launchpad script that sleeps for five mins then flips the status :D | 12:01 |
jam | mgz: at least with feed-pqm /pqm-submit we would check that the branch tip was up to date, and not have to wait for the N async processes that go from branch tip changed, to MP noticing, to approve state. | 12:02 |
allenap | mgz: I'll land jtv's branch in his absence. | 12:07 |
jam | jelmer: both of my api branches have landed | 12:07 |
jelmer | jam: \o/ | 12:07 |
mgz | allenap: he seemed to have remained awake just long enough to re-mark it approved :) | 12:08 |
mgz | ah, there is one of his pending still though | 12:09 |
allenap | I think we're concerned about different branches :) | 12:09 |
mgz | the scary celery exec change I wouldn't touch :) | 12:10 |
jam | mgz: hows the search stuff looking? | 12:18 |
mgz | I have html edited, need to wire everything up | 12:19 |
jam | mgz: did you know about huw's branch? | 12:19 |
jam | mgz: https://code.launchpad.net/~huwshimi/maas/hardware-search | 12:19 |
jam | (It was on the kanban board, but I realize you don't actually see that stuff) | 12:19 |
mgz | jam: no... | 12:20 |
mgz | I shall get that and examine | 12:20 |
mgz | okay, that seems all reasonably straight forward | 12:23 |
jam | mgz: I also should have some mockups for what it should look like | 12:24 |
jam | let me dig them up | 12:24 |
mgz | in lynx? :) | 12:24 |
mgz | I'll socks proxy so I can see the graphical view :D | 12:25 |
jelmer | mgz: have you tried using w3m or links? They should be able to do graphics if you have a framebuffer.. | 12:25 |
jam | mgz: https://docs.google.com/a/canonical.com/drawings/d/1AH_8gCyTYG6LfbjzYYjJQowpT-m8wq_P3oV5kL2XrmY/edit | 12:25 |
jam | and: https://docs.google.com/a/canonical.com/drawings/d/1NnBi_3bzpFjhbC6X8KYbh40qbj-FfxzlPJ40TbmXTGE/edit?pli=1 | 12:25 |
mgz | ta | 12:25 |
jam | the idea is that the search bar at the top should be everywhere, but when you get to the nodes/ page itself, it moves down into a larger view inside the page | 12:26 |
jam | in *my* head the main change is for that page to take an optional 'constraints' (search?) parameter | 12:26 |
jam | which then gets parsed from text form, into a dict, and essentially handed off to the filter code we just landed | 12:26 |
jam | mgz: note that some of that view will not be present in 12.10, some of it is 'future work'. Like if we don't have pagination yet, etc. | 12:27 |
smoser | jtv, did you sort the dhcpd.conf issue i was hitting yesterday? | 12:28 |
jam | and the Node list/search should use the same: 'macaddress (hostname)' that we currently have, rather than the 2 columns that is on that page. | 12:28 |
smoser | ah. i see. reading up. | 12:28 |
jam | rvba: so I can easily test that the helper function returns an indication that the task needs to be retried. Is there a way to test that the retry is done? | 12:42 |
jam | Or do you just patch out retry and assert that it is called? | 12:42 |
rvba | jam: no, you can test the retry for real, see test_rndc_command_is_retried_a_limited_number_of_times in src/provisioningserver/tests/test_tasks.py | 12:44 |
rvba | jam: that's why we've created MultiFakeMethod :) | 12:44 |
jam | rvba: so that test would succeed if the code just raised the exception on the first try | 12:48 |
jam | I don't see it asserting that it is actually retrying | 12:48 |
jam | I see that it is asserting it does eventually stop retrying | 12:48 |
jam | ah, I guess that is 'can_be_retried' | 12:48 |
rvba | jam: indeed, but you're right it's a bit implicit. | 12:49 |
jam | rvba: so how often is the refresh_secrets called? | 12:50 |
jam | rndc only waits 20s before failing | 12:50 |
jam | (10 tries at 2s each) | 12:50 |
rvba | jam: the main reason for retrying the rndc command is that bind sometimes takes some time to start up. | 12:51 |
jam | rvba: so the issue seems to be that it is possible for a queue to exist, but not have credentials to actually talk to the mass server yet | 12:51 |
jam | (mass_url may not be set, you may not have creds, you may not have a nodegroup.uuid yet) | 12:52 |
rvba | jam: I assume s/queue/nodegroup/ | 12:52 |
rvba | Queues are simply created on demand. | 12:53 |
jam | rvba: so I was told to do "async_queue(queue=nodegroup.uuid)" | 12:54 |
jam | in order to get them started on the right provisioning_server | 12:54 |
jam | instead of running it on the master | 12:54 |
jam | (each nodegroup's controller is meant to refresh the tag matching for the nodes under its control) | 12:55 |
jam | (so that master doesn't have to update 100,000 linearly, but each cluster can update their <10k nodes in a row.) | 12:55 |
rvba | Makes sense. | 12:55 |
jam | rvba: I would have thought that you need a queue around to make sure it is running on the right machine/worker/something | 12:55 |
rvba | jam: yeah, but queues are autocreated when the region sends tasks to it or when a cluster connects to a queue. | 12:56 |
rvba | Calling task.apply_async('my queue') will create 'my queue' and send task to it. | 12:57 |
rvba | Then the task just sits there until a celery worker feeds from that queue. | 12:57 |
rvba | All I'm saying is that you don't need to create the queues. | 12:58 |
jam | rvba: ok, but that means we need to call refresh_secrets before we call the task we want to do? | 13:01 |
rvba | jam: indeed. refresh_secrets gives to the cluster worker the credentials it needs to access the API. | 13:02 |
rvba | jam: glancing at the code, that is only called when the cluster controller starts up. | 13:02 |
jam | rvba: so the question is still open whether we can trust that the controller has creds or not. The other api has tests that assert it 'does nothing if it doesn't have creds' | 13:03 |
jam | and we *need* the work to be done | 13:03 |
jam | rvba: I got DC'd for abit. | 13:05 |
roaksoax | morning | 13:05 |
rvba | jam: you're right. Right now we trust that refresh_secrets was called when the cluster got started. | 13:05 |
jam | so the idea of putting in retry is that we need it to run once we get creds, or we can trust that we always have creds, or we always push creds out before we ask for the work to be done. | 13:05 |
rvba | I understand. This is a tricky problem. | 13:08 |
jam | rvba: if we could get an error message back we could go with that | 13:10 |
jam | or we can create our own 'queue' of work that is remaining to be done | 13:10 |
jam | in the db tables | 13:10 |
jam | and then create another async process that makes sure the queue is being worked on. | 13:10 |
jam | (since we can assert the transactional nature of the primary db, but not really the rabbit queues) | 13:10 |
rvba | That is exactly what celery-django does for you :) | 13:10 |
rvba | (A package we don't use atm) | 13:11 |
roaksoax | smoser: around already? | 13:11 |
jam | rvba: so should we try to do that, or should we just live with 'things may get out of date and we should add a "manuallly trigger a refresh"' in case stuff ever gets dropped. ? | 13:12 |
smoser | roaksoax, here. | 13:12 |
roaksoax | smoser: where do you tell commissioning to install packages? | 13:12 |
roaksoax | or, extra packages | 13:12 |
roaksoax | smoser: we need freeipmi-tools | 13:12 |
jam | without maas_url, there isn't even a place to put an anonymous 'something failed' handler. | 13:12 |
jam | that the celery worker can inform us it needs to be retried. | 13:12 |
rvba | Indeed, chicken and egg problem. | 13:13 |
jam | rvba: right, so either we write it as 'assume something has failed until the worker has said it succeeded' or ? | 13:15 |
jam | and if we do that, what task sits around making sure it gets retried? | 13:15 |
jam | you can poke at the API to get workers refreshed | 13:15 |
jam | and we can just call that immediately before we try to do work | 13:15 |
smoser | roaksoax, looking | 13:16 |
smoser | roaksoax, also | 13:16 |
smoser | https://bugs.launchpad.net/maas/+bug/1060942 | 13:16 |
ubot5 | Ubuntu bug 1060942 in MAAS "maas-cluster-celery job dies" [Undecided,New] | 13:16 |
rvba | Yeah, that sounds like a good way to do that quickly. Long term, it would be good to use celery-django for that kind of stuff. | 13:16 |
rvba | If celery-django gives us that possibility that is. I haven't looked into it really, but I think that's what it does: track the status of the tasks within the db. | 13:17 |
smoser | roaksoax, sorry for the slow response. | 13:20 |
smoser | you have to do it in that script there (./etc/maas/commissioning-user-data) | 13:20 |
smoser | that script gets sent to cloud-init verbatum as user-data | 13:20 |
smoser | (which it executes because of '#!') | 13:21 |
smoser | so you'll have to 'apt-get update' there | 13:21 |
roaksoax | smoser: ok cool thanks | 13:22 |
smoser | roaksoax, i'm kind of ton on this. | 13:22 |
smoser | on one side i feel ike i should put more into the imgaes | 13:22 |
smoser | (like maas-enlist and freeipmi-tools) | 13:23 |
smoser | but on the other, we have to be able to deliver updated versions of those *anyway* | 13:23 |
roaksoax | yeah | 13:23 |
smoser | so 'apt-get update && apt-get install' is pretty much required anyway | 13:23 |
roaksoax | putting them would indeed speed up the process, but we depend on they being updated | 13:24 |
roaksoax | at least maas-enlist | 13:24 |
smoser | right. | 13:24 |
smoser | so it wouldn't really speed it up | 13:24 |
smoser | in the end | 13:24 |
smoser | well, the deps would help if they're extensive | 13:24 |
smoser | (but they're not i dont think) | 13:24 |
smoser | rvba, roaksoax do you have thoughts on https://bugs.launchpad.net/maas/+bug/1060942 | 13:25 |
ubot5 | Ubuntu bug 1060942 in MAAS "maas-cluster-celery job dies" [Undecided,New] | 13:25 |
smoser | it looks to me like we've tried twice now to do this and both have failed | 13:25 |
smoser | i'm not exactly sure why celery can't setgid when it starts as root | 13:25 |
roaksoax | smoser: i'm looking at it now | 13:26 |
roaksoax | smoser: the upstart jobs used to set the gid/uid before, when running celerd directly | 13:27 |
roaksoax | celeryd* | 13:27 |
rvba | But now there is communication phase before we can start the celer worker for the cluster controller. | 13:27 |
roaksoax | smoser: which is what maas-region-controller does | 13:28 |
rvba | smoser: could it be caused by a limitation in upstart somewhere? | 13:29 |
roaksoax | rvba: so upstart stats the script as root, the root check in the wrapper passes just fine | 13:29 |
jam | mgz: so I'm heading out for the day, feel free to push up work in progress if you want me to take a look at it tomorrow before you wake up. | 13:30 |
jam | or if there is anything that is blocking you now? | 13:30 |
jam | rvba: nodegroup-ui branch was a merge conflict, not a test failing (from what I can tell) | 13:31 |
jam | rvba: but of course, I read it wrong. | 13:32 |
jam | I agree, the build says 'SUCCESS' at the end. | 13:32 |
mgz | jam: thanks, will do | 13:32 |
mgz | nothing blocking atm | 13:32 |
roaksoax | rvba: what log does cluster-controller creates? celery-cluster.log? | 13:32 |
jam | mgz: we have roughly 1 more day to land it, so I'm trying to make sure everything is as smooth as possible. | 13:33 |
rvba | jam: I don't see the merge conflict but I'll merge trunk anyway, just to be sure :). | 13:33 |
rvba | roaksoax: /var/log/maas/celery.log | 13:34 |
jam | rvba: well, be careful, since otherwise maas lander will reject it because the branch changed after it was approved :) | 13:34 |
smoser | i am confused on whats doing this | 13:34 |
smoser | i added | 13:34 |
smoser | print("user=%s group=%s uid=%s gid=%s curuid=%s curgid=%s" % (user, group, uid, gid, os.getuid(), os.getgid()) | 13:34 |
jam | I think I just misread the error message ,which looks to be 100% bogus as you mentioned. | 13:34 |
smoser | right before the fork/seuid/gid | 13:34 |
smoser | and i see | 13:34 |
mgz | jam: I was counting thursday and friday as two, but we want to be all done by tomorrow? | 13:35 |
jam | mgz: we need some time to get it into packaging,etc. | 13:35 |
smoser | user=maas group=maas uid=113 gid=120 curuid=0 curgid=0 | 13:35 |
jam | and jelmer is gone on friday regardless. | 13:35 |
smoser | so at that point i'm running as 0:0 and trying to go to 113:120 | 13:35 |
jam | so I would say "1-ish" | 13:35 |
mgz | okay, 1-ish it is. | 13:36 |
roaksoax | smoser: yeah that seems to be the issue | 13:36 |
smoser | ? | 13:36 |
smoser | i'm saying it looks right | 13:36 |
roaksoax | smoser: setgid | 13:36 |
smoser | curuid, curgid = (0,0) | 13:37 |
jam | mgz: did you get hardware_details for sampledata ? (not yet, I believe we deprioritized it, vs tag data and having search implemented) | 13:37 |
smoser | i'm just trying to drop gid to 120. | 13:37 |
smoser | why would i not be able to do that? | 13:37 |
roaksoax | smoser: i commented out the setgid and the process starts just fine | 13:37 |
smoser | http://stackoverflow.com/questions/4692720/operation-not-permitted-while-dropping-privileges-using-setuid-function | 13:37 |
mgz | jam: nope, but could trivially do that (something for mem and cpu_count may also be useful) | 13:38 |
smoser | wait. i'm confused by that though. | 13:38 |
roaksoax | smoser: [2012-10-03 09:39:14,571: WARNING/Beat] DBAccessError: (13, 'Permission denied') | 13:39 |
smoser | ah | 13:39 |
smoser | duh | 13:39 |
smoser | stuipd | 13:39 |
roaksoax | smoser: that's besides the gid/uid thing | 13:39 |
jam | mgz: so if you get search up and nobody reviews it, then look at the sampledata stuff :) | 13:41 |
jam | anyway, I'm off, have a good evening | 13:41 |
mgz | later! | 13:42 |
smoser | roaksoax, https://code.launchpad.net/~smoser/maas/lp1060942/+merge/127767 | 13:44 |
smoser | roaksoax, where did you see your error ? | 13:48 |
roaksoax | smoser: it is approaved | 13:49 |
rvba | smoser: if that really fixes the problem, you might want to change to comment also :) | 13:49 |
roaksoax | smoser: in /var/log/celery.log | 13:49 |
roaksoax | rvba: where does celery stores its db? | 13:49 |
roaksoax | or beat? | 13:49 |
smoser | yeah, i'm seeing that now | 13:49 |
rvba | roaksoax: don't know what the default is… hang on. | 13:50 |
roaksoax | http://comments.gmane.org/gmane.comp.python.amqp.celery.user/2375 | 13:51 |
roaksoax | smoser: ^^ | 13:51 |
rvba | roaksoax: looks like the default is to use /var/run/celerybeat-schedule | 13:52 |
roaksoax | rvba: so how can we do this: http://comments.gmane.org/gmane.comp.python.amqp.celery.user/2375 | 13:53 |
rvba | roaksoax: and the region is told explicitly to use /var/lib/maas/celerybeat-region-schedule in the upstart script. | 13:53 |
rvba | roaksoax: are you seing that error? | 13:54 |
smoser | so what is it supposed to be? | 13:54 |
roaksoax | rvba: yeah I'll test/fix | 13:55 |
roaksoax | smoser: it works | 13:55 |
roaksoax | rvba: it works :) | 13:55 |
smoser | what do we need to pass? | 13:55 |
smoser | whats the value we need to pass ? | 13:55 |
roaksoax | smoser: http://paste.ubuntu.com/1258008/ | 13:56 |
smoser | are we easily able to pass that in ? | 13:57 |
smoser | as an argument | 13:57 |
roaksoax | smoser: yes, we pass that in the upstart job | 13:58 |
roaksoax | rvba: should be just pass "/var/lib/maas/celerybeat-region-schedule" or should that be detected automatically | 13:59 |
roaksoax | rvba: it guess it would break make run? | 13:59 |
rvba | roaksoax: you're right, we can make it use a param that would be different in the local config. | 14:00 |
rvba | roaksoax: I can put together a branch to do that. | 14:00 |
roaksoax | rvba: awesome! that'd be great | 14:01 |
smoser | rvba, that just gets us to the next error | 14:01 |
rvba | smoser: what's the next error? | 14:01 |
smoser | http://paste.ubuntu.com/1258023/ | 14:01 |
rvba | smoser: the error in provisioningserver.tasks.upload_dhcp_leases ? | 14:02 |
smoser | but at least the celery seems up at that point | 14:02 |
smoser | it does suck that it dies permenently on that error | 14:02 |
smoser | hm.. | 14:02 |
rvba | Yeah, there is a task failing, we know about this one. | 14:02 |
smoser | yeah, this is really fragile | 14:02 |
rvba | We just need to make the task code deal with the fact that the lease file might not be there. | 14:02 |
smoser | but i'm not sure why upstart isn't re-starting it | 14:02 |
rvba | We're aware of this one. | 14:02 |
rvba | jtv: ^ ;) | 14:02 |
jtv | ? | 14:03 |
rvba | The dhcp task failing because the lease file is not there :). | 14:03 |
jtv | The cluster controller upstart job can't really respawn because the typical reason for failure is that the cluster controller has been rejected. | 14:04 |
rvba | Not urgent but if would be good to tweak the task so that it could cope elegantly with the situation. | 14:04 |
jtv | I agree. | 14:04 |
rvba | Got task from broker: provisioningserver.tasks.refresh_secrets | 14:04 |
rvba | Task provisioningserver.tasks.refresh_secrets[d8c820a8-d7cd-453b-9816-747ee2556927] succeeded in 0.201004981995s | 14:04 |
rvba | Contact with the region controller was ok. | 14:04 |
smoser | jtv, i dont understand the comment above. | 14:12 |
smoser | upstart will handle that most likely | 14:12 |
smoser | if the job dies 5 times in 5 seconds, upstart will give up on it | 14:13 |
jtv | Ah good | 14:13 |
smoser | if it doesn't hit that threshold, then, who cares. | 14:13 |
jtv | Just so long as Right | 14:13 |
jtv | Ahem. | 14:13 |
jtv | Right. | 14:13 |
jtv | Just so long as it doesn't loop wildly. | 14:13 |
smoser | who cares if a daemon keeps spawning that shouldnt | 14:13 |
smoser | it was configured wrong by the admin explicitly | 14:13 |
jtv | In this case we do care, because it makes API requests. | 14:13 |
smoser | in this case. | 14:13 |
smoser | so the way you could do this | 14:14 |
smoser | is a pre-start job | 14:14 |
smoser | that does the check for "am i accepted" | 14:14 |
smoser | and exit 0 if "no" | 14:14 |
smoser | ( i think that sthe right symantics) | 14:14 |
smoser | but then that would only run once | 14:14 |
jtv | We considered something like that, but decided it had to be this way. | 14:14 |
smoser | well then you need to make it re-spawn | 14:15 |
jtv | It needs to repeat in some cases. | 14:15 |
smoser | or its just going to die all the time. | 14:15 |
smoser | upstart respawns daemons for good reason. its generally considered the right thing to do. | 14:15 |
jtv | Well like I said, it's fine here too -- as long as it's not a tight endless loop. | 14:16 |
smoser | see 'man 5 init' look for 'respawn' | 14:16 |
smoser | how did you disable this ? | 14:16 |
jtv | I didn't. | 14:17 |
smoser | is it exiting success on stack trace? | 14:17 |
smoser | i'm confused why it isn't getting respawned | 14:17 |
jtv | I don't have the answer. | 14:17 |
jtv | But Julian told me it wasn't set up to respawn. | 14:18 |
rvba | roaksoax: https://code.launchpad.net/~rvb/maas/explicit-beat-schedule/+merge/127775 | 14:21 |
smoser | roaksoax, do you have ideas? | 14:22 |
allenap | smoser: respawn is not in maas-cluster-controller.maas-cluster-celery.upstart? | 14:23 |
allenap | smoser: Also, that setuid/setgid branch slipped in; I guess Tarmac got it too quickly. | 14:23 |
smoser | allenap, bah. | 14:23 |
smoser | i will do a revert branch | 14:24 |
allenap | smoser: If I get time I'll write a test for this ordering. | 14:24 |
allenap | smoser: In fact, I'll do the revert if you want, and add a test at the same time? | 14:25 |
roaksoax | smoser: why isn't what respawned? maas-cluster-celery due to the leases stack trace? | 14:25 |
roaksoax | ok I need to test the IPMI so gonna setup my network properly | 14:26 |
smoser | https://code.launchpad.net/~smoser/maas/revert-1151/+merge/127778 | 14:26 |
smoser | roaksoax, thats what i'm confused by | 14:26 |
smoser | allenap, well, there is the revert. | 14:26 |
smoser | it doesn't respawn. | 14:27 |
roaksoax | smoser: why are oyou confused by it? | 14:27 |
smoser | why does it not respawn | 14:27 |
smoser | http://paste.ubuntu.com/1258070/ | 14:27 |
smoser | default is to respawn | 14:27 |
smoser | (but i was explicit and added it there) | 14:28 |
roaksoax | smoser: hold on, you want it to respawn due to the failurs of the leases file? | 14:28 |
smoser | it should respawn | 14:30 |
smoser | its an upstart job | 14:30 |
smoser | and if it dies, it should not be fatal | 14:31 |
roaksoax | smoser: it is not fatal | 14:31 |
roaksoax | smoser: i see that error over and over | 14:31 |
roaksoax | smoser: but celery doesn't die | 14:31 |
roaksoax | smoser: it keeps running | 14:31 |
roaksoax | it only error's out | 14:31 |
smoser | i dont think so. | 14:31 |
smoser | it dies | 14:31 |
rvba | Yeah, it's just a failed task. | 14:31 |
smoser | the job dies at least. | 14:31 |
rvba | Nothing critical for celeryd. | 14:31 |
smoser | sudo status maas-cluster-celery stop/waiting | 14:32 |
roaksoax | smoser: did you psas the --schedule? | 14:32 |
smoser | http://paste.ubuntu.com/1258075/ | 14:33 |
roaksoax | smoser: and removed the old pyc files? | 14:33 |
smoser | it dies now because of the dhcp | 14:33 |
smoser | roaksoax, wouldn't you expect to see that pid change there? | 14:35 |
roaksoax | smoser: oh i thnk what it is | 14:36 |
roaksoax | smoser: jtv reported and error on which upstart couldn't track all the instances of celeryd | 14:36 |
roaksoax | smoser: that might be related | 14:36 |
jtv | It wasn't quite that. | 14:36 |
jtv | Upstart couldn't track the fork. As it turned out, there were several forks before the one that actually spawned the celeryd. | 14:37 |
jtv | So upstart couldn't track any instances at all. | 14:37 |
smoser | ok. i'll open a bug here. | 14:38 |
smoser | but this is fairly severe. | 14:38 |
roaksoax | smoser: https://pastebin.canonical.com/75796/ | 14:38 |
smoser | or am i incorrect | 14:38 |
roaksoax | smoser: i think i know why | 14:38 |
jtv | I'm getting bits and pieces of the conversation... what is the problem? | 14:39 |
smoser | maas-cluster-celery is fragile | 14:39 |
roaksoax | smoser: becuase maas-provision start-cluster-controller simply tells it to start celeryd and then it returns and exits | 14:39 |
smoser | in that if anything goes wrong it will die and never start | 14:39 |
smoser | requiring a human | 14:39 |
roaksoax | smoser: so maas-provision start-cluster-controller is not really a daemon | 14:39 |
jtv | roaksoax: it no longer does that | 14:39 |
jtv | It keeps running for as long as celeryd does. | 14:39 |
jtv | Well, it execs celeryd once it's got its approval etc. from the server. | 14:40 |
roaksoax | jtv: right, but for some reason the way it is started seems to not be tracked by upstart, and i think it is because it is not a daemon | 14:42 |
jtv | Still not being tracked? :( | 14:42 |
jtv | And it's so simple now! | 14:42 |
jtv | As it stands, "maas-provision start-cluster-controller" should only exit if (a) the cluster controller has been rejected, or (b) celeryd exits. | 14:44 |
jtv | Wow, that's a pretty lame slogan: "Stiebel Eltron. Originally German." | 14:45 |
smoser | hm.. | 14:45 |
jtv | It's like saying "Pizza Hut. One of us saw a pizza once and liked it so much we named the company after it." | 14:46 |
smoser | i must be misunderstanding something of upstart | 14:46 |
smoser | roaksoax, well. | 14:56 |
smoser | it doesn't respawn because you need the 'respawn' if you want it to | 14:57 |
smoser | ie, you need both: | 14:57 |
smoser | respawn | 14:57 |
smoser | respawn limit 5 60 | 14:57 |
roaksoax | ack | 14:59 |
smoser | suck. why didn't i realize that. | 14:59 |
smoser | anyway | 14:59 |
roaksoax | smoser: why did you revert the setgui/setuid thing? | 15:00 |
smoser | unless allenap was going to do it. | 15:00 |
smoser | allenap, roaksoax just ack https://code.launchpad.net/~smoser/maas/revert-1151/+merge/127778 | 15:01 |
smoser | to get my stupid change out | 15:01 |
allenap | smoser: +1 and approved. I'm writing a test for that code; I've noticed another bug at the same time. | 15:02 |
smoser | jtv, so you're working on something to make the dhcp work? | 15:02 |
smoser | rvba, do you know anything about mirrors in maas? | 15:11 |
rvba | smoser: not really, I just know that we have a config option named 'keep_mirror_list_uptodate' that is not used anywhere yet :/ | 15:12 |
smoser | right. and i saw a list of mirrors in the ui | 15:13 |
smoser | that contained 'archive.ubuntu.com' | 15:13 |
smoser | and could not be selected or changed | 15:13 |
smoser | :) | 15:13 |
smoser | i'm thinking we're kind of SOL on local mirrors at the moment. | 15:13 |
rvba | smoser: that can be changed and updated. See the 'update from' dropdown. | 15:15 |
rvba | smoser: but the 'Default distro series used for deployment' thing is positioned right in the middle of it. And that's confusing. | 15:15 |
smoser | hm.. i was just probably remembering incorrectly | 15:16 |
roaksoax | Daviey: do you happen to know how can I force IPMI to obtian a new IP address? (it is set to DHCP but doesn't get address) | 15:38 |
roaksoax | Daviey: nevermind i got it | 15:46 |
Daviey | roaksoax: how did you do it? | 15:47 |
roaksoax | Daviey: I set it to static, and then back to dhcp and it renewed it | 15:47 |
Daviey | hah, that is how i did it :) | 15:48 |
roaksoax | Daviey: :) | 15:48 |
roaksoax | smoser: http://paste.ubuntu.com/1258243/ so look at the first part | 15:53 |
roaksoax | adding the apt stuff that way? | 15:53 |
smoser | http://paste.ubuntu.com/1258248/ | 15:55 |
smoser | roaksoax, yes. basically right, but the above is safer | 15:55 |
smoser | (and, fwiw, the '-q' to apt doesn't really mean 'quiet' it means "produces output suitable for logging" | 15:56 |
smoser | which is what we'd want) | 15:56 |
roaksoax | smoser: ok cools | 15:59 |
mgz | yup, basically not using \r to print the same line multiple times with progress updates | 15:59 |
roaksoax | smoser: alright, so it looks something like: http://paste.ubuntu.com/1258286/ | 16:15 |
smoser | you have to quote fargs at 42 | 16:17 |
smoser | as it probably contains spaces, right? | 16:17 |
smoser | and you didnt patch signal all the way, right? | 16:18 |
roaksoax | smoser: right :) | 16:25 |
smoser | did you want help with that | 16:26 |
roaksoax | smoser: btw... --power-settings is json | 16:31 |
roaksoax | smoser: it reutrns this: ('IPMI', '{"power_address": "192.168.2.111", "power_pass": "8ruHtjzpGdU", "power_user": "maas"}') | 16:32 |
smoser | right. | 16:32 |
roaksoax | smoser: so like this? signal "$fargs" OK "power-settings sent [$power_settings]" | 16:35 |
smoser | rbasak, how easily could you test something | 16:38 |
smoser | specifically https://code.launchpad.net/~smoser/maas/preserve-sources-list/+merge/127825 | 16:39 |
smoser | roaksoax, let me look. | 16:41 |
smoser | allenap, or rvba, i'd appreciate your input on that merge ^ | 16:41 |
smoser | i'm sure the test could be cleaned up | 16:41 |
rbasak | smoser: my test node is being shipped to the MAAS QA lab now! | 16:42 |
rbasak | smoser: I need to switch my testing to another node, which shouldn't take long, but I have a hard stop soon. I can test it for you in the morning though? Normally it'd be about two commands! | 16:43 |
smoser | hm.. | 16:43 |
smoser | rbasak, how were you telling cloud-init preserve_source_list off | 16:43 |
smoser | err... on | 16:43 |
* rbasak looks for the patch | 16:44 | |
roaksoax | allenap: smoser how do you debug? something seems to have failed :( | 16:44 |
rbasak | smoser: in contrib/preseeds_v2/enlist | 16:45 |
rbasak | smoser: +apt_preserve_sources_list: true | 16:45 |
rbasak | smoser: that's all. And after deployment I fix manually. I presume this still breaks juju though. | 16:45 |
smoser | ah. | 16:45 |
smoser | what breaks juju? | 16:46 |
smoser | this would fix it. | 16:46 |
rbasak | sources.list being wrong after install | 16:46 |
rbasak | since it doesn't use that file on boot | 16:46 |
rbasak | Your fix is a proper fix I presume. | 16:46 |
smoser | right. my patch wuld make juju work. | 16:46 |
smoser | right. | 16:46 |
smoser | but, actually, doe not fix enlistment | 16:46 |
smoser | or commissioning | 16:47 |
smoser | (but those are sort of fixed for you already) | 16:47 |
smoser | by the ephemeral nodes having newer cloud-init | 16:47 |
roaksoax | smoser: https://pastebin.canonical.com/75814/ | 16:47 |
smoser | carp | 16:48 |
smoser | you're out of memory | 16:48 |
smoser | how much memory do you have there? | 16:48 |
roaksoax | smoser: memory as in ram? | 16:48 |
roaksoax | ah right | 16:48 |
roaksoax | dah | 16:48 |
smoser | as in ram | 16:48 |
smoser | :) | 16:48 |
roaksoax | 256 | 16:48 |
smoser | yeah. | 16:48 |
smoser | do you want to try to work around this? | 16:49 |
roaksoax | smoser: it is a VM | 16:49 |
roaksoax | smoser: so i'm just increasing memory | 16:49 |
smoser | oh. ok. | 16:49 |
smoser | i thought it was one of your servers | 16:49 |
smoser | but your vm wont have ipmi | 16:49 |
roaksoax | smoser: nope, just testing whether the scripts are running properly | 16:50 |
smoser | right | 16:50 |
roaksoax | smoser: btw.. you happen to have the link for the pastebin you got yesterday on how to send power settings back to MAAS? | 16:50 |
smoser | roaksoax, i dont remember pastein | 16:51 |
smoser | from you? | 16:51 |
roaksoax | smoser: from matsubara i thnk | 16:51 |
roaksoax | matsubara: ^^ do you know how to send power settings to maas? | 16:51 |
smoser | oh. i think that was something else. | 16:51 |
matsubara | maas-cli api maas node update node-7d828c3e-0902-11e2-8461-00e081ddd1cf power_type=ipmi power_parameters_power_address=192.168.22.33 power_parameters_power_user=root power_parameters_power_pass=ubuntu | 16:53 |
matsubara | roaksoax, ^ | 16:53 |
roaksoax | matsubara: thanks | 16:53 |
matsubara | np | 16:53 |
smoser | roaksoax, but that doens't help us. | 16:54 |
roaksoax | smoser: i know :( | 16:55 |
roaksoax | smoser: shouldn't we just send it as the other data you are sending? | 16:59 |
smoser | roaksoax, right. your patch looks good. we just have to tell that 'maas-signal' python script how to send the ppower settings. | 17:01 |
roaksoax | yeah | 17:02 |
roaksoax | smoser: we are still going to do the enlistment setting temporary ipmi credentials right? | 17:18 |
smoser | well, we dont have a way to post those at the moment. | 17:19 |
roaksoax | allenap: ^^ | 17:20 |
roaksoax | smoser: can i ssh to the commissioning images? | 17:23 |
smoser | i wouldnt think so. | 17:25 |
smoser | roaksoax, what i'd suggest is preparing the ephemeral image to let you in. | 17:26 |
roaksoax | smoser: will do | 17:26 |
roaksoax | smoser: so something is wrong and never returns from commissioning, but I think i know why | 17:26 |
roaksoax | smoser: and don't have console nor a monitor :( | 17:27 |
smoser | what system? | 17:27 |
smoser | physical system? | 17:27 |
smoser | do those hp microservers ipmi have remote serial console ? | 17:27 |
Daviey | seemed not. | 17:29 |
roaksoax | smoser: and the problem is that there's an issue with the kernel not correctly recognizing ipmi port and stuff | 17:33 |
smoser | oh joy. | 17:33 |
Daviey | you have to modprobe with custom stuff | 17:35 |
Daviey | modprobe ipmi_si type=kcs ports=0xca2 | 17:36 |
Daviey | But even so, with that - i didn't think you got console? | 17:36 |
roaksoax | Daviey: yeah i added that but doesn't seem to be working, | 17:43 |
roaksoax | smoser: ok it says that module ipmi_si does not exist on /etc/modules | 18:07 |
roaksoax | smoser: which basically tells us that we can't access the bmc :) | 18:08 |
smoser | ? | 18:08 |
smoser | where would that module come from? | 18:08 |
smoser | https://code.launchpad.net/~smoser/maas/preserve-sources-list/+merge/127825 | 18:08 |
smoser | ^ comments nplease! | 18:08 |
roaksoax | smoser: i'd say it is in the kernel | 18:09 |
smoser | smart @$!$ | 18:11 |
smoser | what says that, roaksoax | 18:11 |
roaksoax | smoser: rmmod but i just realizd i probably dont even need it if the modufle hasn;t been loaded | 18:13 |
smoser | so modprobe it? | 18:13 |
smoser | feel free to do that in the commissioning script | 18:13 |
roaksoax | smoser: my default? | 18:14 |
roaksoax | smoser: i alreayd did that but was removing it first, before loading it | 18:14 |
roaksoax | the thing is if the module is loaded already, then you need to remove it to load it again | 18:14 |
roaksoax | so the parameters take effect | 18:14 |
smoser | yes. | 18:16 |
smoser | you do. | 18:16 |
smoser | right. | 18:16 |
smoser | so that sucks. | 18:16 |
roaksoax | smoser: so it commissioned this time but the it didn't seem to have done the ipmi stuff | 18:17 |
roaksoax | smoser: so how can I enable ssh access? besides of course ssh-import-id | 18:17 |
smoser | http://bazaar.launchpad.net/~maas-maintainers/maas/trunk/revision/1089 | 18:18 |
smoser | roaksoax, what i'd do is just add another user | 18:18 |
roaksoax | oh wait i guess i can just install openssh-server | 18:18 |
smoser | mount loopback, add user, add ssh keys umount | 18:18 |
smoser | its installed, | 18:18 |
smoser | but by default you wont have access as there is no passowrd and no user and no ssh keys installed | 18:18 |
smoser | :) | 18:18 |
roaksoax | smoser: right, so can't i create a user from the commissioning script? | 18:19 |
smoser | ah. yeah, you could easily enough. | 18:19 |
roaksoax | smoser: does it run as 'ubuntu' user? | 18:19 |
smoser | that runs as root | 18:19 |
roaksoax | ack! | 18:20 |
roaksoax | smoser: and how to prevent it from powering off? | 18:22 |
roaksoax | trying to avoid having to mount it :) | 18:24 |
smoser | roaksoax, sleep! | 18:27 |
roaksoax | smoser: yeah figured :) | 18:27 |
smoser | i think that script is what powers off, right? | 18:27 |
roaksoax | nope | 18:27 |
smoser | yeah it does | 18:28 |
smoser | just disable the call to 'poweroff' | 18:28 |
smoser | err.. 'shutdown' | 18:28 |
roaksoax | ah lol | 18:28 |
smoser | (it calls that via 'trap' on exit) | 18:28 |
smoser | so just make that function return true | 18:28 |
smoser | or false | 18:29 |
smoser | doesn't actually matter | 18:29 |
smoser | :) | 18:29 |
roaksoax | :) | 18:31 |
roaksoax | smoser: i think what's happening is that it is not detecting IPMI so it continues normally | 18:31 |
roaksoax | smoser: yeah so it seems it is an issue with ipmi detection | 18:37 |
roaksoax | smoser: err modules | 18:37 |
roaksoax | smoser: ok so what i'm gonna do is simply enable the modules from the script, and look for errors i guess | 18:40 |
smoser | well, load the module in the script | 18:41 |
smoser | and generally ignore errors form that. | 18:41 |
roaksoax | smoser: like "modprobe XYZ || true" | 18:41 |
roaksoax | smoser: right so I was thinking, however, that what if the module is loaded (ipmi_si and needs to be reloaded?) | 18:42 |
roaksoax | dah | 18:42 |
roaksoax | duhh | 18:42 |
roaksoax | i guess i can do the same | 18:42 |
smoser | roaksoax, the script isprobably not set -e | 18:45 |
smoser | i dont generally do that. | 18:45 |
roaksoax | smoser: so do you think it is safe to wait 10 seconds for an IP address? | 18:59 |
smoser | safe as in too long or too short? | 18:59 |
roaksoax | smoser: enough time | 18:59 |
roaksoax | yes | 18:59 |
smoser | i think you might as well give it 60 at least. | 18:59 |
roaksoax | smoser: right, so I will give it 60 only if we had to change from Static to DHCP | 19:00 |
roaksoax | makes better sense that way? | 19:00 |
smoser | well, if you're dhcping in general | 19:02 |
smoser | why would you want to risk giving up early | 19:02 |
roaksoax | smoser: right, but if the machine turns on and starts doing comissioning, then it would be safe to assume that IPMI has an IP address already | 19:03 |
smoser | roaksoax, so then its dhcp request shoudl come back quickly | 19:07 |
smoser | or dhcp was the wrong setting | 19:08 |
roaksoax | smoser: right but for example, what happens if the image doesn't get an IP address quickly? | 19:11 |
roaksoax | smoser: it sits and waits for one right? | 19:11 |
smoser | an image? | 19:11 |
smoser | as in what | 19:11 |
smoser | node booting ephemeral image ? | 19:12 |
roaksoax | smoser: yes | 19:12 |
smoser | pxeboot annoying will time out | 19:12 |
roaksoax | smoser: right, but ok, we pass pxeboot, the image also request the IP address right? | 19:13 |
smoser | well,the kernel does. but only for the interface that got pxe booted | 19:15 |
smoser | (unless there is a bug) | 19:15 |
smoser | and i guess its not really the kernel | 19:15 |
smoser | its the initramfs | 19:15 |
smoser | roaksoax, you know how to access the maas db? | 19:29 |
roaksoax | smoser: yes, what do you need? | 19:30 |
smoser | i ended up getting in with sudo -Hu maas psql maasdb | 19:36 |
roaksoax | smoser: sudo maas shell sshoudl do | 19:36 |
smoser | i needed db | 19:38 |
smoser | not shell | 19:38 |
smoser | how do i put it into debug ? | 19:40 |
smoser | the api server | 19:40 |
roaksoax | smoser: ah lol :) but you can modify the db from the shell | 19:40 |
roaksoax | smoser: that i don't know | 19:40 |
roaksoax | smoser: btw... how can we log all the stuff that the commissioning script does? | 19:40 |
smoser | well, the stuff it calls gets logged back to the server | 19:41 |
smoser | which is what i'm testing) | 19:41 |
smoser | and why i was asking such silly things | 19:41 |
roaksoax | ah!! cool | 19:44 |
roaksoax | smoser: btw.. i think overtime it would be a good idea to make the commissioning/enlistment use the proxy if available | 19:44 |
smoser | proxy? | 19:45 |
smoser | roaksoax, http://paste.ubuntu.com/1258781/ | 20:01 |
smoser | ./maas-signal --config my.config --post power_type=ipmi --post "power_parameters={'blob': 'foo'}" WORKING "credsasdasdfasdfasdf" | 20:02 |
smoser | if you call the pastebin there, like that. it will post power params successfully. | 20:03 |
roaksoax | smoser: awesome thanks | 20:07 |
roaksoax | smoser: i'm trying to debug why it is not executing the script automatically but it is doing what it should if I run it manually on the ephemeral image | 20:07 |
roaksoax | smoser: any help would be appreciated | 20:08 |
roaksoax | http://paste.ubuntu.com/1258793/ | 20:08 |
smoser | roaksoax, http://paste.ubuntu.com/1258807/ | 20:14 |
smoser | replace the one there with this paste ^ | 20:14 |
smoser | and then call it like | 20:14 |
smoser | ./maas-signal --config my.config --power-type=ipmi '--power-parameters={}' WORKING "credsasdasdfasdfasdf" | 20:15 |
roaksoax | smoser: ok cool | 20:15 |
smoser | roaksoax, how did you think it was going to run that | 20:17 |
smoser | in your paste i dont see how it would run anything | 20:17 |
roaksoax | smoser: yeah just noticed :) | 20:17 |
smoser | oh. | 20:17 |
smoser | ah | 20:17 |
smoser | add_bin != add_script | 20:17 |
roaksoax | smoser: yeah, so, how do you think it should be called? | 20:18 |
roaksoax | smoser: should we do it from maas-signal directly? | 20:18 |
smoser | hm.. | 20:21 |
smoser | roaksoax, i think i'd just have main in that top level script handle this specifically. | 20:24 |
smoser | ie, add_bin to add it | 20:25 |
smoser | and then from main just invoke your script into a temp output | 20:25 |
smoser | then call maas-signal .... --power-par... | 20:25 |
roaksoax | smoser: so maas-signal is only called once right? | 20:45 |
smoser | no. | 20:45 |
smoser | it gets called before and after each script with WORKING | 20:45 |
smoser | and a status [1/X] | 20:45 |
smoser | and then once at the end | 20:45 |
smoser | it posts all the files | 20:45 |
smoser | with OK | 20:45 |
smoser | you can either add the params to the "OK" call | 20:46 |
smoser | or make a WORKING call separate | 20:46 |
smoser | i'm pretty sure the WORKING no longer do anything | 20:46 |
smoser | :) | 20:46 |
smoser | other than return "OK" | 20:46 |
smoser | roaksoax, does that all make sense? | 20:56 |
roaksoax | smoser: http://pastebin.ubuntu.com/1258893/ | 21:02 |
smoser | roaksoax, | 21:06 |
smoser | a.) uotes are bad | 21:06 |
smoser | b.) you have to say "WORKING" | 21:07 |
smoser | (and you have to say that before the one runs finished. | 21:07 |
smoser | well, quotes are bad is what it really amounts to. | 21:07 |
smoser | that make sense? | 21:08 |
smoser | i have to run for a few hours | 21:09 |
roaksoax | smoser: ok it works... but the settngs are not set in the server :S | 21:35 |
roaksoax | q | 21:35 |
smoser | roaksoax turn debug on on the server | 21:37 |
smoser | and you might see | 21:37 |
smoser | http://paste.ubuntu.com/1258943/ | 21:37 |
smoser | and look in the DB | 21:37 |
smoser | http://paste.ubuntu.com/1258944/ | 21:38 |
smoser | that above is after i did: | 21:38 |
smoser | ./maas-signal --config my.config --power-type=ipmi '--power-parameters={"a":"b"}' WORKING "credsasdasdfasdfasdf" | 21:38 |
smoser | it insists that it be valid json | 21:38 |
smoser | but thats about it really | 21:38 |
roaksoax | smoser: ah yes... i wonder why ... | 21:45 |
smoser | roaksoax, any thing furthre? | 22:38 |
smoser | we're pretty close. definitely the cmdline tool there can now post data. | 22:39 |
smoser | and it seems like you have a pretty good handle on getting it from the ipmi card | 22:39 |
smoser | if you'd like we can add some data manipulation into the 'maas-signal' if that is easier to work with rather than you feeding it a json blob | 22:40 |
roaksoax | smoser: yeah that could be even easier | 22:44 |
roaksoax | smoser: and no i dind't get any further | 22:45 |
smoser | ok. | 22:45 |
smoser | i have to run, and do not intend on being back tonight. | 22:45 |
roaksoax | smoser: ack! | 22:45 |
smoser | you figure out something that makes sense to you as a format | 22:45 |
smoser | have your tool dump that to a file | 22:45 |
roaksoax | smoser: i was just planning on "ip_address,user,pass" | 22:46 |
smoser | and i can write something to read and post it back | 22:46 |
roaksoax | smoser: as an argument | 22:46 |
smoser | sure. thats easy. | 22:46 |
roaksoax | and then split it in python and make it json | 22:46 |
smoser | yep. | 22:46 |
smoser | k. good night. | 22:46 |
roaksoax | night | 22:49 |
bigjools | morning | 23:15 |
roaksoax | bigjools: howdy | 23:20 |
roaksoax | bigjools: the meta-data api stuff for the power parameters doesn't seem to be saving | 23:40 |
bigjools | roaksoax: show me your client code | 23:41 |
roaksoax | bigjools: http://paste.ubuntu.com/1259090/ | 23:41 |
roaksoax | bigjools: check the part of power_parms in maas-signal | 23:42 |
roaksoax | bigjools: I see things like : MAASAPIBadRequest: Bad power_type 'ipmis' | 23:42 |
roaksoax | if I sent a bad power type | 23:42 |
bigjools | power_parms | 23:42 |
roaksoax | or bad parameters | 23:42 |
bigjools | there's your prob | 23:42 |
bigjools | oh wait | 23:42 |
roaksoax | bigjools: power_parms is converted into json, then passed to the server, we know it gets there/... but doesn't save | 23:43 |
roaksoax | bigjools: here's the whole thing: http://paste.ubuntu.com/1259093/ | 23:44 |
bigjools | roaksoax: `are you putting it in POST data? | 23:45 |
roaksoax | bigjools: yes, it is exaclt the same as sending stuff the other commissioning stuff | 23:46 |
roaksoax | bigjools: so for example, if the power_params are not json, MAAS complains | 23:46 |
bigjools | ok so it gets that far then | 23:46 |
bigjools | how do you know it's not saving ? | 23:46 |
roaksoax | bigjools: http://paste.ubuntu.com/1258943/ | 23:47 |
roaksoax | bigjools: becuase they are not displayed on the UI for the commissioned node | 23:47 |
roaksoax | bigjools: so it doesn't auto start the node | 23:47 |
bigjools | does everything else sent in the signal() get saved? | 23:48 |
bigjools | files, basically | 23:48 |
roaksoax | bigjools: yes | 23:49 |
bigjools | well that's odd then | 23:49 |
bigjools | can you check the database itself | 23:49 |
bigjools | rather than UI | 23:49 |
roaksoax | bigjools: >>> node.power_type | 23:50 |
roaksoax | u'' | 23:50 |
bigjools | argh | 23:50 |
bigjools | I see the bug | 23:50 |
bigjools | is node.power_parameters set? | 23:50 |
roaksoax | >>> node.power_type | 23:51 |
roaksoax | u'' | 23:51 |
roaksoax | >>> node.power_parameters | 23:51 |
roaksoax | {u'power_address': u'192.168.2.121', u'power_pass': u'KP0eOSy9Q9X4RZ', u'power_user': u'maas' | 23:51 |
roaksoax | it seems like it is | 23:51 |
bigjools | the code forgot to set the power_type | 23:51 |
bigjools | you can hack it locally to continue testing | 23:51 |
bigjools | I'll land a fix | 23:51 |
bigjools | sorry! | 23:51 |
roaksoax | bigjools: hehe no worries :) | 23:52 |
roaksoax | bigjools now it workds :D | 23:53 |
roaksoax | yuaaaay | 23:53 |
bigjools | \o/ | 23:53 |
bigjools | FWIW you should be able to call that any time to update power params | 23:53 |
roaksoax | bigjools: ok cool, but it is authenticated right? | 23:54 |
bigjools | all metadata access is authed | 23:54 |
bigjools | well, 99% | 23:54 |
roaksoax | bigjools: we might need that unauth for enlistment though :( | 23:54 |
bigjools | roaksoax: didn't we have this discussion already? :) | 23:55 |
bigjools | the answer was that you don't, the oauth key should be passed to enlisting nodes | 23:55 |
roaksoax | bigjools: yeah but smoser had another idea that we discussed this morning | 23:55 |
roaksoax | bigjools: it is basically to only set a temporary user/password for IPMI and send it back in enlistent | 23:55 |
bigjools | ok | 23:56 |
bigjools | let me think about this later | 23:56 |
roaksoax | sure, talk to smoser though :) | 23:56 |
* roaksoax has been all day working on this thing | 23:57 | |
roaksoax | so happy it works now | 23:57 |
roaksoax | lol | 23:57 |
bigjools | heh | 23:57 |
bigjools | this is why we are developers | 23:57 |
bigjools | better than drugs | 23:57 |
roaksoax | lol | 23:59 |
Generated by irclog2html.py 2.7 by Marius Gedminas - find it at mg.pov.lt!