/srv/irclogs.ubuntu.com/2014/04/07/#maas.txt

basicerSo I have my curtin preseed doing some custom partition stuff, and it works okay, but grub fails to install.01:03
basicerWhere can I see what commands its trying to run so I can fix it ?01:04
=== CyberJacob|Away is now known as CyberJacob
gmbjtv1: I’ve realised that I forgot to finish reviewing https://code.launchpad.net/~jtv/maas/boot-image-dict/+merge/214210 for you; profuse apologies. I’ll get right on it once I’ve finished this branch.07:20
gmbbigjools: Do we have a sample of what bootresources.yaml should look like  in order to pull in HWE kernels?07:21
gmbJust realised that my documentation reads “update bootresources.yaml appropriately” which is a bit like saying “use a goodly measure of spices”07:21
bigjoolsgmb: it's all in the subarchs07:21
bigjoolsso amd64/hwe-t for example07:22
gmbbigjools: Thanks.07:22
bigjoolsgmb: see http://maas.ubuntu.com/images/ephemeral-v2/daily/streams/v1/index.json07:22
gmbbigjools: Ah, perfect!07:22
gmbExcellent typo… “Hardware enoblement”07:23
* gmb fixes07:24
gmbbigjools: Could you use tags to set HWE kernels?07:40
bigjoolsgmb: no, it's driven entirely from subarchj07:41
* gmb is still murky about tag magic07:41
bigjools"tags are cheesy"07:41
gmb:)07:41
gmbbigjools: https://code.launchpad.net/~gmb/maas/hwe-docs/+merge/21447607:50
gmbCheese-free, hopefully07:50
bigjoolsgmb: cheers, will be on it shortly07:51
bigjoolsgmb: there;s code at the bottom of the diff07:56
gmbbigjools: Ah, sorry. colo-branch-using fail.08:08
gmbFixing.08:08
gmbbigjools: Don08:10
gmbe08:10
gmbbigjools: Thanks for the review. All good points. On it now.08:21
bigjoolsgmb, allenap: https://bugs.launchpad.net/maas/+bug/130215609:08
ubot5Ubuntu bug 1302156 in MAAS "maas upgrade 0072_remove_ipmi_autodetect fails" [Undecided,Incomplete]09:08
jtv1bigjools: I'm just putting up a fix for bug 1302772.09:17
ubot5bug 1302772 in MAAS 1.5 "update of maas-cluster-controller on trusty dumps traceback and crashes" [Critical,Triaged] https://launchpad.net/bugs/130277209:17
=== jtv1 is now known as jtv
bigjoolsjtv: you're a star09:18
bigjoolsjtv: I shall leave one of gmb or allenap to review that, as I am EOD now.  Thank you!09:18
jtvnp09:18
jtvAnd I see that gmb assigned me the card.09:19
gmbI’ll review it.09:19
gmbSeems fair09:19
jtvAccording to the timestamp, about 3 minutes before I ever mentioned it.  :)09:19
gmb:)09:19
jtvThanks!09:19
jtvDid you  finish reviewing my other branch?09:20
gmbjtv: I’m on it now.09:20
jtvAh OK09:20
* gmb designates self reviewmeister for this morning.09:20
jtvSchnell!  Schnell!09:20
strikovjtv: hey10:08
strikovjtv: how about extending TestMain to (1) have/pull multiple labels (2) have/pull .gz image to make sure that unpacking works10:10
jtvstrikov: Useful in themselves, but would complicate a test that is already quite overweight and potentially brittle...  It's always tempting to throw more complications into these tests, but also quite costly.10:34
jtvIf we can unit-test these points, great.10:34
jtv(Today's a holiday for me, so not around much)10:34
jtvIf we do the unpacking, then we can unit-test it.  If it's Simplestreams' job, then it's really something for the Simplestreams tests.10:36
strikovjtv: Okay, thanks, got your point. Enjoy your holiday :)10:44
jtvThanks!10:44
strikovjtv: I started to work on tests related to image unpacking (we do it, not ss) but met with one serious issue: euc2tgz tool (we use it to convert .img to .tgz) requires root permissions12:30
tych0hi allenap, so it looks like i do want to trigger the celery job manually on the import process13:41
tych0i guess you guys must do this for tests and stuff, is there a way to do that?13:41
jtvstrikov: ah yes, that tool... one thing you can try is fakeroot.13:55
jtvIt's like sudo, but fake.  :-)13:55
jtvfakeroot runs a child process that thinks it's root, a lot like sudo, but it doesn't actually get any of the privileges.13:55
jtvtych0: you can just run a celery task (i.e. a function defined with the @task decorator) by calling its delay() method.13:57
jtv(Yes, the functions have methods :)13:57
tych0jtv: ok, so i just import it and do footask.delay()?14:09
tych0cool14:09
jtvtych0: Yes.  And don't forget, when you need to patch out a celery task for a test, you actually need to patch out its delay() method!14:10
tych0jtv: yep, we actually want to run this one this time :-)14:10
jtvAh, that's easier.  :)14:10
jtvDuring tests, celery is wired up to execute tasks synchronously.14:11
jtvIn fact we've been sort of doubting the wisdom of that choice — because some of it gets run implicitly during tests for no good reason.14:11
webbrandonany of you ever ran into these when mining: http://paste.ubuntu.com/7217201/ ?  I just started a priate network server and all my miners get that, trying to think of why?14:25
jtvMining..?14:26
webbrandonoops14:26
webbrandonwromg chan14:26
jtvoic14:26
jtvgmb, rvba: I'm backporting my fix for the config bug (where the old "boot" section in pserv.yaml broke the upgrade).  No objections?14:47
rvbajtv: sounds good to me.14:47
tych0hi jtv, i'm getting http://paste.ubuntu.com/7217337/14:51
jtvtych0: new one on me... one of the guys in Europe may be able to help.14:53
tych0ok14:53
tych0rvba: ^^ ?14:53
tych0or allenap14:53
jtvCould be a celery change.14:53
tych0i think this is probably something you've always had to do14:54
tych0i vaguely remember rvba describing to me this before, at least14:54
jtvHmm yes, it's not a change in celery — it's something we do.14:54
rvbatych0: yes, hang on…14:54
rvbatych0: Wrong celery config.14:56
tych0ok, how do i pass the right one?14:56
rvbatych0: you have two celery config in /etc/maas, have a look at the upstart script to see what variable you need to define to load the right config.14:56
tych0ok14:58
tych0hi rvba, sorry for being slow, but: http://paste.ubuntu.com/7217420/15:13
tych0i'm not sure what variables to export exactly, based on my /etc/init/maas-cluster-celery.conf15:13
AskUbuntuMAAS / nodes have limited internet access after install | http://askubuntu.com/q/44456316:03
tych0allenap: any thoughts on my question above?16:18
allenaptych0: otp, biab16:18
tych0allenap: sure, no worries16:20
allenaptych0: Are you setting CELERY_CONFIG_MODULE in the environment before invoking this?16:24
allenapIt has a similar approach to Django’s settings module.16:24
allenaptych0: Are you trying to kick off the import job? That can be done via the remote API, but you don’t get any feedback, so I assume you want to run it directly?16:26
tych0allenap: yeah, viz. the paste above16:28
tych0i might be doing it wrong, though16:28
tych0allenap: oh, if we can do that via the api that is good enough16:28
allenaptych0: I was looking at the wrong paste :-/16:28
tych0allenap: we are going to run the import directly16:28
tych0and then run the celery job to do the import16:29
tych0and then check if it imported everything16:29
tych0so i guess it would be easiest if we could run it directly so it could fail for us16:29
tych0but not super important, since it probably (:-) won't fail16:29
tych0the import job itself is much more likely to fail16:29
allenaptych0: I don’t follow :) Do the import directly then run the celery job to do the import? Do you mean the celery job to report back to the region what’s been imported (report-boot-images or something like that)?16:31
tych0yep16:32
tych0the second one16:32
tych0ideally we'd run that celery job manually too16:32
tych0because we don't want to be stuck in a wait loop because it broke16:32
allenaptych0: Yeah :-/ Okay, I’ll see if I can figure out why that setting isn’t available.16:33
tych0allenap: ok, cool, thanks16:34
tych0i tried looking at celery16:34
tych0scary stuff :-)16:34
allenaptych0: Instead of /etc/maas in PYTHONPATH, use /usr/share/maas16:37
allenapOr, both.16:37
tych0allenap: bingo, thank you16:38
tych0now i get back an AsyncResult object, which isuppose mean things worked?16:38
allenaptych0: /etc/maas/maas_local_celeryconfig_cluster.py will probably only be readable by root:maas, so make sure you’re running as root or in the maas group; you won’t get any warning if it can’t read that module because of permissions.16:39
allenaptych0: Might have :) I don’t know what one of those is.16:39
tych0http://paste.ubuntu.com/7217779/16:39
allenapI don’t even know what a result backend is.16:41
allenaprvba: Are you around? ^16:41
lazyPowergreetings maasers. I've run into an interesting issue with maas on trusty server that appeared after I rebooted the machine. http://i.imgur.com/QsOuhYZ.jpg16:41
lazyPowerit has to do with iscsi target host, and the tgt bootstrapping, this error was hit before but skipped. Now that i've rebooted the maas region-controller (Which is also the cluster-controller) i now get halted on this during the pxe boot process16:42
tych0allenap: hmm, ok16:42
tych0it isn't showing up in the logs, so i don't think it is actually running16:42
=== deegee__ is now known as drussell
allenaplazyPower: I suspect that’s https://bugs.launchpad.net/maas/+bug/1300548. That will be fixed in trusty soon, but if you look at the diff in that change you can see that all you need to do is create a missing symlink (and probably restart tgt).16:45
ubot5Ubuntu bug 1300548 in maas (Ubuntu) "tgt targets do not persist after a reboot" [Undecided,Fix released]16:45
allenaptych0: Make sure you’ve got both /etc/maas *and* /usr/share/maas in PYTHONPATH.16:46
allenapI don’t think that’ll fix the bug you’re seeing, but it needs to be done anyway.16:46
tych0allenap: ok, yeah, didn't fix anything :-(16:48
lazyPoweraha! i knew i found it but closed the tab, and was unable to refind it.16:50
lazyPowerthanks allenap16:50
allenaptych0: I think this is happening because, when you call .delay(), it expects to be able to send that task off to a queue. In MAAS’s case, RabbitMQ.17:07
allenaptych0: You need to find the function to call that actually executes the body of the code.17:07
allenaprun() maybe, but that doesn’t seem to do anything here.17:08
tych0hmm, ok17:09
tych0just fiddling with somethign else, will look at it in a sec17:09
rvbaallenap: tych0: back with a decent internet connection…17:10
allenaptych0: You’re better off just calling maas-import-pxe-files!17:11
tych0allenap: that's what we're planning to do17:11
tych0but we need to also trigger the celery job to have it collect the images17:11
allenapThat’s all the task in p.tasks.report_boot_images does: calls sudo -n -E maas-import-pxe-files.17:11
rvbatych0: right, you can't use r.failed(): MAAS doesn't store the result of the tasks it sends.17:11
tych0otherwise we can't create a node17:11
tych0allenap: yeah, that's what we were doing before17:12
tych0now maas does things via celery vs. a simple filesystem check17:12
tych0so we hav eto trigger the celery job too17:12
allenaptych0: Yeah, ignore about the last 120 seconds from me :)17:13
allenapPossibly more :)17:14
tych0ideally we'd be able to trigger the celery job manually as well17:14
allenapI was getting muddled between report_boot_images and import_boot_images.17:14
tych0so i ncase that fails we get a non-zero exit code and know to inform the user17:14
allenaprvba: So, how do we execute a task directly? .run?17:18
rvbaallenap: task.apply_async()17:19
allenaptych0, rvba: I’ve just realised what’s going to cause an issue. The credentials aren’t in p.cache.cache (fwiw, p.cache.initialize() needs to be called early too).17:20
allenaprvba: Doesn’t apply_async() arrange for the task to be called later?17:21
rvbaallenap: yes, that's what I thought you meant.  Oh, you meant call the task's code manually?17:21
allenaprvba: Yeah. I think .run() is the badger.17:22
allenaptych0: Getting the credentials into p.cache.cache will result in total hair loss.17:23
allenapreport_boot_images() needs them to talk to the API.17:23
allenaptych0: Well… you might be able to fudge it.17:24
allenaptych0: Do you have a one:two:three credentials thing for the API to hand at the point you need to call report_boot_images()?17:25
tych0yep17:25
tych0so i can pass those in somehow17:25
allenaptych0: Here goes:17:27
allenapfrom provisioningserver import cache17:28
allenapcache.initialize()17:28
allenapfrom provisioningserver import auth17:28
allenapauth.record_api_credentials(“one:two:three”)17:28
allenapfrom provisioningserver import tasks17:28
allenaptasks.report_boot_images(http_proxy=…)17:28
allenapWatch out for the quotes above; my machine automatically changes them to typographical quotes.17:28
tych0yeah, saw that17:28
tych0r.e. the quotes17:28
tych0allenap: cool, will try now17:29
allenaptych0: Eugh, ignore the http_proxy bit. I am yet again getting muddles.17:30
allenapmuddled too.17:30
allenaptych0: I have to go now, but I’ll be back in ~2 hours.17:34
tych0allenap: ok, it looks like this won't work exactly, but i think i can get it to work17:34
tych0need to source /etc/maas/celery.conf and some other stuff17:34
tych0but this is a lot further than i would have gotten myself17:34
tych0i will let you know what i come up with17:35
allenaptych0: Ah, yes! Source /etc/maas/maas_cluster.conf and export MAAS_URL and CLUSTER_UUID.17:35
tych0yep17:35
tych0arg17:41
tych0403 forbidden17:41
tych0http://paste.ubuntu.com/7218034/17:41
Fishy_whats it mean when I can't import boot images on a new install, and I get  "  File "/usr/lib/python2.7/dist-packages/formencode/schema.py", line 145, in _to_python     value_dict, state) Invalid: The input field 'username' was not expected."17:45
rvbaFishy_: doesn't look good at all, could you please file a bug with the full stack trace, the version of MAAS you're using, the exact MAAS configuration, and what you've done to get that stacktrace?17:50
Fishy_okay17:51
Fishy_fresh install off CD17:51
Fishy_https://bugs.launchpad.net/maas/+bug/130393517:59
ubot5Ubuntu bug 1303935 in MAAS "unable to import boot images in fresh install" [Undecided,New]17:59
rvbaFishy_: ta.  I can't have a look at it right now but we will get back to you quickly.18:01
Fishy_cool!18:03
Fishy_im sure its me18:03
_bjornewhy im always get this answer in my access.log?!  200 is ok, 404 is error ----> GET /MAAS/metadata//2012-03-01/user-data HTTP/1.1" 404 200 "-" "Python-urllib/2.718:54
tych0hi smoser, what can you tell me about the new import script and tftp timeouts?19:23
tych0it looks like my /var/lib/maas/tftp is totally empty, i assume it shouldn't be?19:23
smoserthat woudl seem bad.19:24
smoserbut it might be goign to ta different place.19:25
tych0smoser: http://paste.ubuntu.com/7218450/19:25
smosertych0, that seems plausible19:26
tych0smoser: ok, so that might work (?)19:27
tych0i could bomb boot-resources and try again19:27
smosertych0, id' avhe to look closer, but generally it seems sane.19:32
smoseri dont knwo about tftp timeouts though19:32
smoseryou ahve more info there?19:32
tych0smoser: no, it pxe boots, gets an ip, and then never gets a kernel from tftp19:33
tych0smoser: is there some more info i can collect somewhere?19:33
smoserdoes it get the pxe config ?19:36
smoserthis is intel ?19:36
tych0smoser: amd64, no, it doesn't get the pxe config19:37
tych0PXE-E32: TFTP open timeout19:38
tych0PXE-M0F: Exiting Intel Boot Agent.19:38
AskUbuntuMAAS Controller & KVM Help | http://askubuntu.com/q/44464819:41
smosertych0, is the pserv running ?19:42
tych0well19:42
tych0i restarted it but i don't think it came up, actually19:42
tych0sec19:42
tych0heh...19:43
tych0http://paste.ubuntu.com/7218525/19:43
tych0upstart's maas-pserv.log19:43
smosertych0, is it crashing on a tftp request?19:48
smoseror crashing just when run19:48
tych0smoser: got it figure dout actually19:48
tych0will file a bug shortly19:48
tych0smoser: should i still expect things in tgt-admin --show?19:57
smosertych0, yes19:57
tych0now iscsistart is timing out19:57
tych0and there isn't anything there19:57
tych0(in tgt-admin --show19:57
tych0)19:57
Jeffrey_Good afternoon, I am currently using MaaS 1.5 Trusty Thar and after a fast install using curtin on a node the server doesn't reboot. I was wondering if this is the expected action? I was using a previous MaaS 1.4 Saucy and after a fast install the machine would reboot and then be ready.19:58
smoserconsole log is the most useful.20:06
smoserif it as trusty ephemeral image20:06
smoserthen you will have /var/log/cloud-init-output.log20:07
smoserand /var/log/cloud-init.log , looking for WARN will probably shoul you something20:07
Jeffrey_smoser: Alright awesome thanks for your help! I'm going to go try some things now.20:09
allenaptych0: How did you get on in the end?20:36
tych0allenap: https://bugs.launchpad.net/maas/+bug/1302772 was what i was hitting20:36
ubot5`Ubuntu bug 1302772 in MAAS "update of maas-cluster-controller on trusty dumps traceback and crashes" [Critical,In progress]20:36
allenaptych0: Do you have a workaround for that?21:01
tych0allenap: yeah, just delete that whole section from the config file21:02
allenaptych0: We’ll have a fix for that in time for Trusty, so as long as you can bear it for now it should be fine.21:03
tych0allenap: yep, that worked21:03
allenapTip top.21:03
=== CyberJacob is now known as CyberJacob|Away
=== CyberJacob|Away is now known as CyberJacob
lazyPowerlooks like we have some new fun stuff on the horizon with import-pxe-files, i got an invalid key signature error21:20
lazyPowerhttp://paste.ubuntu.com/7218962/21:20
lazyPowerhowever, re-running the command completed successfully. Maybe a temporary hiccup21:21
tych0lazyPower: those images probably aren't signed by the archive key21:24
lazyPowerits strange that re-running the command completed without issue. I think it was just a temporary hiccup?21:25
tych0shoudl be signed by the key in ubuntu-cloudimage-keyring it hink21:25
lazyPowerbecause if its failing crc validation and bailing on the whole process, it should have repeated the same behavior21:25
=== ubot5` is now known as ubot5
=== CyberJacob is now known as CyberJacob|Away
=== bigjools_ is now known as bigjools

Generated by irclog2html.py 2.7 by Marius Gedminas - find it at mg.pov.lt!