[01:03] So I have my curtin preseed doing some custom partition stuff, and it works okay, but grub fails to install. [01:04] Where can I see what commands its trying to run so I can fix it ? === CyberJacob|Away is now known as CyberJacob [07:20] jtv1: I’ve realised that I forgot to finish reviewing https://code.launchpad.net/~jtv/maas/boot-image-dict/+merge/214210 for you; profuse apologies. I’ll get right on it once I’ve finished this branch. [07:21] bigjools: Do we have a sample of what bootresources.yaml should look like in order to pull in HWE kernels? [07:21] Just realised that my documentation reads “update bootresources.yaml appropriately” which is a bit like saying “use a goodly measure of spices” [07:21] gmb: it's all in the subarchs [07:22] so amd64/hwe-t for example [07:22] bigjools: Thanks. [07:22] gmb: see http://maas.ubuntu.com/images/ephemeral-v2/daily/streams/v1/index.json [07:22] bigjools: Ah, perfect! [07:23] Excellent typo… “Hardware enoblement” [07:24] * gmb fixes [07:40] bigjools: Could you use tags to set HWE kernels? [07:41] gmb: no, it's driven entirely from subarchj [07:41] * gmb is still murky about tag magic [07:41] "tags are cheesy" [07:41] :) [07:50] bigjools: https://code.launchpad.net/~gmb/maas/hwe-docs/+merge/214476 [07:50] Cheese-free, hopefully [07:51] gmb: cheers, will be on it shortly [07:56] gmb: there;s code at the bottom of the diff [08:08] bigjools: Ah, sorry. colo-branch-using fail. [08:08] Fixing. [08:10] bigjools: Don [08:10] e [08:21] bigjools: Thanks for the review. All good points. On it now. [09:08] gmb, allenap: https://bugs.launchpad.net/maas/+bug/1302156 [09:08] Ubuntu bug 1302156 in MAAS "maas upgrade 0072_remove_ipmi_autodetect fails" [Undecided,Incomplete] [09:17] bigjools: I'm just putting up a fix for bug 1302772. [09:17] bug 1302772 in MAAS 1.5 "update of maas-cluster-controller on trusty dumps traceback and crashes" [Critical,Triaged] https://launchpad.net/bugs/1302772 === jtv1 is now known as jtv [09:18] jtv: you're a star [09:18] jtv: I shall leave one of gmb or allenap to review that, as I am EOD now. Thank you! [09:18] np [09:19] And I see that gmb assigned me the card. [09:19] I’ll review it. [09:19] Seems fair [09:19] According to the timestamp, about 3 minutes before I ever mentioned it. :) [09:19] :) [09:19] Thanks! [09:20] Did you finish reviewing my other branch? [09:20] jtv: I’m on it now. [09:20] Ah OK [09:20] * gmb designates self reviewmeister for this morning. [09:20] Schnell! Schnell! [10:08] jtv: hey [10:10] jtv: how about extending TestMain to (1) have/pull multiple labels (2) have/pull .gz image to make sure that unpacking works [10:34] strikov: Useful in themselves, but would complicate a test that is already quite overweight and potentially brittle... It's always tempting to throw more complications into these tests, but also quite costly. [10:34] If we can unit-test these points, great. [10:34] (Today's a holiday for me, so not around much) [10:36] If we do the unpacking, then we can unit-test it. If it's Simplestreams' job, then it's really something for the Simplestreams tests. [10:44] jtv: Okay, thanks, got your point. Enjoy your holiday :) [10:44] Thanks! [12:30] jtv: I started to work on tests related to image unpacking (we do it, not ss) but met with one serious issue: euc2tgz tool (we use it to convert .img to .tgz) requires root permissions [13:41] hi allenap, so it looks like i do want to trigger the celery job manually on the import process [13:41] i guess you guys must do this for tests and stuff, is there a way to do that? [13:55] strikov: ah yes, that tool... one thing you can try is fakeroot. [13:55] It's like sudo, but fake. :-) [13:55] fakeroot runs a child process that thinks it's root, a lot like sudo, but it doesn't actually get any of the privileges. [13:57] tych0: you can just run a celery task (i.e. a function defined with the @task decorator) by calling its delay() method. [13:57] (Yes, the functions have methods :) [14:09] jtv: ok, so i just import it and do footask.delay()? [14:09] cool [14:10] tych0: Yes. And don't forget, when you need to patch out a celery task for a test, you actually need to patch out its delay() method! [14:10] jtv: yep, we actually want to run this one this time :-) [14:10] Ah, that's easier. :) [14:11] During tests, celery is wired up to execute tasks synchronously. [14:11] In fact we've been sort of doubting the wisdom of that choice — because some of it gets run implicitly during tests for no good reason. [14:25] any of you ever ran into these when mining: http://paste.ubuntu.com/7217201/ ? I just started a priate network server and all my miners get that, trying to think of why? [14:26] Mining..? [14:26] oops [14:26] wromg chan [14:26] oic [14:47] gmb, rvba: I'm backporting my fix for the config bug (where the old "boot" section in pserv.yaml broke the upgrade). No objections? [14:47] jtv: sounds good to me. [14:51] hi jtv, i'm getting http://paste.ubuntu.com/7217337/ [14:53] tych0: new one on me... one of the guys in Europe may be able to help. [14:53] ok [14:53] rvba: ^^ ? [14:53] or allenap [14:53] Could be a celery change. [14:54] i think this is probably something you've always had to do [14:54] i vaguely remember rvba describing to me this before, at least [14:54] Hmm yes, it's not a change in celery — it's something we do. [14:54] tych0: yes, hang on… [14:56] tych0: Wrong celery config. [14:56] ok, how do i pass the right one? [14:56] tych0: you have two celery config in /etc/maas, have a look at the upstart script to see what variable you need to define to load the right config. [14:58] ok [15:13] hi rvba, sorry for being slow, but: http://paste.ubuntu.com/7217420/ [15:13] i'm not sure what variables to export exactly, based on my /etc/init/maas-cluster-celery.conf [16:03] MAAS / nodes have limited internet access after install | http://askubuntu.com/q/444563 [16:18] allenap: any thoughts on my question above? [16:18] tych0: otp, biab [16:20] allenap: sure, no worries [16:24] tych0: Are you setting CELERY_CONFIG_MODULE in the environment before invoking this? [16:24] It has a similar approach to Django’s settings module. [16:26] tych0: Are you trying to kick off the import job? That can be done via the remote API, but you don’t get any feedback, so I assume you want to run it directly? [16:28] allenap: yeah, viz. the paste above [16:28] i might be doing it wrong, though [16:28] allenap: oh, if we can do that via the api that is good enough [16:28] tych0: I was looking at the wrong paste :-/ [16:28] allenap: we are going to run the import directly [16:29] and then run the celery job to do the import [16:29] and then check if it imported everything [16:29] so i guess it would be easiest if we could run it directly so it could fail for us [16:29] but not super important, since it probably (:-) won't fail [16:29] the import job itself is much more likely to fail [16:31] tych0: I don’t follow :) Do the import directly then run the celery job to do the import? Do you mean the celery job to report back to the region what’s been imported (report-boot-images or something like that)? [16:32] yep [16:32] the second one [16:32] ideally we'd run that celery job manually too [16:32] because we don't want to be stuck in a wait loop because it broke [16:33] tych0: Yeah :-/ Okay, I’ll see if I can figure out why that setting isn’t available. [16:34] allenap: ok, cool, thanks [16:34] i tried looking at celery [16:34] scary stuff :-) [16:37] tych0: Instead of /etc/maas in PYTHONPATH, use /usr/share/maas [16:37] Or, both. [16:38] allenap: bingo, thank you [16:38] now i get back an AsyncResult object, which isuppose mean things worked? [16:39] tych0: /etc/maas/maas_local_celeryconfig_cluster.py will probably only be readable by root:maas, so make sure you’re running as root or in the maas group; you won’t get any warning if it can’t read that module because of permissions. [16:39] tych0: Might have :) I don’t know what one of those is. [16:39] http://paste.ubuntu.com/7217779/ [16:41] I don’t even know what a result backend is. [16:41] rvba: Are you around? ^ [16:41] greetings maasers. I've run into an interesting issue with maas on trusty server that appeared after I rebooted the machine. http://i.imgur.com/QsOuhYZ.jpg [16:42] it has to do with iscsi target host, and the tgt bootstrapping, this error was hit before but skipped. Now that i've rebooted the maas region-controller (Which is also the cluster-controller) i now get halted on this during the pxe boot process [16:42] allenap: hmm, ok [16:42] it isn't showing up in the logs, so i don't think it is actually running === deegee__ is now known as drussell [16:45] lazyPower: I suspect that’s https://bugs.launchpad.net/maas/+bug/1300548. That will be fixed in trusty soon, but if you look at the diff in that change you can see that all you need to do is create a missing symlink (and probably restart tgt). [16:45] Ubuntu bug 1300548 in maas (Ubuntu) "tgt targets do not persist after a reboot" [Undecided,Fix released] [16:46] tych0: Make sure you’ve got both /etc/maas *and* /usr/share/maas in PYTHONPATH. [16:46] I don’t think that’ll fix the bug you’re seeing, but it needs to be done anyway. [16:48] allenap: ok, yeah, didn't fix anything :-( [16:50] aha! i knew i found it but closed the tab, and was unable to refind it. [16:50] thanks allenap [17:07] tych0: I think this is happening because, when you call .delay(), it expects to be able to send that task off to a queue. In MAAS’s case, RabbitMQ. [17:07] tych0: You need to find the function to call that actually executes the body of the code. [17:08] run() maybe, but that doesn’t seem to do anything here. [17:09] hmm, ok [17:09] just fiddling with somethign else, will look at it in a sec [17:10] allenap: tych0: back with a decent internet connection… [17:11] tych0: You’re better off just calling maas-import-pxe-files! [17:11] allenap: that's what we're planning to do [17:11] but we need to also trigger the celery job to have it collect the images [17:11] That’s all the task in p.tasks.report_boot_images does: calls sudo -n -E maas-import-pxe-files. [17:11] tych0: right, you can't use r.failed(): MAAS doesn't store the result of the tasks it sends. [17:11] otherwise we can't create a node [17:12] allenap: yeah, that's what we were doing before [17:12] now maas does things via celery vs. a simple filesystem check [17:12] so we hav eto trigger the celery job too [17:13] tych0: Yeah, ignore about the last 120 seconds from me :) [17:14] Possibly more :) [17:14] ideally we'd be able to trigger the celery job manually as well [17:14] I was getting muddled between report_boot_images and import_boot_images. [17:14] so i ncase that fails we get a non-zero exit code and know to inform the user [17:18] rvba: So, how do we execute a task directly? .run? [17:19] allenap: task.apply_async() [17:20] tych0, rvba: I’ve just realised what’s going to cause an issue. The credentials aren’t in p.cache.cache (fwiw, p.cache.initialize() needs to be called early too). [17:21] rvba: Doesn’t apply_async() arrange for the task to be called later? [17:21] allenap: yes, that's what I thought you meant. Oh, you meant call the task's code manually? [17:22] rvba: Yeah. I think .run() is the badger. [17:23] tych0: Getting the credentials into p.cache.cache will result in total hair loss. [17:23] report_boot_images() needs them to talk to the API. [17:24] tych0: Well… you might be able to fudge it. [17:25] tych0: Do you have a one:two:three credentials thing for the API to hand at the point you need to call report_boot_images()? [17:25] yep [17:25] so i can pass those in somehow [17:27] tych0: Here goes: [17:28] from provisioningserver import cache [17:28] cache.initialize() [17:28] from provisioningserver import auth [17:28] auth.record_api_credentials(“one:two:three”) [17:28] from provisioningserver import tasks [17:28] tasks.report_boot_images(http_proxy=…) [17:28] Watch out for the quotes above; my machine automatically changes them to typographical quotes. [17:28] yeah, saw that [17:28] r.e. the quotes [17:29] allenap: cool, will try now [17:30] tych0: Eugh, ignore the http_proxy bit. I am yet again getting muddles. [17:30] muddled too. [17:34] tych0: I have to go now, but I’ll be back in ~2 hours. [17:34] allenap: ok, it looks like this won't work exactly, but i think i can get it to work [17:34] need to source /etc/maas/celery.conf and some other stuff [17:34] but this is a lot further than i would have gotten myself [17:35] i will let you know what i come up with [17:35] tych0: Ah, yes! Source /etc/maas/maas_cluster.conf and export MAAS_URL and CLUSTER_UUID. [17:35] yep [17:41] arg [17:41] 403 forbidden [17:41] http://paste.ubuntu.com/7218034/ [17:45] whats it mean when I can't import boot images on a new install, and I get " File "/usr/lib/python2.7/dist-packages/formencode/schema.py", line 145, in _to_python value_dict, state) Invalid: The input field 'username' was not expected." [17:50] Fishy_: doesn't look good at all, could you please file a bug with the full stack trace, the version of MAAS you're using, the exact MAAS configuration, and what you've done to get that stacktrace? [17:51] okay [17:51] fresh install off CD [17:59] https://bugs.launchpad.net/maas/+bug/1303935 [17:59] Ubuntu bug 1303935 in MAAS "unable to import boot images in fresh install" [Undecided,New] [18:01] Fishy_: ta. I can't have a look at it right now but we will get back to you quickly. [18:03] cool! [18:03] im sure its me [18:54] <_bjorne> why im always get this answer in my access.log?! 200 is ok, 404 is error ----> GET /MAAS/metadata//2012-03-01/user-data HTTP/1.1" 404 200 "-" "Python-urllib/2.7 [19:23] hi smoser, what can you tell me about the new import script and tftp timeouts? [19:23] it looks like my /var/lib/maas/tftp is totally empty, i assume it shouldn't be? [19:24] that woudl seem bad. [19:25] but it might be goign to ta different place. [19:25] smoser: http://paste.ubuntu.com/7218450/ [19:26] tych0, that seems plausible [19:27] smoser: ok, so that might work (?) [19:27] i could bomb boot-resources and try again [19:32] tych0, id' avhe to look closer, but generally it seems sane. [19:32] i dont knwo about tftp timeouts though [19:32] you ahve more info there? [19:33] smoser: no, it pxe boots, gets an ip, and then never gets a kernel from tftp [19:33] smoser: is there some more info i can collect somewhere? [19:36] does it get the pxe config ? [19:36] this is intel ? [19:37] smoser: amd64, no, it doesn't get the pxe config [19:38] PXE-E32: TFTP open timeout [19:38] PXE-M0F: Exiting Intel Boot Agent. [19:41] MAAS Controller & KVM Help | http://askubuntu.com/q/444648 [19:42] tych0, is the pserv running ? [19:42] well [19:42] i restarted it but i don't think it came up, actually [19:42] sec [19:43] heh... [19:43] http://paste.ubuntu.com/7218525/ [19:43] upstart's maas-pserv.log [19:48] tych0, is it crashing on a tftp request? [19:48] or crashing just when run [19:48] smoser: got it figure dout actually [19:48] will file a bug shortly [19:57] smoser: should i still expect things in tgt-admin --show? [19:57] tych0, yes [19:57] now iscsistart is timing out [19:57] and there isn't anything there [19:57] (in tgt-admin --show [19:57] ) [19:58] Good afternoon, I am currently using MaaS 1.5 Trusty Thar and after a fast install using curtin on a node the server doesn't reboot. I was wondering if this is the expected action? I was using a previous MaaS 1.4 Saucy and after a fast install the machine would reboot and then be ready. [20:06] console log is the most useful. [20:06] if it as trusty ephemeral image [20:07] then you will have /var/log/cloud-init-output.log [20:07] and /var/log/cloud-init.log , looking for WARN will probably shoul you something [20:09] smoser: Alright awesome thanks for your help! I'm going to go try some things now. [20:36] tych0: How did you get on in the end? [20:36] allenap: https://bugs.launchpad.net/maas/+bug/1302772 was what i was hitting [20:36] Ubuntu bug 1302772 in MAAS "update of maas-cluster-controller on trusty dumps traceback and crashes" [Critical,In progress] [21:01] tych0: Do you have a workaround for that? [21:02] allenap: yeah, just delete that whole section from the config file [21:03] tych0: We’ll have a fix for that in time for Trusty, so as long as you can bear it for now it should be fine. [21:03] allenap: yep, that worked [21:03] Tip top. === CyberJacob is now known as CyberJacob|Away === CyberJacob|Away is now known as CyberJacob [21:20] looks like we have some new fun stuff on the horizon with import-pxe-files, i got an invalid key signature error [21:20] http://paste.ubuntu.com/7218962/ [21:21] however, re-running the command completed successfully. Maybe a temporary hiccup [21:24] lazyPower: those images probably aren't signed by the archive key [21:25] its strange that re-running the command completed without issue. I think it was just a temporary hiccup? [21:25] shoudl be signed by the key in ubuntu-cloudimage-keyring it hink [21:25] because if its failing crc validation and bailing on the whole process, it should have repeated the same behavior === ubot5` is now known as ubot5 === CyberJacob is now known as CyberJacob|Away === bigjools_ is now known as bigjools