[01:03] <basicer> So I have my curtin preseed doing some custom partition stuff, and it works okay, but grub fails to install.
[01:04] <basicer> Where can I see what commands its trying to run so I can fix it ?
[07:20] <gmb> jtv1: I’ve realised that I forgot to finish reviewing https://code.launchpad.net/~jtv/maas/boot-image-dict/+merge/214210 for you; profuse apologies. I’ll get right on it once I’ve finished this branch.
[07:21] <gmb> bigjools: Do we have a sample of what bootresources.yaml should look like  in order to pull in HWE kernels?
[07:21] <gmb> Just realised that my documentation reads “update bootresources.yaml appropriately” which is a bit like saying “use a goodly measure of spices”
[07:21] <bigjools> gmb: it's all in the subarchs
[07:22] <bigjools> so amd64/hwe-t for example
[07:22] <gmb> bigjools: Thanks.
[07:22] <bigjools> gmb: see http://maas.ubuntu.com/images/ephemeral-v2/daily/streams/v1/index.json
[07:22] <gmb> bigjools: Ah, perfect!
[07:23] <gmb> Excellent typo… “Hardware enoblement”
[07:24]  * gmb fixes
[07:40] <gmb> bigjools: Could you use tags to set HWE kernels?
[07:41] <bigjools> gmb: no, it's driven entirely from subarchj
[07:41]  * gmb is still murky about tag magic
[07:41] <bigjools> "tags are cheesy"
[07:41] <gmb> :)
[07:50] <gmb> bigjools: https://code.launchpad.net/~gmb/maas/hwe-docs/+merge/214476
[07:50] <gmb> Cheese-free, hopefully
[07:51] <bigjools> gmb: cheers, will be on it shortly
[07:56] <bigjools> gmb: there;s code at the bottom of the diff
[08:08] <gmb> bigjools: Ah, sorry. colo-branch-using fail.
[08:08] <gmb> Fixing.
[08:10] <gmb> bigjools: Don
[08:10] <gmb> e
[08:21] <gmb> bigjools: Thanks for the review. All good points. On it now.
[09:08] <bigjools> gmb, allenap: https://bugs.launchpad.net/maas/+bug/1302156
[09:17] <jtv1> bigjools: I'm just putting up a fix for bug 1302772.
[09:18] <bigjools> jtv: you're a star
[09:18] <bigjools> jtv: I shall leave one of gmb or allenap to review that, as I am EOD now.  Thank you!
[09:18] <jtv> np
[09:19] <jtv> And I see that gmb assigned me the card.
[09:19] <gmb> I’ll review it.
[09:19] <gmb> Seems fair
[09:19] <jtv> According to the timestamp, about 3 minutes before I ever mentioned it.  :)
[09:19] <gmb> :)
[09:19] <jtv> Thanks!
[09:20] <jtv> Did you  finish reviewing my other branch?
[09:20] <gmb> jtv: I’m on it now.
[09:20] <jtv> Ah OK
[09:20]  * gmb designates self reviewmeister for this morning.
[09:20] <jtv> Schnell!  Schnell!
[10:08] <strikov> jtv: hey
[10:10] <strikov> jtv: how about extending TestMain to (1) have/pull multiple labels (2) have/pull .gz image to make sure that unpacking works
[10:34] <jtv> strikov: Useful in themselves, but would complicate a test that is already quite overweight and potentially brittle...  It's always tempting to throw more complications into these tests, but also quite costly.
[10:34] <jtv> If we can unit-test these points, great.
[10:34] <jtv> (Today's a holiday for me, so not around much)
[10:36] <jtv> If we do the unpacking, then we can unit-test it.  If it's Simplestreams' job, then it's really something for the Simplestreams tests.
[10:44] <strikov> jtv: Okay, thanks, got your point. Enjoy your holiday :)
[10:44] <jtv> Thanks!
[12:30] <strikov> jtv: I started to work on tests related to image unpacking (we do it, not ss) but met with one serious issue: euc2tgz tool (we use it to convert .img to .tgz) requires root permissions
[13:41] <tych0> hi allenap, so it looks like i do want to trigger the celery job manually on the import process
[13:41] <tych0> i guess you guys must do this for tests and stuff, is there a way to do that?
[13:55] <jtv> strikov: ah yes, that tool... one thing you can try is fakeroot.
[13:55] <jtv> It's like sudo, but fake.  :-)
[13:55] <jtv> fakeroot runs a child process that thinks it's root, a lot like sudo, but it doesn't actually get any of the privileges.
[13:57] <jtv> tych0: you can just run a celery task (i.e. a function defined with the @task decorator) by calling its delay() method.
[13:57] <jtv> (Yes, the functions have methods :)
[14:09] <tych0> jtv: ok, so i just import it and do footask.delay()?
[14:09] <tych0> cool
[14:10] <jtv> tych0: Yes.  And don't forget, when you need to patch out a celery task for a test, you actually need to patch out its delay() method!
[14:10] <tych0> jtv: yep, we actually want to run this one this time :-)
[14:10] <jtv> Ah, that's easier.  :)
[14:11] <jtv> During tests, celery is wired up to execute tasks synchronously.
[14:11] <jtv> In fact we've been sort of doubting the wisdom of that choice — because some of it gets run implicitly during tests for no good reason.
[14:25] <webbrandon> any of you ever ran into these when mining: http://paste.ubuntu.com/7217201/ ?  I just started a priate network server and all my miners get that, trying to think of why?
[14:26] <jtv> Mining..?
[14:26] <webbrandon> oops
[14:26] <webbrandon> wromg chan
[14:26] <jtv> oic
[14:47] <jtv> gmb, rvba: I'm backporting my fix for the config bug (where the old "boot" section in pserv.yaml broke the upgrade).  No objections?
[14:47] <rvba> jtv: sounds good to me.
[14:51] <tych0> hi jtv, i'm getting http://paste.ubuntu.com/7217337/
[14:53] <jtv> tych0: new one on me... one of the guys in Europe may be able to help.
[14:53] <tych0> ok
[14:53] <tych0> rvba: ^^ ?
[14:53] <tych0> or allenap
[14:53] <jtv> Could be a celery change.
[14:54] <tych0> i think this is probably something you've always had to do
[14:54] <tych0> i vaguely remember rvba describing to me this before, at least
[14:54] <jtv> Hmm yes, it's not a change in celery — it's something we do.
[14:54] <rvba> tych0: yes, hang on…
[14:56] <rvba> tych0: Wrong celery config.
[14:56] <tych0> ok, how do i pass the right one?
[14:56] <rvba> tych0: you have two celery config in /etc/maas, have a look at the upstart script to see what variable you need to define to load the right config.
[14:58] <tych0> ok
[15:13] <tych0> hi rvba, sorry for being slow, but: http://paste.ubuntu.com/7217420/
[15:13] <tych0> i'm not sure what variables to export exactly, based on my /etc/init/maas-cluster-celery.conf
[16:03] <AskUbuntu> MAAS / nodes have limited internet access after install | http://askubuntu.com/q/444563
[16:18] <tych0> allenap: any thoughts on my question above?
[16:18] <allenap> tych0: otp, biab
[16:20] <tych0> allenap: sure, no worries
[16:24] <allenap> tych0: Are you setting CELERY_CONFIG_MODULE in the environment before invoking this?
[16:24] <allenap> It has a similar approach to Django’s settings module.
[16:26] <allenap> tych0: Are you trying to kick off the import job? That can be done via the remote API, but you don’t get any feedback, so I assume you want to run it directly?
[16:28] <tych0> allenap: yeah, viz. the paste above
[16:28] <tych0> i might be doing it wrong, though
[16:28] <tych0> allenap: oh, if we can do that via the api that is good enough
[16:28] <allenap> tych0: I was looking at the wrong paste :-/
[16:28] <tych0> allenap: we are going to run the import directly
[16:29] <tych0> and then run the celery job to do the import
[16:29] <tych0> and then check if it imported everything
[16:29] <tych0> so i guess it would be easiest if we could run it directly so it could fail for us
[16:29] <tych0> but not super important, since it probably (:-) won't fail
[16:29] <tych0> the import job itself is much more likely to fail
[16:31] <allenap> tych0: I don’t follow :) Do the import directly then run the celery job to do the import? Do you mean the celery job to report back to the region what’s been imported (report-boot-images or something like that)?
[16:32] <tych0> yep
[16:32] <tych0> the second one
[16:32] <tych0> ideally we'd run that celery job manually too
[16:32] <tych0> because we don't want to be stuck in a wait loop because it broke
[16:33] <allenap> tych0: Yeah :-/ Okay, I’ll see if I can figure out why that setting isn’t available.
[16:34] <tych0> allenap: ok, cool, thanks
[16:34] <tych0> i tried looking at celery
[16:34] <tych0> scary stuff :-)
[16:37] <allenap> tych0: Instead of /etc/maas in PYTHONPATH, use /usr/share/maas
[16:37] <allenap> Or, both.
[16:38] <tych0> allenap: bingo, thank you
[16:38] <tych0> now i get back an AsyncResult object, which isuppose mean things worked?
[16:39] <allenap> tych0: /etc/maas/maas_local_celeryconfig_cluster.py will probably only be readable by root:maas, so make sure you’re running as root or in the maas group; you won’t get any warning if it can’t read that module because of permissions.
[16:39] <allenap> tych0: Might have :) I don’t know what one of those is.
[16:39] <tych0> http://paste.ubuntu.com/7217779/
[16:41] <allenap> I don’t even know what a result backend is.
[16:41] <allenap> rvba: Are you around? ^
[16:41] <lazyPower> greetings maasers. I've run into an interesting issue with maas on trusty server that appeared after I rebooted the machine. http://i.imgur.com/QsOuhYZ.jpg
[16:42] <lazyPower> it has to do with iscsi target host, and the tgt bootstrapping, this error was hit before but skipped. Now that i've rebooted the maas region-controller (Which is also the cluster-controller) i now get halted on this during the pxe boot process
[16:42] <tych0> allenap: hmm, ok
[16:42] <tych0> it isn't showing up in the logs, so i don't think it is actually running
[16:45] <allenap> lazyPower: I suspect that’s https://bugs.launchpad.net/maas/+bug/1300548. That will be fixed in trusty soon, but if you look at the diff in that change you can see that all you need to do is create a missing symlink (and probably restart tgt).
[16:46] <allenap> tych0: Make sure you’ve got both /etc/maas *and* /usr/share/maas in PYTHONPATH.
[16:46] <allenap> I don’t think that’ll fix the bug you’re seeing, but it needs to be done anyway.
[16:48] <tych0> allenap: ok, yeah, didn't fix anything :-(
[16:50] <lazyPower> aha! i knew i found it but closed the tab, and was unable to refind it.
[16:50] <lazyPower> thanks allenap
[17:07] <allenap> tych0: I think this is happening because, when you call .delay(), it expects to be able to send that task off to a queue. In MAAS’s case, RabbitMQ.
[17:07] <allenap> tych0: You need to find the function to call that actually executes the body of the code.
[17:08] <allenap> run() maybe, but that doesn’t seem to do anything here.
[17:09] <tych0> hmm, ok
[17:09] <tych0> just fiddling with somethign else, will look at it in a sec
[17:10] <rvba> allenap: tych0: back with a decent internet connection…
[17:11] <allenap> tych0: You’re better off just calling maas-import-pxe-files!
[17:11] <tych0> allenap: that's what we're planning to do
[17:11] <tych0> but we need to also trigger the celery job to have it collect the images
[17:11] <allenap> That’s all the task in p.tasks.report_boot_images does: calls sudo -n -E maas-import-pxe-files.
[17:11] <rvba> tych0: right, you can't use r.failed(): MAAS doesn't store the result of the tasks it sends.
[17:11] <tych0> otherwise we can't create a node
[17:12] <tych0> allenap: yeah, that's what we were doing before
[17:12] <tych0> now maas does things via celery vs. a simple filesystem check
[17:12] <tych0> so we hav eto trigger the celery job too
[17:13] <allenap> tych0: Yeah, ignore about the last 120 seconds from me :)
[17:14] <allenap> Possibly more :)
[17:14] <tych0> ideally we'd be able to trigger the celery job manually as well
[17:14] <allenap> I was getting muddled between report_boot_images and import_boot_images.
[17:14] <tych0> so i ncase that fails we get a non-zero exit code and know to inform the user
[17:18] <allenap> rvba: So, how do we execute a task directly? .run?
[17:19] <rvba> allenap: task.apply_async()
[17:20] <allenap> tych0, rvba: I’ve just realised what’s going to cause an issue. The credentials aren’t in p.cache.cache (fwiw, p.cache.initialize() needs to be called early too).
[17:21] <allenap> rvba: Doesn’t apply_async() arrange for the task to be called later?
[17:21] <rvba> allenap: yes, that's what I thought you meant.  Oh, you meant call the task's code manually?
[17:22] <allenap> rvba: Yeah. I think .run() is the badger.
[17:23] <allenap> tych0: Getting the credentials into p.cache.cache will result in total hair loss.
[17:23] <allenap> report_boot_images() needs them to talk to the API.
[17:24] <allenap> tych0: Well… you might be able to fudge it.
[17:25] <allenap> tych0: Do you have a one:two:three credentials thing for the API to hand at the point you need to call report_boot_images()?
[17:25] <tych0> yep
[17:25] <tych0> so i can pass those in somehow
[17:27] <allenap> tych0: Here goes:
[17:28] <allenap> from provisioningserver import cache
[17:28] <allenap> cache.initialize()
[17:28] <allenap> from provisioningserver import auth
[17:28] <allenap> auth.record_api_credentials(“one:two:three”)
[17:28] <allenap> from provisioningserver import tasks
[17:28] <allenap> tasks.report_boot_images(http_proxy=…)
[17:28] <allenap> Watch out for the quotes above; my machine automatically changes them to typographical quotes.
[17:28] <tych0> yeah, saw that
[17:28] <tych0> r.e. the quotes
[17:29] <tych0> allenap: cool, will try now
[17:30] <allenap> tych0: Eugh, ignore the http_proxy bit. I am yet again getting muddles.
[17:30] <allenap> muddled too.
[17:34] <allenap> tych0: I have to go now, but I’ll be back in ~2 hours.
[17:34] <tych0> allenap: ok, it looks like this won't work exactly, but i think i can get it to work
[17:34] <tych0> need to source /etc/maas/celery.conf and some other stuff
[17:34] <tych0> but this is a lot further than i would have gotten myself
[17:35] <tych0> i will let you know what i come up with
[17:35] <allenap> tych0: Ah, yes! Source /etc/maas/maas_cluster.conf and export MAAS_URL and CLUSTER_UUID.
[17:35] <tych0> yep
[17:41] <tych0> arg
[17:41] <tych0> 403 forbidden
[17:41] <tych0> http://paste.ubuntu.com/7218034/
[17:45] <Fishy_> whats it mean when I can't import boot images on a new install, and I get  "  File "/usr/lib/python2.7/dist-packages/formencode/schema.py", line 145, in _to_python     value_dict, state) Invalid: The input field 'username' was not expected."
[17:50] <rvba> Fishy_: doesn't look good at all, could you please file a bug with the full stack trace, the version of MAAS you're using, the exact MAAS configuration, and what you've done to get that stacktrace?
[17:51] <Fishy_> okay
[17:51] <Fishy_> fresh install off CD
[17:59] <Fishy_> https://bugs.launchpad.net/maas/+bug/1303935
[18:01] <rvba> Fishy_: ta.  I can't have a look at it right now but we will get back to you quickly.
[18:03] <Fishy_> cool!
[18:03] <Fishy_> im sure its me
[18:54] <_bjorne> why im always get this answer in my access.log?!  200 is ok, 404 is error ----> GET /MAAS/metadata//2012-03-01/user-data HTTP/1.1" 404 200 "-" "Python-urllib/2.7
[19:23] <tych0> hi smoser, what can you tell me about the new import script and tftp timeouts?
[19:23] <tych0> it looks like my /var/lib/maas/tftp is totally empty, i assume it shouldn't be?
[19:24] <smoser> that woudl seem bad.
[19:25] <smoser> but it might be goign to ta different place.
[19:25] <tych0> smoser: http://paste.ubuntu.com/7218450/
[19:26] <smoser> tych0, that seems plausible
[19:27] <tych0> smoser: ok, so that might work (?)
[19:27] <tych0> i could bomb boot-resources and try again
[19:32] <smoser> tych0, id' avhe to look closer, but generally it seems sane.
[19:32] <smoser> i dont knwo about tftp timeouts though
[19:32] <smoser> you ahve more info there?
[19:33] <tych0> smoser: no, it pxe boots, gets an ip, and then never gets a kernel from tftp
[19:33] <tych0> smoser: is there some more info i can collect somewhere?
[19:36] <smoser> does it get the pxe config ?
[19:36] <smoser> this is intel ?
[19:37] <tych0> smoser: amd64, no, it doesn't get the pxe config
[19:38] <tych0> PXE-E32: TFTP open timeout
[19:38] <tych0> PXE-M0F: Exiting Intel Boot Agent.
[19:41] <AskUbuntu> MAAS Controller & KVM Help | http://askubuntu.com/q/444648
[19:42] <smoser> tych0, is the pserv running ?
[19:42] <tych0> well
[19:42] <tych0> i restarted it but i don't think it came up, actually
[19:42] <tych0> sec
[19:43] <tych0> heh...
[19:43] <tych0> http://paste.ubuntu.com/7218525/
[19:43] <tych0> upstart's maas-pserv.log
[19:48] <smoser> tych0, is it crashing on a tftp request?
[19:48] <smoser> or crashing just when run
[19:48] <tych0> smoser: got it figure dout actually
[19:48] <tych0> will file a bug shortly
[19:57] <tych0> smoser: should i still expect things in tgt-admin --show?
[19:57] <smoser> tych0, yes
[19:57] <tych0> now iscsistart is timing out
[19:57] <tych0> and there isn't anything there
[19:57] <tych0> (in tgt-admin --show
[19:57] <tych0> )
[19:58] <Jeffrey_> Good afternoon, I am currently using MaaS 1.5 Trusty Thar and after a fast install using curtin on a node the server doesn't reboot. I was wondering if this is the expected action? I was using a previous MaaS 1.4 Saucy and after a fast install the machine would reboot and then be ready.
[20:06] <smoser> console log is the most useful.
[20:06] <smoser> if it as trusty ephemeral image
[20:07] <smoser> then you will have /var/log/cloud-init-output.log
[20:07] <smoser> and /var/log/cloud-init.log , looking for WARN will probably shoul you something
[20:09] <Jeffrey_> smoser: Alright awesome thanks for your help! I'm going to go try some things now.
[20:36] <allenap> tych0: How did you get on in the end?
[20:36] <tych0> allenap: https://bugs.launchpad.net/maas/+bug/1302772 was what i was hitting
[20:36] <ubot5`> Ubuntu bug 1302772 in MAAS "update of maas-cluster-controller on trusty dumps traceback and crashes" [Critical,In progress]
[21:01] <allenap> tych0: Do you have a workaround for that?
[21:02] <tych0> allenap: yeah, just delete that whole section from the config file
[21:03] <allenap> tych0: We’ll have a fix for that in time for Trusty, so as long as you can bear it for now it should be fine.
[21:03] <tych0> allenap: yep, that worked
[21:03] <allenap> Tip top.
[21:20] <lazyPower> looks like we have some new fun stuff on the horizon with import-pxe-files, i got an invalid key signature error
[21:20] <lazyPower> http://paste.ubuntu.com/7218962/
[21:21] <lazyPower> however, re-running the command completed successfully. Maybe a temporary hiccup
[21:24] <tych0> lazyPower: those images probably aren't signed by the archive key
[21:25] <lazyPower> its strange that re-running the command completed without issue. I think it was just a temporary hiccup?
[21:25] <tych0> shoudl be signed by the key in ubuntu-cloudimage-keyring it hink
[21:25] <lazyPower> because if its failing crc validation and bailing on the whole process, it should have repeated the same behavior