[07:20] <rvba> bigjools: Do you really need to resort to raw SQL in get_hostname_ip_mapping?
[07:21] <bigjools> hoist. petard. etc
[07:21] <bigjools> rvba: yes.
[07:21] <bigjools> same as the other same-named func does
[07:23] <rvba> Ah, right, I think I see why now…
[07:24] <bigjools> me too
[07:24] <bigjools> debugging it made it clear, it's not the same dict
[07:24] <rvba> bigjools: I /think/ Django grew support for "Distinct ON".
[07:25] <bigjools> well, still not sure why it's lazy, but uh
[07:27] <bigjools> oh maybe it's a memory thing
[07:30] <bigjools> rvba: "I'd rather properly test the behavior"
[07:30] <rvba> bigjools: in this branch you've added StaticIPAddressManager.get_hostname_ip_mapping but it's not used anywhere yet… do you have a follow-up branch that makes use of it?
[07:30] <bigjools> testing behaviour: that's what I am avoiding
[07:30] <bigjools> it's slow
[07:31] <bigjools> yes I am using it locally but struggling to get tests to pass
[07:31] <rvba> Yeah, I meant end-to-end behavior.  For the reason I stated.  But I'm okay with mocking, it's just not as reliable (and that's why you missed the fact that the old get_hostname_ip_mapping wasn't doing the right thing).
[07:32] <bigjools> these tests are a nightmare
[07:32] <bigjools> they are running the same code twice because of signals
[07:33] <rvba> The ones in test_dns?
[07:33] <bigjools> what wasn't it doing?
[07:33] <bigjools> yes test_dns
[07:33] <bigjools> make_nodegroup etc etc causes signals
[07:34] <rvba> Your tests aren't exercising the code up to the point where get_hostname_ip_mapping is called.
[07:34] <bigjools> so the same dns code is run
[07:34] <bigjools> rvba: yes I know, that is the intention
[07:34] <rvba> Ah, okay, cool then.
[07:34] <bigjools> it's tested elsewhere
[07:34] <bigjools> well, ideally in these :)
[07:35] <bigjools> rvba: every time create_managed_nodegroup() is called in the test it fires a dns job through a signal
[07:35] <rvba> bigjools: yeah, but until we call 'self.patch(settings, 'DNS_CONNECT', True)' that doesn't trigger a real task.
[07:36] <rvba> bigjools: you'll see that 'self.patch(settings, 'DNS_CONNECT', True)' is called *after* all the setup is done.
[07:36] <bigjools> true, but, still feels like one of the main reasons tests are slow
[07:36] <rvba> It's possible.
[07:36] <bigjools> they have gone massively slow on me lately, something has changed
[07:37] <rvba> I didn't notice anything this dramatic on my side.
[07:37] <bigjools> and we have not put enough importance on the fact that fast tests are a necessity
[07:37] <rvba> I agree with that.
[07:39] <bigjools> hurray a passing test
[07:39]  * bigjools fixes others
[07:41] <bigjools> rvba: very glad you made create_nodegroup_with_lease()
[07:41] <bigjools> easy to change to create static ip instead and thusly fix many tests in one fell swoop
[07:42] <rvba> Encapsulation FTW :)
[07:43] <bigjools> wooo passing tests
[07:43] <bigjools> I'll push a new rev in a sec
[07:45] <bigjools> rvba: annoyingly there's a bi-dependency between node and dns now, so inline imports are needed  :(
[07:45] <bigjools> rvba: dns.py can't import from maasserver.models.staticipaddress
[07:45] <rvba> I see.
[07:46] <bigjools> rvba: anyway new rev pushed, can you have a look please?  I am going to eat, back in a little while
[07:47] <rvba> bigjools: looking at the code, I think you're right and the only reason why lazydict was introduced was to make use of __slots__ and thus save memory space in case the mapping is huge.
[07:47] <rvba> bigjools: sure, looking now.
[07:47] <bigjools> yeah
[07:47] <bigjools> bbiab
[08:27] <bigjools> rvba: "deviation from the original design"... not as I understood it!  We spoke yesterday and said that dynamic IPs do not need dns CNAME entries
[09:48] <jtv> gmb: I remember seeing a bunch of "6" suffixes on keywords in an IPv6 sample config.
[09:48] <gmb> jtv: Yeah, it really is pretty much that simple :)
[09:49] <bigjools> Ran 2647 tests in 1889.564s
[09:49] <gmb> (at least on my test network, on a small scale)
[09:49] <bigjools> urgh
[09:49] <jtv> Why, why, why does factory.make_node() perform two update queries on the node it just created, and then another query to load back the object!?
[09:50] <gmb> bigjools: The v6 stuff couldn’t have increased runtimes could it? I.e. if there’s lots of “if IP in network” where network is a /64; that kind of thing?
[09:50] <bigjools> doubt it
[09:50] <gmb> Mind you we’re not doing it in many places yet.
[09:50] <bigjools> exactly
[09:50] <bigjools> profile.  we need a profile
[09:57] <bigjools> rvba: the branch is all done now, if you want to look again.  All I did was fix a couple of failing tests and remove the dns update for new leases.
[09:58]  * jtv longs for a utilities package
[10:01] <jtv> gmb: did you land your make-a-random-ipv6-address factory method?
[10:02] <gmb> jtv: Yes. it’s factory.get_random_ipv6_address().
[10:02] <rvba> bigjools: LGTM
[10:05] <jtv> gmb: wha!?  Why not just make_ipv6_address()?
[10:05] <jtv> Maybe I should have said...
[10:05] <rvba> bigjools: are you witnessing a general slowdown or is it confined to a subset of the tests?
[10:05] <bigjools> rvba: general
[10:06] <bigjools> rvba: thanks, BTW
[10:17] <rvba> The branch I described this morning is up for review: https://code.launchpad.net/~rvb/maas/utopic-syslinux/+merge/224182
[11:04] <jtv> rvba: blason added a file to the list of boot loader files, but it's present in syslinux 4 and 6 alike.
[11:04] <jtv> poweroff.com, I think
[11:04] <rvba> Yeah.  I think we can live with a bit of duplication here.
[11:32] <AskUbuntu> Unable to add nodes to ubuntu MAAS server on static vlan | http://askubuntu.com/q/487549
[13:00] <rvba> bigjools: gmb:  turns out we *can* use omshell to query the leases.  I just tested it and it works.
[13:00] <gmb> rvba: Sweet.
[13:10] <blake_r> jtv: "maasutils" package would be very nice
[13:10] <blake_r> jtv: "ensure_dir" is not in maasserver, ugh!
[13:12] <jtv> blake_r: yeah, so far we've mostly had to dump generic helpers in provisioningserver, because provisioningserver can't import from maasserver (but vice versa works).
[13:13] <blake_r> jtv: oh yeah forgot, was thinking it was the other way around
[13:14] <jtv> There shouldn't be a need to remember...  "maasutils"
[13:20] <rvba> blake_r: btw, FYI using a symlink /syslinux/ at the root of the TFTP dir worked.
[13:21] <blake_r> rvba: i was looking at that
[13:21] <blake_r> rvba: so you created a symlink from current/syslinux -> /usr/lib/syslinux ?
[13:21] <rvba> blake_r: exactly
[13:22] <rvba> blake_r: but syslinux 4 and 6 seem to have different requirements for what *really* needs to be stored at the top level.
[13:22] <blake_r> rvba: i thought pxelinux expected it all to be in tftproot
[13:22] <blake_r> rvba: does it check both?
[13:22] <rvba> blake_r: syslinux 6 definitely does
[13:23] <blake_r> rvba: cool
[13:23] <blake_r> rvba: that makes it easy
[13:23] <rvba> Yeah
[14:14] <rvba> newell_: Hi there.  I see you haven't had time to work on the MP I marked as "Needs fixing". I realize what I asked is probably a bit more work than you had anticipated because it implies dealing with forms.  But in the long run, it's a worthy investment (for the various reasons I stated in my review);  I'm happy to give you a hand with that if you want.
[14:35] <jtv> rvba: lander's tests broke on your branch.  :(
[14:35] <jtv> The syslinux one.
[14:36] <rvba> jtv: I saw that.  The failure looks completely unrelated.
[14:36] <jtv> Yeah.  And I remember that was a function I disliked for some reason.
[14:36] <jtv> I wonder if it's a function I messed with.
[14:37] <rvba> jtv: it's a spurious failure.  It passes locally from time to time.
[14:38] <jtv> You mean it _fails_ locally from time to time, I hope?  It's been passing for me.
[14:39] <rvba> Not sure, I haven't run it a great number of times.
[14:40] <rvba> But yeah, it seems success is more frequent than failure.
[14:40] <jtv> I bet there's an undetected clash in networks or ranges.
[14:40] <rvba> I'm going to investigate but I'd like to get my syslinux branch landed first.
[14:44] <newell_> rvba, I haven't worked on that because I was told that there were some other items that were higher priority, so I have been working on those.
[14:44] <newell_> Yes, your help would be appreciated
[14:45] <newell_> Once I have finished the higher priority items that I am currently working on, I will ping you so you we can go over it.
[14:45] <newell_> Thanks for lending a helping hand.
[14:48] <rvba> newell_: okay, cool.
[15:03] <d_`> where in the source is the startup script for bootstrapping a node?
[15:07] <d_`> based on packet captures on both hosts, they're making a bunch of DNS queries that are failing, but based on the hosts they're querying, I'm guessing some of them are supposed to fail?
[15:39] <jtv> Who's up for a code review?  https://code.launchpad.net/~jtv/maas/arp-on-failed-entries/+merge/224322
[15:42] <roaksoax> blake_r: ^^
[15:42] <blake_r> roaksoax: sure
[15:43] <roaksoax> blake_r: thanks!
[15:52] <blake_r> jtv: done!
[15:52] <jtv> Thanks blake_r!
[15:52] <blake_r> jtv: np
[15:53] <jtv> You say that, but actually it hasn't even been proven that np isn't the same thing as p.
[15:54] <blake_r> Yeah that makes since! But I will keep saying "np"!
[15:57] <rvba> blake_r: rarg, poweroff.com is poweroff.c32 in Utopic's syslinux.
[15:57] <blake_r> roaksoax: it should have both
[15:58] <blake_r> rvba: ^
[15:58] <blake_r> rvba: poweroff.com at /usr/lib/syslinux
[15:58] <blake_r> rvba: from syslinux-common package
[15:58] <rvba> That's not the case.  I just installed syslinux-common and syslinux-dev.
[15:58] <blake_r> rvba: ugh
[15:59] <blake_r> rvba: http://packages.ubuntu.com/search?searchon=contents&keywords=poweroff.com&mode=exactfilename&suite=utopic&arch=any
[16:00] <rvba> blake_r: http://paste.ubuntu.com/7695727/
[16:01] <blake_r> rvba: that is so wierd, guess the packages site is incorrect
[16:01] <blake_r> rvba: guess need a symlink from poweroff.com -> poweroff.c32
[16:01] <rvba> blake_r: indeed. Maybe roaksoax will know how that could happen.
[16:01] <rvba> blake_r: ewww
[16:01] <blake_r> rvba: temp[late needs to work fro both trusty and utopic
[16:02] <rvba> blake_r: I know.
[16:02] <jtv> And precise?  jtv interjects with a stupid look
[16:02] <rvba> But I don't want to do a symlink if we can avoid it.
[16:03] <rvba> blake_r: I wonder if, with the other symlink I just added it won't just work out of the box if we remove poweroff.com from the list of things we copy over.
[16:04] <rvba> blake_r: I guess it's worth a shot.  I'll test that tomorrow.
[16:08] <rvba> blake_r: are, I didn't realize the template was referencing the file directly…
[16:08] <rvba> s/are/arg/
[16:10] <loki27_>  I have question about charms deployment and juju .. If i deploy the same charm , on different machine, (Let's say mysql and rabbitMQ) , how will the deployment work their relation, will the mysql instances be standalone , or are they going to replicate the same data trough all instances ?
[16:10] <rvba> blake_r: isn't there a way to say: "poweroff.com or poweroff.c32"?
[16:11] <jtv> loki27_: you may have more luck in #juju with questions specific to a Juju charm.
[16:12] <loki27_> haa great
[16:12] <jtv> (Not saying nobody will know, but I think it's the best chance)
[16:13] <jtv> gmb: looks like you put code in make_interface_settings to leave static_ip_range_{low,high} set to None if the random range has no room for them.
[16:13] <jtv> Guess what happens when you enter None in the dict that goes into a POST?
[16:13] <gmb> jtv: Not exactly; it was to allow for the old (dynamic only) behaviour to be maintained… I’m guessing that that doesn’t work properly :/
[16:14] <jtv> Right.  It blows up, but only on rare runs.
[16:14]  * jtv doesn't really know what gmb just said, but finds enough common ground to bluff his way through
[16:15] <jtv> These are the things that need documenting in code!
[16:15] <gmb> I thought I had; my apologies :/
[16:15] <gmb> jtv: Can you paste me a failure?
[16:15] <jtv> Urgh.  I think I just lost that shell.  Let me reproduce it...
[16:16] <blake_r> rvba: I don't think so
[16:16] <blake_r> rvba: I will look and see
[16:17] <rvba> Okay, ta.
[17:08] <blake_r> rvba: from what I can find, doesn't seem to be anything about handling a missing file or alternatives
[18:06] <lazypower> strange. When i remote into my VMAAS nodes and issue df -h i dont see a root parition listed
[18:06] <lazypower> is this expected behavior?
[18:31] <d_`> lazypower: you don't see a root partition? how is your system operating?
[18:33] <lazypower> d_`: my sentiments exactly
[18:34] <lazypower> let me rebootstrap maas, spin up a machine with juju and pastebin the output for df -h
[18:58] <lazypower> http://paste.ubuntu.com/7696549/
[18:58] <lazypower> d_`: sorry for the delay, but thats what i see out of df -h on my maas nodes
[18:59] <lazypower> they all have 8gb disks allocated. This is a new one on me where df doesn't report all mounted filesystems.
[19:05] <blake_r> lazypower: whats the content of /etc/fstab
[19:18] <lazypower> blake_r: LABEL=cloudimg-rootfs / ext4 defaults 0 0
[19:18] <lazypower> that's the only line in there - so is this a byproduct of using some kind of overlayfs?
[19:28] <blake_r> lazypower: that is correct
[19:28] <blake_r> lazypower: i mean the corrent fstab
[19:29] <lazypower> ok
[19:29] <blake_r> lazypower: what does blkid, show
[19:29] <lazypower> this appears to have changed recently - within the last 4 months. I recall a while back df-h showed the contents of / mounted and its disk space
[19:29] <lazypower> blkid shows nothing, returns 0
[19:29] <lazypower> do i need to pass it an arg?
[19:29] <blake_r> no
[19:30] <blake_r> try "ls /dev | grep sd"
[19:30] <lazypower> nothin
[19:31] <blake_r> try
[19:31] <blake_r> try "ls /dev | grep vd"
[19:31] <lazypower> that yields 2 disks. vda device and vda1
[19:31] <blake_r> so your in a physical machine?
[19:31] <lazypower> well, disk / partition
[19:31] <blake_r> i mean virtual machine
[19:31] <lazypower> i'm in a virtual machine. correct
[19:31] <lazypower> i'm orchestrating VMAAS. my HOST (metal) is the region/cluster controller
[19:32] <lazypower> i'm using kvm w/ about 13 nodes pre-allocated as my maas cluster.
[19:32] <blake_r> so vda1 is your "/"
[19:32] <blake_r> fdisk /dev/vda
[19:32] <blake_r> print
[19:32] <blake_r> should show partition, correct?
[19:32] <lazypower> correct
[19:32] <blake_r> what version of maas?
[19:33] <blake_r> apt-cache policy maas
[19:33] <lazypower>   Installed: 1.5.1+bzr2269-0ubuntu0.1
[19:34] <blake_r> can you view the curtin log, as the node is installing ubuntu?
[19:34] <lazypower> i'm not sure whre that lives - i normally open up the VM from VMM and watch the logs there.
[19:34] <blake_r> okay, do this
[19:35] <blake_r> on the webui of maas, go to the settings page and set the global kernel parameters to this value
[19:35] <blake_r> console=ttyS0
[19:35] <blake_r> then "start" a node
[19:35] <blake_r> then on your host do "virsh console {name-of-vm}"
[19:35] <blake_r> that will show the booting log and curtin output
[19:36] <lazypower> right on, neat
[19:36] <blake_r> you should do "virsh console", quickly
[19:36] <lazypower> yeah i'm not seeing anything. should i see the TFTP output?
[19:37] <lazypower> er, the pxe boot sequence
[19:41] <lazypower> blake_r: ah, found the problem. I've got the curtin log now.
[19:41] <blake_r> lazypower: perfect
[19:47] <lazypower> blake_r: am i looking for something specific in here or do i need to pastebin this for you?
[19:48] <blake_r> lazypower: oh thought you saw the problem
[19:48] <blake_r> lazypower: yeah you can just pastebin the whole thing
[19:48] <lazypower> [    2.565478] EXT4-fs (sda1): mounted filesystem with ordered data mode. Opts: (null)
[19:48] <lazypower> thats teh only thing i see pertinent to what i'm looking for...
[19:49] <lazypower> ok let me re run this with output redirected to a logfile
[19:54] <lazypower> blake_r: http://paste.ubuntu.com/7696786/
[19:55] <lazypower> wait, i see this here. init bottom says something about tmpfs as the overlayfs
[19:55] <lazypower> wat
[19:56] <blake_r> thats just for install
[19:57] <blake_r> your system reboots
[19:57] <lazypower> ah ok
[19:57] <blake_r> does this new node have the same problem?
[20:01] <lazypower> blake_r: yeah :(
[20:02] <blake_r> lazypower: did you change the preseed in anyway
[20:03] <lazypower> blake_r: nosir
[20:03] <blake_r> lazypower: that is so wierd
[20:03] <blake_r> lazypower: i would recommend re importing the boot images
[20:03] <blake_r> lazypower: rm -rf /var/lib/maas/boot-resources/current/*
[20:03] <blake_r> lazypower: then "maas-import-pxe-files"
[20:04] <blake_r> lazypower: just to see if that is it
[20:04] <lazypower> already on it. i wiped the entire directory though
[20:04] <lazypower> /var/lib/maas/boot-resources
[20:04] <blake_r> that should be okay
[20:47] <lazypower> blake_r: yeah still seeing the same behavior after reimporting the pxe images
[20:48] <blake_r> lazypower: you are deploying precise, can you try to deploy trusty
[20:48] <lazypower> strange... my default series is set to trusty
[20:49] <lazypower> ah wait, i'm wrong.
[20:53] <lazypower> blake_r: mind = blown
[20:53] <lazypower> works as expected with trusty
[20:53] <blake_r> lazypower: strange precise should work
[20:53] <blake_r> lazypower: alteast it is working now
[20:54] <lazypower> i mean, it appears to work. the thing that got me was being unable to report disk usage
[20:54] <lazypower> i'm deploying a mongodb cluster, and ran out of disk space thanks to preallloc - which is what prompted this discovery session
[20:56] <blake_r> might ask in #ubuntu-server they might know an answer for precise
[21:00] <lazypower> thanks for the troubleshooting tutorial blake_r. this was informative
[21:00] <blake_r> lazypower: np
[21:01] <lazypower> i thought i had a good grasp on maas until this :P
[21:01] <lazypower> now i realize there's so much more