[01:49] <kurt_> bigjools: not to sound silly, but I've not really worked with bz.  Is there a howto somewhere on how I would install this?
[01:50] <bigjools> bz?
[01:50] <kurt_> https://wiki.ubuntu.com/Bzr
[01:50] <kurt_> reading about it now...
[01:51] <bigjools> apt-get install bzr
[01:51] <bigjools> bzr checkout --lightweight lp:maas
[01:51] <bigjools> is all you need
[01:51] <kurt_> thnx, will try that...
[01:52] <bigjools> actually
[01:52] <kurt_> ?
[01:53] <bigjools> bzr checkout --lightweight lp:maas/1.0
[01:53] <bigjools> you don't want trunk
[01:53] <kurt_> eh, too late...
[01:53] <bigjools> heh
[01:53] <kurt_> can I stop in the middle?
[01:53] <bigjools> well it might work, try it out
[01:53] <kurt_> lol
[01:53] <bigjools> just don't run the dev version, it'll conflict with installed packages
[01:54] <bigjools> but the vdenv stuff should be ok
[01:54] <kurt_> it appears to have installed - is there an easy way of removing/reversing that install?
[02:04] <kurt_> ok, it didn't install, just create the folder branch
[02:18] <bigjools> it's just creating a dir for you, remove it if you don't want it
[02:58] <jtv> roaksoax, smoser: bigjools & I would like to know how the nodes obtain their ephemeral images.  Cobbler provides those over iscsi?
[03:00] <kurt_> but what's the trick to getting the components in place after you have the directory?
[03:01] <roaksoax> jtv: its all PXE
[03:02] <jtv> roaksoax: but that's not a transport mechanism — how do the nodes obtain the images?
[03:05] <roaksoax> jtv: yes iscsi. So cobbler ismply profives the kernel and initrd but the images are accessed via iscsi
[03:05] <roaksoax> jtv: howeveer, a cobbler profile is the one who has the information about where it is located
[03:05] <roaksoax> jtv: that's a kernel argument passed to the pxe file
[03:06] <roaksoax> jtv: you guys also need to make sure it that users can pass custom kernel parameters
[03:07] <jtv> One thing at a time though… we're in the process of replacing cobbler, so we need to know how we will provide the images to the nodes without cobbler.
[03:08] <roaksoax> jtv: cobbler doesn't really provide the images. We simply tell the kernel on the PXE file where to obtain the image from
[03:10] <bigjools> what is the option that tells it?
[03:12]  * jtv strides out for urgent coffee
[03:12]  * roaksoax expanding
[03:15] <bigjools> is it set in the preseed that it gets from the url specified on the cmd line?
[03:15] <roaksoax> give me a sec please
[03:15] <bigjools> ok :)
[03:25] <roaksoax> bigjools: http://pastebin.ubuntu.com/1041810/
[03:26] <roaksoax> jtv: http://pastebin.ubuntu.com/1041810/
[03:26] <roaksoax> jtv: i meant to write this as a response to your email
[03:26] <kurt_> bigjools:  from our earlier conversation - what gets the bzr components in to place?
[03:27] <bigjools> kurt_: what do you mean bzr components?
[03:27] <bigjools> by*
[03:27] <kurt_> I now have a "1.0" directory in my home directory
[03:27] <kurt_> it has a lot of sub directories
[03:27] <bigjools> inside there you'll see a vdenv/
[03:27] <kurt_> how do I know where to install things?
[03:28] <bigjools> look at it's README
[03:28] <bigjools> its
[03:28] <bigjools> don't necessarily run it blindly until you understand what it's doing
[03:28] <kurt_> I did - as I recall it gave the basic install process for maas. i'll look again
[03:28] <bigjools> kurt_: wrong readme
[03:28] <bigjools> look in *vdenv*
[03:29] <kurt_> kk
[03:29] <bigjools> you're not installing maas, just using still in the source to set up VMs
[03:29] <bigjools> s/still/stuff/
[03:30] <bigjools> roaksoax: can you explain more about iscsi?
[03:30] <roaksoax> bigjools: in short, maas-import-ephemerals sets up iscsi.
[03:31] <bigjools> is it a local daemon?
[03:31] <roaksoax> bigjools: yeah, it is tgtd running on the maas server
[03:31] <bigjools> ah
[03:31] <roaksoax> bigjools: so the only thing needed is that we are able to tell what kernel arguments to use in the PXE file
[03:32] <roaksoax> bigjools: are you guys using the concept of "profiles" "systems" or anything similar?
[03:32] <bigjools> no
[03:32] <bigjools> not yet anyway :)
[03:32] <roaksoax> bigjools: I could expand on how everything works for you guys to have a better view of what's happening with cobbler
[03:32] <bigjools> we're trying to understand the process and see what works best
[03:33] <bigjools> one of the nasty things about cobbler is that it needs to keep syncing its DB
[03:34] <bigjools> now, I'm trying to remember how juju selects a different profile
[03:35] <roaksoax> bigjools: it doesn't
[03:36] <bigjools> preseed, I mean, sorry
[03:36] <roaksoax> bigjools: it doesn't
[03:36] <bigjools> :)
[03:37] <roaksoax> bigjools: there's 1 preseed for enlistment. 1. for commissioning. 1. for deployments
[03:37] <bigjools> so all the deployment preseeds have the juju stuff at the bottom?
[03:39] <roaksoax> bigjools: no, there is one "snippet" that sources form a ks_meta variable (kickstart meta data variable). This snippet is maas_preseed and is sourced on the commissioning and the installation preseed
[03:39] <bigjools> ah right that was it
[03:40] <roaksoax> so the maas_preseed content is given by MAAS to cobbler in base64, and it's being decoded when the preseed is being obtained
[03:41] <bigjools> it all comes back to me now
[03:42] <roaksoax> bigjools: hheh
[03:42] <bigjools> thanks roaksoax
[03:42] <roaksoax> bigjools: anyways, I'll try to exapnd more on the whole process tomorrow.
[03:42] <roaksoax> i'm off to bed
[03:42] <roaksoax> have a good one
[03:42] <bigjools> roaksoax: sleep well - we'll trudge on
[04:33] <jtv1> bigjools: wip branch → https://code.launchpad.net/~jtv/maas/import-to-tftp/+merge/110451
[04:39] <bigjools> jtv1: ok - late lunch here, back in a bit
[04:39] <jtv1> ack
[05:57] <jtv1> Hmm where should I put tests for my shell scripts?
[06:09] <bigjools> jtv1: why bash and not python?
[06:09]  * bigjools just got back
[06:09] <jtv1> Turned out things stayed relatively simple after all.
[06:10] <jtv1> I figured for this, shell had relatively little overhead to get to a working script.
[06:11] <jtv1> Cyclomatic complexity is low.
[06:12] <bigjools> fairy muff
[06:34] <bigjools> jtv1: does cobbler cope with the race condition?
[06:34] <bigjools> don't think it does does it?
[06:34] <jtv1> Well it may just be restricted in its concurrency enough…  Really don't know.
[06:35] <bigjools> jtv: well, ISTM that it can't know if something is booting when the files are replaced
[06:35] <jtv> Maybe not — but that's not really needed anyway.
[06:36] <bigjools> mebbe
[06:36] <jtv> There's always tricks like adding a serial number to the directory name, so actual contents never change once created.
[06:36] <jtv> (And then you can delete anything that's not the latest and also old enough that nobody should be booting from it any more)
[06:38] <jtv> Basically MVCC much like postgres does.
[06:39] <bigjools> future work I guess
[06:41] <jtv> Indeed.  And we can do that in two phases:
[06:41] <jtv> 1. Fix the actual problem by storing each image in a separate directory.
[06:41] <jtv> 2. Offline GC wants disks actually start filling up.  :)
[06:41] <jtv> s/wants/once/
[06:48] <bigjools> jtv: so I think the script looks ok enough but as you say it'd be nice to know it works
[06:49] <jtv> bigjools: it's really deafeningly simple, because it does not deal with ephemeral images, nor with preseeds.
[06:49] <bigjools> famous last words
[06:50] <jtv> It downloads the pxelinux.0 for each architecture's latest release, and the linux & initrd.gz for each release, for each architecture.
[06:50] <jtv> And insofar as there are any changes, those are moved into the TFTP tree according to the filesystem layout I documented.
[06:55] <jtv> AFAICT the only reason why preseeds are mentioned at all in the existing script is that they need to be defined with the Cobbler profile.  We can bypass that completely because that information comes from the database.  Nothing to do with providing images.
[07:00] <jtv> Oh FIDDLESTICKS
[07:00] <jtv> Seriously, wget, SERIOUSLY?
[07:00] <jtv> You do all this stuff and then “file://” URLs are too hard for you?
[07:00] <bigjools> heh
[07:00] <jtv> NOT FUNNY
[07:02] <jtv> This undermines my test pretty thoroughly.
[07:03] <jtv> And why?  Why?
[07:03] <jtv> Don't answer that.  I know: because nobody ever tests this kind of code.
[07:05] <jtv> Grrr.  I need to rethink this.
[07:10] <bigjools> heh
[07:10] <bigjools> right.  Morning rvba!  And, good night everyone.
[07:11] <rvba> \o bigjools.
[07:11] <rvba> Hi jtv.
[07:12] <rvba> jtv: time for a tiny review? https://code.launchpad.net/~rvb/maas/bug-1013275/+merge/110475
[07:12] <ubot5`> Ubuntu bug 110475 in mozilla-thunderbird (Ubuntu) "“TB crash”" [High,Invalid]
[07:13] <jtv> Good morning rvba
[07:13] <jtv> Sure, I'll take it
[07:16] <jtv> rvba: done
[07:19] <rvba> ta
[07:25] <jtv> rvba: maybe you can give me some feedback on this idea.  In order to test my download script (replacement for maas-import-isos, basically) I want to pass wget a file:// url.  The test shouldn't download from the real Ubuntu servers, obviously.
[07:25] <jtv> But guess what: wget does not support file://!
[07:26] <jtv> I'm thinking, I could make it work if I insert a fake wget in the path that just links to curl.
[07:26] <jtv> Oh blast, no, I'm passing other options to wget.  :-(
[07:30] <jtv> Hmm...  Maybe I can work around that.
[07:31] <rvba> Or you can use the http fixture to create a temporary http server.
[07:33] <rvba> jtv: from maastesting.httpd import HTTPServerFixture
[07:33] <jtv> Seems a bit heavyweight…
[07:35] <jtv> There.  I've got an easy way to make it use curl instead.
[07:36] <jtv> By the way, I guess we should document our dependency on distro-info.
[07:36] <rvba> How will you manage to deal with the arguments you pass to wget?
[07:36] <rvba> When we will introduce the dependency, we definitely will.
[07:37] <jtv> We've been using distro-info for some time now.
[07:37] <rvba> Oh, right in scripts/maas-import-isos.
[07:37] <jtv> Yup.
[07:37] <jtv> I just added it to the hacking doc.
[07:38] <rvba> Cool.  I filled a bug about using it to list the supported distros inside MAAS, I thought you were talking about that.
[07:38] <jtv> No, just the script.
[07:39] <jtv> My own script uses it as well.
[07:39] <jtv> In particular, my script uses it to get the current Ubuntu release.
[07:39] <jtv> (In addition to the list of releases, of course).
[07:43] <rvba> By the way, this script also sets up the profiles in cobbler.  I suppose you're simply dropping that right?
[07:43] <jtv> Yes.
[07:43] <jtv> Well, actually dropping it will be a separate step, of course.
[07:43] <jtv> But I'm not re-creating it.
[07:43] <rvba> Sure, good.
[07:43] <jtv> The profile creation does one other thing that I don't do: store preseeds.  We do that from the database.
[07:44] <jtv> Not a job for a download script.
[07:44] <rvba> Right, Gavin and I are working on that.
[07:44] <rvba> We're not sure we will need to add the notion of 'profile' right now.  So we will create something simple to get started and then we will iterate on that.
[07:45] <jtv> Makes sense.  My guess is we'll need _something_ that's similar to a profile, but it's hard to say now where all the parts will come together.
[07:45] <rvba> Exactly.
[07:46] <rvba> We will start by generating preseed files "on the fly" given the proper arguments (node, release, type of preseed {commissioning, etc}).
[07:47] <jtv> On node status change, I guess?
[07:48] <jtv> (BTW bear in mind that we may not know which of its MAC addresses a node will use during pxe boot)
[07:48] <rvba> No, really "on the fly", so it will be generated when we will need it.
[07:51] <jtv> Doesn't that usually coincide with node status changes?
[07:52] <rvba> Well, we will add a way to actually see the preseed file in the UI.  This is not linked to the status being changed.
[07:52] <rvba> But obviously, we will expose that on the metadata API too.
[07:53] <jtv> You're probably aware, but just in case: you can now access the metadata for any given node, in dev/demo mode.
[07:55] <rvba> Yep, I've seen this.
[08:04] <Daviey> czajkowski: hola
[08:05] <czajkowski> Daviey: care to reply to maas mail with some suggested faq
[08:05] <czajkowski> :)
[08:05] <Daviey> czajkowski: nah
[08:05] <Daviey> but thanks for the offer.
[08:12] <jtv> You know?  This is what I'm here for.
[08:12] <jtv> The love.
[08:13] <czajkowski> jtv: he's a charmer !
[08:13] <jtv> czajkowski: whatever you do, don't fall for his smooth manners.
[08:14] <Daviey> :D
[08:14] <Daviey> czajkowski: i will. i'm pulling your chain.
[08:15] <jtv> You “yank” someone's chain, or “pull” their legs.  Pick one.
[08:15] <jtv> Because otherwise, it sounds suspiciously like you're trying to flush czajkowski down the toilet.
[08:17] <Daviey> jtv: aww, the key is to not specifically identify the meaning.
[08:17]  * jtv mumbles something about coding style
[08:17] <Daviey> lol
[08:18] <Daviey> jtv: Which successes did you have overnight?
[08:19] <jtv> That's a rather personal question.
[08:19] <czajkowski> jtv: hah
[08:19] <Daviey> jtv: Over MY night, your morning :)
[08:19] <jtv> But hey, I don't have Daviey's smooth manners do I?
[08:19] <Daviey> <-- smooth to the core.
[08:19] <jtv> Oh, THAT night.
[08:20] <jtv> Well, I have a working download script for the regular (install) initrd/kernels.
[08:20] <Daviey> \o/
[08:20] <jtv> I'm working up tests to run it through its paces; just got the first one passing.
[08:20] <jtv> So I seem to be on the right track as far as that is concerned.
[08:21] <Daviey> jtv: and it injects them where?
[08:21] <jtv> Injects what where?
[08:21] <Daviey> jtv: it downloads them.. but stores them where?
[08:22] <jtv> In /var/lib/tftpboot/ according to the documented directory structure.
[08:23] <jtv> Not the test, of course; that downloads from a local temporary directory and stores to a local temporary directory.
[08:24] <jtv> Daviey: https://code.launchpad.net/~jtv/maas/import-to-tftp/+merge/110451
[08:24] <jtv> Hope you'll find it easy to follow, as shell script goes.
[08:24] <Daviey> jtv: I thought it was going to be stored in something like /var/lib/maas/XXX/, and generate /var/lib/tftpboot/ structure as a separate step?
[08:24] <jtv> Arguably — but the intermediate step is in a temp directory.
[08:25] <Daviey> jtv: ok, looks neat :)
[08:26] <jtv> If downloading a kernel fails it won't leave you with an updated initrd.gz that doesn't match your remaining old kernel, if that's what you mean.
[08:26] <Daviey> jtv: side note, we need to make sure that we can support importing, without having internet access aswell.
[08:26] <jtv> The archive URL is configurable.
[08:27] <jtv> Also, download failures on pxelinux.0 are ignored.
[08:27] <Daviey> jtv: it's more a case of, i can see /var/lib/tftpboot getting inconsistent, so being able to regenerate based on what maas already knows is useful IMO
[08:27] <Daviey> but yes, this is a good first step in that direction
[08:27] <jtv> Well this is only for downloads.  Generating files is a whole other step.
[08:27] <Daviey> right
[08:27] <Daviey> sounds good :)
[08:28] <Daviey> i'll stop distracting you now.
[08:28] <Daviey> :)
[08:28] <jtv> :)
[08:28] <jtv> Got a call coming up anyway
[08:28] <Daviey> jtv: My intent is Tues and Thurs, others by need.  You can see too much of eachother, right.
[08:29] <jtv> Quite
[09:35] <jtv> By the way, Daviey, has anyone tried the new random-node metadata access yet to your knowledge?
[09:48] <Daviey> jtv: random node?
[09:48] <Daviey> Sorry, can you remind me what this is?
[09:50] <jtv> Daviey: where you get access to any node's metadata.
[09:50] <jtv> Perhaps I should say arbitrary.
[09:50] <jtv> But we accept the word in Random Access Memory…
[09:51] <Daviey> ahh
[09:52] <Daviey> i see.. i explored it myself.. but not used it in anger as yet
[09:52] <jtv> Well if the exploration turned up no problems, then I'll call that a success at least as far as it goes.
[09:52] <Daviey> jtv: \o/
[09:53] <Daviey> Always good to end the week with noted success.
[09:55] <jtv> Well I'm quite happy already that I've finally got a new download script up for review!
[09:57]  * jtv stretches legs
[10:10] <Daviey> \o/
[10:18] <rvba> Daviey: sure.
[10:19] <Daviey> rvba / allenap: Do you have a list of deps you still need from pypi?
[10:20] <Daviey> the correct fix should really be to package them up IMO
[10:20] <Daviey> Alternatively, our brothers in openstack-ci have a local pypi mirror for the deps they need.  That is another alternative.
[10:21] <rvba> Daviey: allenap will confirm that but for some reason, python bootstrap.py insists on downloading stuff.  stuff=zc.buildout.  Installing the package does not solve the problem.
[10:22] <Daviey> i *love* buildout.. it does so much for us. :)
[10:26] <rvba> Daviey: is it easy to spoof pypi.python.org to localhost?
[10:28] <allenap> rvba: We can point buildout/distribute wherever we want.
[10:28] <allenap> rvba, Daviey: I thought of just putting the packages in my public_html on people.c.c.
[10:28] <allenap> As an experiment.
[10:28] <rvba> allenap: the problem is the 'python bootstrap.py' run right?
[10:29] <allenap> rvba: The problem is everything :)
[10:29] <allenap> rvba: I /think/ we can address that, but we also need the deps for after that.
[10:30] <rvba> allenap: putting the deps in ~/.buildout should be enough for "after that".
[10:31] <allenap> rvba: Okay. We need to store those deps somewhere. What we use for Jenkins we should also use for ourselves, don't you think?
[10:33] <rvba> allenap: well, since all of us have internet access the current situation works fine for us.  If we need to hack something real quick to get Jenkins to work then it's fine by me.
[10:33] <allenap> rvba: Okay, deal. We figure out the zc.buildout thing, then hack the cache for Jenkins.
[10:33] <allenap> As in zc.buildout gets a proper fix, Jenkins gets duct tape?
[10:34] <rvba> allenap: WFM :)
[10:37] <rvba> allenap: I like Daviey's idea of spoofing pypi.python.org.  But I'm not sure what it means exactly.
[10:39] <allenap> rvba: Using lp:~allenap/maas/effing-download-cache, http_proxy=http://localhost:1234 make build --> http://pastebin.ubuntu.com/1042191/ (i.e. it tries to download zc.buildout, fails, but then builds *fine*, pos)
[10:43] <rvba> allenap: well, if matsubara-lunch can try that on the Jenkins box and it works, then \o/.
[10:44] <allenap> rvba: It's caused by the index.obtain(...) line in bootstrap.py.
[11:04] <matsubara> allenap, rvba: doesn't seem to work
[11:06] <matsubara> allenap, rvba https://pastebin.canonical.com/68213/
[11:08] <Daviey> rvba: I mean having a fake pypi mirror, that is bascially all
[11:08] <Daviey> rvba: see, this is what openstack does http://pypi.openstack.org/
[11:08] <Daviey> mirror of the tar packages they care for.
[11:15] <allenap> matsubara: lp:~allenap/maas/effing-download-cache now has a fix for the always-downloading-zc.buildout problem. Can you try populating ~/.buildout/cache/dist (I can get you content here if you don't have any somewhere), then try building this branch?
[11:17] <matsubara> allenap, https://pastebin.canonical.com/68214/ that's the content of the cache/dist dir
[11:18] <matsubara> allenap, still got the MissingDistribution
[11:29] <matsubara> ah the problem is that the cache in the jenkins machine is missing psycopg
[11:44] <allenap> matsubara: I'm going out now, but I'll be back later. I've added virtualenv support to effing-download-cache, which allows us to install buildout without using its ungood bootstrap.py script.
[11:49] <rvba> matsubara: allenap is the real specialist here but I'm back from lunch and ready to help you if I can.
[12:20] <matsubara> rvba, cool. I was uploading the files to the cache and now am re-running make build
[12:57] <matsubara> rvba, looks like make build worked this time when I updated the cache (without updating the branch with the latest changes from allenap)
[12:58] <rvba> matsubara: \o/.
[13:06] <allenap> matsubara: \o/
[13:07] <allenap> matsubara: Which changes, if any, did you get from my branch? I'll get them reviewed and landed.
[13:07] <matsubara> allenap, rvba: oh, sorry!!! I ran the make build in the wrong terminal and of course, locally, it just work! (/me is embarrassed)
[13:08]  * matsubara fetches brown paper bag
[13:08] <allenap> Haha :)
[13:08] <rvba> :)
[13:21] <matsubara> allenap, rvba: make build is now complaining it can't find lxml: MissingDistribution: Couldn't find a distribution for 'lxml==2.3.2'.
[13:21] <allenap> matsubara: Ah, that has to be installed on the system.
[13:21] <matsubara> but that's installed as a package. shouldn't it fallback to the system one rather than trying to download one or use from the cache?
[13:21] <matsubara> unless...
[13:22] <matsubara> ii  python-lxml    2.3-0.1build1  pythonic binding for the libxml2 and libxslt
[13:22] <matsubara> grrrr
[13:22] <matsubara> the system one is old!
[13:22] <matsubara> No LSB modules are available.
[13:22] <matsubara> Distributor ID: Ubuntu
[13:22] <matsubara> Description:    Ubuntu 11.10
[13:22] <matsubara> Release:        11.10
[13:22] <matsubara> Codename:       oneiric
[13:22] <matsubara> argh, jenkins machine is running oneiric
[13:22] <rvba> !
[13:22] <allenap> Oh dear god.
[13:22] <matsubara> I think we have a bigger issue now
[13:22] <allenap> Haha :)
[13:22] <rvba> Indeed :(
[13:23] <matsubara> ok, I'll have to talk to Larry about this. there are too many things running on that machine for me to just dist-upgrade. I'll let you know through the list. sorry about the confusion
[13:24] <allenap> matsubara: Good luck! :)
[13:24] <matsubara> thanks :-)
[13:24] <allenap> matsubara: A schroot would work.
[13:25] <matsubara> hmmm that's a good idea. Any good docs to set that up? Maybe I can setup a schroot and a exclusive node to run MAAS on without the need to update the whole OS of the Jenkins master
[13:26] <matsubara> ok, I'll sort it out with Larry as I'll definitely will need help. If I can't get done this today, I'll email the list an update
[13:26] <matsubara> and if you have a good doc about the schroot let me know
[13:31] <allenap> matsubara: https://dev.launchpad.net/Running/Schroot is a doc I followed one. The man page is really good too.
[13:31] <allenap> s/one/once/
[13:40] <matsubara> cool. thanks!
[13:46] <jtv> I'm off.  nn everyone!
[14:10] <allenap> rvba: Got time for a review? https://code.launchpad.net/~allenap/maas/effing-download-cache/+merge/110534
[14:10] <rvba> allenap: sure.
[14:12] <allenap> Thanks!
[15:10] <rvba> roaksoax: Hi Andres, I've got a question for you.  I'm working on the preseed snippets/template and I notice that the variables are indicated by a '$' sign (e.g. $http_port).  That's all right but now, in a snippet (maas_proxy to be precise), I've found a different syntax: "d-i     mirror/http/proxy string http://@@server@@:8000/".  Would you know why we have a different syntax here?
[15:11] <rvba> roaksoax: I suspect this means that this is interpolated at a later stage (i.e. not during the preseed generation).  Am I right?
[15:12] <roaksoax> rvba: yeah, it seemed to me as they first used @@server@@ but then changed to use $
[15:14] <rvba> roaksoax: Ah ok.
[15:14] <rvba> Thanks.
[15:15] <roaksoax> rvba: although, it also seems that @@server@@ is a global variable, while '$' is a local variable, per system, profile, distro etc
[15:19] <rvba> roaksoax: oh, I think it's only a funky cobbler extension to Cheetah templates.  I just spotted http://paste.ubuntu.com/1042515/ in cobbler/templar.py.
[15:28] <roaksoax> I see
[15:28] <roaksoax> yeah it is a funky thing indeed