=== bigjools-afk is now known as bigjools [04:17] jtv: why would maasserver tests do this? [04:17] ImportError: cannot import name oauth [04:17] I can't reproduce in bin/py [04:17] Isn't that the python-django-piston problem? [04:17] Where _on our dev systems_ you got this error if you installed python-django-piston, but on Q/A systems you didn't? [04:17] grar [04:18] I'll remove the egg and see [04:18] We ended up with the hypothesis that it was caused by mixing buildout and packages. [04:18] yeah [04:18] buildout's copy needs to be junked,we can't use it at all [04:18] Still a load of test errors? Damn. [04:24] down to 11 failures, 6 errors [04:24] from testresources import FixtureResource [04:24] ImportError: cannot import name FixtureResource [04:24] hmm [04:27] Latest version on pypy is from January; still has FixtureResource in __init__.py as before. [04:28] *pypi [04:28] python-testresources 0.2.4-1ubuntu1 doesn't have it [04:28] Uhm [04:28] wtf [04:28] so check what that ubuntu1 upload is [04:28] someone may have uploaded without sending upstream [04:29] trunk is : [04:29] revno: 60 [04:29] tags: 0.2.5 [04:29] branch nick: trunk [04:29] timestamp: Fri 2012-01-27 17:59:50 +1300 [04:29] yeah it should be in 0.2.5 [04:29] I have the egg with it in [04:29] why is my buildout not picking it up [04:29] oh god [04:29] 0.2.5 is on pypi [04:29] it's not packaged, that's all [04:30] but I thought buildout would pick the egg over the installed package based on version [04:30] seems not :( [04:30] nope [04:30] it trusts you have it all, if you allow it to use system packages at all [04:31] something in my quantal upgrade must have pulled it in [04:36] Down to: FAILED (SKIP=1, errors=7, failures=2) [04:37] bigjools: need anything reviewed for that? [04:38] no, removing the package fixed itg [04:38] Well, any of the fixes you came up with? [04:38] jtv: can you look at the top error here [04:38] http://paste.ubuntu.com/1218063/ [04:38] yeah, starting on fixes now :) [04:38] See? I've been waiting for a chance to get in on this :) [04:39] * bigjools grabs a sandwich, back in 10 [04:50] That top error is a Django-specified method in our XMLField getting hit by a signature change. [04:51] Looks fixable. [04:57] yeah [04:57] also means blue squad is not on quantal :) [04:59] Review? https://code.launchpad.net/~jtv/maas/quantal-get_db_prep_lookup/+merge/125632 [04:59] I'm trying to test it on a quantal canonistack instance, but having some trouble getting set up. [04:59] ok [05:00] Did you get buildout complaining that it can't find lxml 2.3.2? I'm trying 2.3.5 instead. [05:00] nope [05:00] (My fix does pass tests on precise) [05:01] Could you be unwittingly benefiting from caching or something? [05:01] Like lxml 2.3.2 not being available for quantal, but buildout using your existing egg oblivious to the upgrade? [05:06] running tests with your branch [05:06] I have 2.3.2 in cache [05:06] I'm trying to run just 'make test', but it is failing because testscenarios has no attribute WithScenarios. [05:06] morning jam [05:06] hi bigjools [05:06] jam: hehe - see my email to maas-devel [05:06] this seems to only be in 'maastesting' vs 'maasserver' so I can skip it [05:07] are you on quantal? [05:07] bigjools: this is on precise, though, not Q [05:07] ok [05:07] do you have python-testresources installed? [05:07] if so remove it [05:08] jtv: approved [05:08] thx [05:08] passes tests now [05:08] well, that one does [05:08] Damn, there goes my momentary elation [05:09] jtv: although, it seems to have made the NoRabbit errors go away too [05:09] no idea how! [05:09] Urr [05:09] 2 failures left [05:09] Well don't complain. :) [05:09] both apidoc [05:09] Yeah [05:10] And suspiciously dependency-no-longer-works-the-way-we're-used-to-like. [05:10] whut? [05:11] bigjools: 'make' installs testresources [05:11] so it is listed as a dependency [05:11] ha [05:11] Oh joy my canonistack instance is full [05:12] I don't have it from 'apt-get install', though. [05:12] so I should do 'apt-get install' and try to match that version to the one for buildout? [05:12] jam: oh that is fine [05:12] no, you need the buildout one [05:12] the one we need is not packaged [05:13] k [05:14] jam: so I have no idea why you get that error then, it works fine here [05:14] you could see which actual file it's importing to make sure it's the one you expect [05:17] bigjools: the isolation doesn't seem to be working properly. [05:17] it is loading the system testscenarios (which was later than the buildbot one) [05:17] I'm surprised it loads from system before loading from buildbot [05:17] note that I need testscenarios for other code [05:17] apt-get remove python-testresources [05:17] so I can uninstall it for now [05:18] bigjools: apt-get remove python-testscenarios [05:18] not resources [05:18] however, [05:18] oh [05:18] I will need it to do u1db tests [05:18] we set up buildout to prefer system packages [05:18] so if bin/test would load from buildout cache before system, my life would be better [05:18] bigjools: isn't the whole point that buildout should override system? [05:19] it does, if you tell it to [05:19] (use system if it matches, but use the custom defined one if present) [05:19] I mean, it doesn;t compare versions [05:19] so if you tell it in buildout.cfg you want that package, then it'll use it [05:19] unless it's already installed [05:20] I think it was done like this to reduce the dependency on pypi, otherwise it downloads a lot of stuff [05:20] bigjools: I have testscenarios 0.3 installed in ~/.buildout/eggs [05:20] I also have 0.2? installed in /usr/lib [05:20] but when I run 'bin/test' it loads the system one [05:20] not the buildout one [05:20] yes that's expected [05:21] bigjools: I'm pretty sure the 'reduce dependency on pypi' was that it wouldn't download packages whose versions you already ha [05:21] had. [05:21] sadly buildout does not compare versions :( [05:21] bigjools: it did for me with ipython-0.12 vs ipython-0.12.1 [05:22] when versions.cfg said 0.12 and I had 0.12.1 installed globally [05:22] it would try to download until I updated [05:22] the versions.cfg to point to 0.12.1 [05:22] now, possibly at *runtime* it doesn't compare versions [05:22] but it does seem to at 'make' time [05:22] jam: it didn't download because you had a system one installed [05:22] it ignores versions completely [05:22] bigjools: it *did* download until I made the version in versions.cfg match the system one [05:23] I'm quite sure about that [05:23] oh? lifeless ^ ? [05:27] ? [05:27] bigjools: whats up ? [05:27] lifeless: version thing you mentioned earlier [05:28] bigjools: if there is a pth file or egg metadata [05:28] bigjools: if there isn't, I don't think it can tell, and that will depend on the package [05:29] so I think what you want is for the 'build' step to say 'if you have the version in sys.path, use it, else download to the egg cach', and then the 'run' step to say 'if it is in the egg cache, use it, else use the system one' [05:30] just found out today that there is a 'great firewall of UAE': http://www.du.ae/Documents/Annex%201-IAM%20Regulatory%20Policy%20ver%201%200%2029July2008.pdf [05:30] I was a bit surprised that online dating is one of the blocked sites. [05:30] "social networking" is listed, but isn't that facebook/twitter/g+ ? [05:30] and those certainly aren't blocked [05:30] Yeah, blocked in many countries though. [05:31] jtv: yeah fb was blocked in syria for a while, might even be now with the fighting [05:31] Usually all kids and subversives know how to get around it, but “decent” people get kept out of the 21st century. [05:31] jtv: well the first blocked thing is places that let you get around blocking [05:31] but there are lots of ways to do that :) [05:32] but VOIP is also prohibited [05:32] I guess because phone calls are expensive here, and they want to keep their monopoly [05:32] (something like $0.40/min to call internationally) [05:32] Here too, as are VPNs. We've had Google's appserver blocked, dropbox, twitter, youtube... The consequences of blocking aren't usually taken into account because the people who do this are rarely fully aware of what they're damaging. [05:33] Didn't the UAE also have Etisalad with its infamous *.com SSL certificates? [05:34] jtv: the main carrier here is Etisalat (has been for a long time) I hadn't heard about *.com though [05:34] that's pretty worrying [05:34] my *specific* carrier is the only competitor Du, but that is just because of whatever region I'm in [05:34] Yup. Issued for the purpose of censorship. Trusted by all clients, at least until a recent surge of awareness. [05:35] My understanding is that typically these countries regulate their incumbent carriers' censorship policies onto other carriers as well. [05:35] jtv: the ultimate MitM attack, carried out by your provider, and government approved. [05:35] CA-approved even. [05:35] which is why when the government does it, it is so much scarier than when j-random-citizen tries [05:36] quantal is doing my head in, no sound at all today [05:36] Time to start thinking about dual-boot setups. [05:37] So that you can track which of your hardware is really, natively supported in the upgrade before you commit fully. [05:38] I got a message saying that I may not have a usable desktop after the upgrade. [05:39] jtv: there is some way to get raw content out of django, though, as I think we're trying to do that for the xmlfield [05:39] (for the file storage discussion) [05:40] You mean you support a choice of encodings in the xml, without re-encoding for storage purposes? [05:40] jtv: https://docs.djangoproject.com/en/dev/howto/custom-model-fields/https://docs.djangoproject.com/en/dev/howto/custom-model-fields/ [05:40] jtv: actually, we are just trying to make it so we can run custom xpath_* queries [05:40] which isn't directly supported by django [05:40] but it sounds like something that would let you do bytea as well. [05:42] jtv: what did you think to my packaging suggestion? [05:42] It may have been something to do with the dynamicism of field conversions in django. [05:42] bigjools: saw the thread, but didn't look too closely. Which was your suggestion? [05:42] jtv: you need to read it [05:42] it's a little involved [05:42] OK, I'll dig it up [05:44] jam: when you write field conversions in django, you get very little information about what the intended conversion is. Basically you get “a value,” and it's up to you to figure out whether that needs any conversion. Sometimes the conversion seems to go in unexpected directions, too. [05:47] bigjools: your suggestion sounds good to me — but then I never got around to understanding the distribution of responsibilities between maas-dns and maas-dhcp. Until recently I thought maas-dhcp was de facto the seedling of the cluster-controller package. [05:48] (Small note: the provisioning server may require maas-cli, since it uses apiclient) [05:48] gah [05:50] Meanwhile, my experiments on quantal suggest that we need: https://code.launchpad.net/~jtv/maas/quantal-updates/+merge/125634 [05:51] jtv: why do we need bzr? [05:51] Ohhhh — that's the one Gavin added for now-unclear reasons, innit [05:51] commandant [05:51] I think it can go, I asked on the list earlier [05:52] In any case, 2.5.1 will break the build and 2.6.0b2 doesn't. [05:52] 2.5.1 works here [05:52] Cached? [05:52] well - it gets downloaded [05:52] hmmm [05:52] On a fresh canonistack instance it said no distribution found. [05:53] nfi then [05:53] Ahh, I think that was before I ran an upgrade on it. [05:53] also, why are we specifying precise versions [05:53] why not > [05:54] Didn't know you could. [05:54] or >= [05:54] I guess ideally at this stage, we'd be tracking Quantal's versions. [05:54] I'd rather not be specific [05:54] we have to QA it either way [05:55] Doesn't that carry a risk of accidentally relying on changes that are newer than what quantal has/ [05:55] ? [05:56] no, because we have to QA it [05:56] we should only get specific when a particular problem is found [05:57] otherwise we'll forever be amending versions in buildout.cfg [05:58] versions.cfg even [05:58] So… want me to slap in a bunch of “>=” then? [05:59] we need to review it, but don't do it now. [05:59] OK [05:59] your point above would equally apply to relying on versions that are less than quantals [05:59] I do find I need a newer lxml though. [06:00] Yes, absolutely. [06:00] ok [06:01] What was causing that piston error about _is_string again? You had a fix for that, right? [06:01] yeah remove your local piston egg and cache [06:01] we need to remove it from buildout and put it in required-deps [06:02] Weird. This is on a clean system. [06:02] jtv: I'd be interested in voice chatting with you about: https://code.launchpad.net/~jtv/maas/node-nodegroup-api-ui/+merge/125624 [06:02] the only use of bzrlib that I can find is src/maascli/__init__.py:from bzrlib import osutils [06:02] just to get an understanding of that system, and review your code while I'm there. [06:02] jam: OK — but I'll need a few moments to prepare. [06:07] allenap: you are using osutils.get_unicode_argv(), but it seems to be the only use of bzrlib [06:08] Is there a plan to use more? Or can we just split out that code rather than bringing in all of bzr as a dependency? [06:08] yes I was wondering the same [06:08] right, so it was certainly useful for commandant [06:08] but now that we aren't doing that [06:08] the dependency is quite small [06:08] allenap won't be around for about 2 hours though [06:11] jam: ready now — name your medium [06:11] jtv: whatever is your favorite :) [06:12] Face to face over coffee!? [06:12] jtv: that would be best, I agree [06:12] Deal. [06:12] Until you get here, I'll set up a hangout. [06:13] jam: I've invited you. [06:14] bigjools: removing my piston egg didn't help, and I don't think I have a cache. Wonder what else could be going on. [06:14] jtv: clean and rebuild [06:15] well just make check will do that [06:15] this definitely fixed things for me earlier [06:36] Not for me. :( [06:40] jtv: have you installed the quantal python-django-piston? [06:40] it has the fix in it [06:40] for sure [06:40] I don't even see it in my available packages. [06:41] you need 0.2.3-1ubuntu2 [06:42] Guess I need some repository that isn't configured. [06:42] like quantal? :) [06:43] it's right there https://launchpad.net/ubuntu/+source/python-django-piston/0.2.3-1ubuntu2 [06:44] This is a quantal instance. [06:44] apt-get update [06:44] I wonder why it doesn't pick up the existence of that package. [06:44] Yeah, been doing that. [06:44] how are you trying to install? [06:44] dselect [06:44] ?! [06:44] Nothing there with “piston” in the name. [06:45] apt-get install python-django-piston [06:45] if you want to query availability, use "apt-cache search " [06:45] !? How can that work in apt and not dselect? [06:45] jtv: I am only a bot, please don't think I'm intelligent :) [06:45] Oh shut up. [06:45] dpkg -l shows if it's installed [06:45] I know about apt-cache search, it's just a hassle compared to interactive pattern searching. [06:46] don't use dselect [06:46] But why doesn't it see the package? [06:46] Oh God it has its own cached package lists… [06:46] NFI [06:46] I don't even have it installed [06:47] friends don't let friends use dselect [06:48] I've been used to it for about a decade. [06:48] I found it friendlier than aptitude. [06:48] don't use aptitide [06:48] aptitude even [06:49] while it works most of the time, it does have a different dependency resolver to apt, and apt is the reference [06:52] jtv: https://code.launchpad.net/~julian-edwards/maas/fix-quantal-piston/+merge/125638 [06:52] Some people swear by aptitude's resolver, apparently. Again, I'm more used to apt's. [06:53] it can offer some interesting resolutions to conflicts, but they are likely to break things even more. [06:53] Reviewed. [06:53] cheers [06:53] Had some weird trouble with the “lxml >= 2.3.2” in Precise, will try in Quantal. [06:57] oh? [06:58] The error is one of those things that make you go “yes, and…?” [06:58] «Error: Picked: lxml = 2.3.5» [06:58] :) [06:58] oooooo I know [06:58] Let's turn that smile up-side down. [06:58] we have explicit versions turned on iirc [06:58] bugger [06:59] allow-picked-versions = false [06:59] bugger [06:59] why, I wonder [06:59] we really need to start uding system packages more instead of pypi [06:59] using* [07:00] morning rbasak [07:00] argh [07:00] morning rvba [07:00] Hi bigjools. [07:02] Hi rvba [07:03] \o jtv. [07:03] Yes, rbasak does raise the suspicion of queue-jumping. [07:08] I have to reboot, because plugging in new usb device seems to still have a bug that is now about 4 years old which makes the mouse disconnect >:( [07:08] It'll get triaged any day now, I'm sure [07:09] srsly annoying. [07:09] brb [07:13] rvba: would appreciate your input on the django incantations I applied here → https://code.launchpad.net/~jtv/maas/node-nodegroup-api-ui/+merge/125624 [07:13] jtv: ok, I'll have a look. [07:13] Thanks [07:32] rvba: did you see my packaging suggestion around clusters etc? [07:32] bigjools: not yet. [07:33] rvba: check out maas-devel [07:35] Reading it right now. [07:35] jtv: I've reviewed your branch. Looks ok but I've got a few remarks. [07:35] Thanks for the additional notes. [07:37] bigjools: I've made some experiments on how to route tasks in celery yesterday btw. [07:37] rvba: cool, all ok? [07:38] bigjools: yes, queue creation works, routing ok, broadcast messages ok. [07:38] jtv: thanks for fixing the get_db_prep_lookup thing [07:38] np [07:38] good - I played around a long time ago and it was ok in my experiments so good to hear you went well too [07:39] I'm happy at last to have validation for my habit of “if it's somebody else's signature, accept *args & **kwargs and pass them in the upcall.” :) [07:42] GNARGH [07:42] bigjools: btw, when running the test suite on quantal, I've got only 2 failures, all related to the api doc. [07:42] usb disconnection AGAIN [07:42] rvba: yes that is all that is left now [07:43] bigjools: not so bad after all then :) [07:43] rvba: that's because we've been fixing them all day ;) [07:43] * bigjools reboots AGAIN === bigjools-afk is now known as bigjools [08:17] jam, bigjools: Yes, bzrlib's there just for osutils. I considered breaking out that code. It's a big dependency, but I think it's fairly low priority to fix that. [08:28] allenap: i guess bzr=2.5.1 fails on Q because bzr-2.6b? is there [09:13] jam, jelmer: http://pastebin.ubuntu.com/1218325/ [09:31] when do I need to care about django model identity... [09:31] self.assertEqual(xmlbytes, Node.objects.get(id=node.id).hardware_details) [09:31] self.assertEqual(xmlbytes, node.hardware_details) [09:31] not equivalent [09:34] what I want it seems: [09:34] self.assertEqual(xmlbytes, Node.objects.get(id=node.id).hardware_details) [09:34] self.assertEqual(xmlbytes, node.hardware_details) [09:34] bah! cp [09:34] node = reload_object(node) [10:51] poke mgz for great justice [10:52] and jelmer for goodness sake [10:53] jam, just about to nip out, but nearly there (despite django) with setting cpu_count/memory [10:53] jam: Yeah, I think so. It's not able to arrange for an earlier version of bzrlib because we allow site packages. [10:53] mgz: just curious where your db patch is at, since I cant evaluate tags until we have the column [10:54] ah, I still need to actually propose the branch... [10:57] jam: pushed to lp:~gz/maas/temp and I need to run [11:51] morning [11:51] rvba: is the webui only to be meant installed on the region? [11:52] roaksoax: yes [11:53] cool [12:24] I'm running into something strange with django.piston and nested fields [12:25] It doesn't seem to restrict the output to just the fields that are specified [12:26] jam: sounds like more joy [12:26] mgz: I'm trying to confirm w/ macaddress_set since that was doing ('mac_address',) [12:26] but I can't seem to get it to only return just the 'name' and 'resource_uri' of the tag [12:26] of related tags. [12:29] jam: I can't get UPDATE to work at all in a raw query... [12:29] rvba: ^^ Specifically, I'm trying to expose tags, but just saying "NODE_DISPLAYED_FIELDS = (..., 'tags')" means the full tag gets exposed. [12:32] jam: can't you apply what we've done with 'macaddress_set' (in DISPLAYED_NODE_FIELDS)? [12:33] jam: DISPLAYED_NODE_FIELDS= (…, ('macaddress_set', ('mac_address',)),…) [12:33] rvba: that doesn't seem to do what we think it does [12:33] I can't confirm mac_address yet, as I haven't found a test that actually tests the values in macaddress_set [12:33] however, if I put ('tags', ('name', )) [12:33] I still get all the fields [12:34] jam: let me have a look at the code. [12:35] rvba: lp:///~jameinel/maas/tag-list-api [12:35] has a failing test [12:35] that shows what fields are being exposed. [12:36] jam: cool, I'll have look. [12:36] rvba: and I see tests about 'POST_limited_fields' but none of them seem to test the actual content of the macaddress_set field, just that it exists [12:37] jam: indeed. [12:37] rvba: and I do feel like what we are doing *should* work, from places like http://stackoverflow.com/questions/2215352/django-piston-how-can-i-exclude-nested-fields-from-handler-results-is-it-even [12:38] and here: https://bitbucket.org/jespern/django-piston/wiki/FAQ#!why-does-piston-use-fields-from-previous-handlers [12:39] but that doesn't seem to actually work [12:40] jelmer: any update to where you got to? [12:43] okay, I think I have code ready for the tag stuff now jam [12:46] jam: I don't understand why it does not work. I'll have to put my hazmat suit on and debug piston. [12:47] rvba: so at least I'm not crazy [12:47] ...now kapil will wonder why you want to dress up as him... [12:48] haha :) [12:51] mgz: https://code.launchpad.net/~jameinel/maas/tag-list-api/+merge/125697 up for review [12:53] jam: I got it. [12:54] jam: it's using TagHandler.fields. [12:55] rvba: which I thought the definition in NodeHandler is supposed to override it for exposing tags on node [12:55] if you know a way to do it, I would be fine turning the tags attribute into a simple list of names [12:55] rather than a list of objects [12:55] jam: let me investigate some more… [12:55] but we are using the default 'get' handler, and I don't know how to easily override it. [13:08] jam: I have put up some branches for review [13:10] jam: It does not seem to be possible to do that with piston (I even found a fork on github which adds this possibility https://bitbucket.org/liberation/django-piston); the only way is by adding a custom method like this: http://paste.ubuntu.com/1218559/ [13:10] rvba: well, that change looks good to me, so I'll go with that. Thansk [13:10] np [13:11] rvba: which is opposite to what I read https://bitbucket.org/jespern/django-piston/wiki/FAQ#!why-does-piston-use-fields-from-previous-handlers [13:11] but hey, tag_names actually works better for my purpose, I think [13:13]