[00:11] hi roaksoax and smoser [00:11] bigjools, hey [00:12] this morning would have been better had my wife not woken me up from the deepest sleep ever [00:17] bigjools, :-( [00:17] lp:~smoser/+junk/backdoor-image/ [00:17] has "backdoor" [00:17] cool [00:17] * bigjools grabs [00:17] just point it at an image and it will backdoor it [00:18] user 'backdoor' [00:18] I don't normally do backdoor [00:20] --user bigjools [00:20] :) [00:21] it's the ephemeral image I want in this case, right? [00:22] right. [00:22] /var/lib/ephemeral/.....disk.img [00:22] its probably most proper to 'restart tgt' afterwards [00:22] but i've never actually had to do that. [00:23] but clearly if it had open filehandles, it could be confusing. [00:23] yeah [00:23] I was going to ask - is it possible to proxy tgt via the clusters? at the moment ephemerals get pulled from the region [00:24] i dont know. i think the cluster would just want to run tgt also. [00:24] you want that to be as close as possible [00:24] its block level io [00:24] if i understand you correctly [00:24] bigjools, oh. [00:25] i thought about one thign that might be screwing you. [00:25] i dont think proxy settings or mirrors get sent down to commissioning [00:25] it's just that it's a bottleneck right now - we might need to start copying ephemerals to clusters [00:25] right. thats what i was saying. i think you should plan on copying ephemerals to clusters. [00:25] * bigjools tries commissioning again [00:25] i have to run [00:26] thanks for the script [00:26] i've tested it on / [00:26] but not actually on an ephemeral image. [00:26] ok :) [00:26] i've looked at it though (when pointed at an image) [00:26] but on / it added a user that i could ssh in as [00:26] so it seemed ok. [00:26] later. [00:26] cheers [00:27] oooo [00:27] I saw an error flash by [00:27] something about tty error [00:27] * bigjools tries to log in [00:28] tty error. [00:28] that is strange. [00:29] I think it mentioned stderr too [00:29] Oct 5 00:27:16 10-0-0-100 kernel: [ 25.467047] init: cloud-config main process (899) terminated with status 1 [00:29] Oct 5 00:27:16 10-0-0-100 kernel: [ 25.885676] init: cloud-final main process (1049) terminated with status 1 [00:30] copy off /var/log/cloud-init.log [00:30] and /var/log/cloud-init-output.log [00:30] (you cna proably sudo apt-get install pastebinit) [00:30] and do it that way [00:30] ProcessExecutionError: Unexpected error while running command. [00:30] Command: ['locale-gen', 'en_US.UTF-8'] [00:30] Exit code: 1 [00:30] there [00:30] I can scp it then paste, one sec [00:32] smoser: no cloud-init-output file, but here's cloud-init.log: http://pastebin.ubuntu.com/1261098/ [00:33] look around line 246 onwards [00:36] bigjools, /var/lib/cloud/instance/cloud-config.txt and /var/lib/cloud/instance/user-data.txt.i [00:37] i really have no idea why 'local-gen' would fail like that. but its not deadly. [00:37] cloud-config.txt is empty [00:38] user data has plenty [00:39] ah. yeah, it would be user-data. [00:39] there is no cloud-config in that. [00:39] but user-data has that script. and that *shoudl* get run. [00:41] http://paste.ubuntu.com/1261109/ is its contents [00:44] bigjools, something is stopping you from reaching runlevel 3 i think. [00:44] so the cloud-final job is not running [00:45] bigjools, what does [00:45] 'runlevel' [00:45] and sudo status rc [00:45] show? [00:47] N 2 [00:47] and [00:47] rc stop/waiting [00:48] it must be that tty error [00:48] smoser: ^ [00:49] oh wait. that is wierd. [00:49] cloud-final ran [00:50] try running it again [00:50] sudo start cloud-final [00:50] start: Job failed to start [00:56] I can't find any log for it [00:56] ok. try it more manually. i guess. [00:56] sudo cloud-init modules --mode=final --debug [00:57] need --debug before modules [00:58] boom [00:59] well it has a lot of tracebacks but it did complete [00:59] machine powered off [00:59] smoser: so something is wrong with the upstart conf perhaps? [01:00] log: http://paste.ubuntu.com/1261129 [01:02] how much memory on the system? [01:02] the FALLBACK stff is not really a big deal. its operating as mostly designed. [01:02] 2Gb [01:03] the issue is that it logs to rsyslog, but then from within it, the userdata its running calls /sbin/poweroff [01:03] it's an HP cube [01:03] and rsyslog gets killed [01:03] so logging breaks. [01:03] but id ont understand why it wouldnt have run [01:03] i have no idea really. [01:03] why does it only happen when console=ttyS0 is specified? [01:03] something failed. [01:04] it doesn't make any sense to me. [01:04] jeez, 29C at 11am, gonna be hot today [01:06] I'll retry and see if I can see that error on the console, now I know when to look for it [01:08] ah it did it on enlistment too [01:08] stty: standard-input: input/output error [01:09] I am going to file a critical bug [01:10] the stty error is maybe not related. [01:10] I am thinking that [01:10] can you get dmesg [01:10] and basically just tar up [01:10] /var/log/ === matsubara is now known as matsubara-afk [01:11] it just doesn't make sense. [01:11] bigjools, now that i think about it the tee /dev/tty2 is probably a bad idea. [01:11] as maybe the tee is in this situation too. [01:12] since we're /sbin/halting [01:12] the tee might get killed and cause unnecessary angst. [01:12] i like the log though. [01:12] maybe we should have the script do [01:12] sh -c 'sleep 10 && /bin/poweroff' & [01:13] i can come up with something better too, but basically just amke sure that cloud-init is done. [01:14] smoser: why does "start cloud-final" fail? [01:14] isn't that a hint? [01:14] https://bugs.launchpad.net/ubuntu/+source/cloud-init/+bug/1061977 [01:14] Ubuntu bug 1061977 in MAAS "Machine fails to commission when console=ttyS0 is present on kernel opts" [Critical,Triaged] [01:15] can youg get dmesg [01:15] and the /var/log attachd there. [01:15] yeah [01:18] it is /dev/console [01:18] it completely is that [01:18] the fact that running it trhoug upstart died [01:18] and then manually did not [01:18] because its output will go to /dev/console when you do it via upstart [01:19] what from /var/log do you want? [01:19] maybe the kernel has a buffer and just gets fed up at some point. [01:19] jsut get it all [01:19] k [01:19] if you dont mind. [01:19] you didn't have any private data there :) [01:19] haha :) [01:21] oh. yeah. [01:21] while you're on that machine [01:21] can you just try [01:21] echo HI MOM | sudo tee /dev/console [01:21] and see what it does [01:22] i think we're gonna get an input output error basically. [01:22] tee: /dev/console: Input/output error [01:22] yeah. so it makes sense. [01:22] but i thought the kernel was supposed to be smarter. [01:22] you do have 'console=tty1' on the cmdline also, right? [01:22] yes [01:26] so there is no physical serial port, right? [01:26] just to be sure. [01:26] there isn't - unless the BMC is providing one surruptitiosly [01:28] hm.. [01:28] interesting. [01:28] bigjools, you see kernel output on the monitor, right? [01:28] I do [01:30] bigjools, the localegen died because it tried to write to its stdout [01:39] bigjools, so i've got to go afk. [01:39] but i'd like it if you reviewd my patch for https://code.launchpad.net/~smoser/maas/preserve-sources-list/+merge/127825 [01:39] smoser: ok [01:40] particularly, i'm not happy about the test, it seems long winded, but didn't know how to make better. [01:40] i dont really know what to do for your ttyS0 issue. [01:40] other than not specifying console= at all [01:40] which imo seems busted. [01:40] smoser: we have to remove it [01:40] as a default. [01:40] busted, but less busted than not booting at all [01:40] well, to be fair, you are the first person to see this. [01:41] how much testing has it had though? [01:41] not on VMs [01:41] more than a few systems. [01:41] I bet the others all have serial console [01:41] well, yes. [01:41] and so will the real target audience. [01:42] so changing the default (which is hard coded in source code) to accomodate a little toy system [01:42] doesn't make a lot of sense to me. [01:43] well hang on [01:43] you don't know that every system will have a serial console [01:45] you're right. [01:45] but i don't that it generally fails so badly if that is the case. [01:46] kernel bug? [01:48] http://www.mjmwired.net/kernel/Documentation/networking/netconsole.txt [01:48] i've always wanted to play with that. [01:54] smoser: I'll improve your test and land your branch [01:57] thanks. i'll poke the kernel guys tomorrow am. [01:57] but i dont have any ideas. === jtv1 is now known as jtv [02:11] http://www.mail-archive.com/linux-kernel@vger.kernel.org/msg246433.html [02:39] howdy [02:50] bigjools: howdy [02:54] roaksoax: hey [02:55] bigjools: so how do you feel about the packaging? [02:55] roaksoax: positive [02:55] bigjools: alright, i'm gonna try to upload to archive tomorrow so we can test upgrades from precise [02:56] roaksoax: \o/ [02:56] and get community to find more bugs if any [02:56] that's would help us too [02:56] I did a fix the other day to prevent questions when ugprading, fingers crossed :) [02:57] bigjools: which one is it? [02:58] roaksoax: config for cluster controller [02:59] it looks at DEFAULT_MAAS_URL if it can find it [02:59] bigjools: the autodetection of maas-url, yeah I saw [02:59] bigjools: it works [02:59] I did test it :) [02:59] bigjools: yeah me too, it works :) [02:59] are you still thinking about changing the sed stuff? [03:00] bigjools: i don't know TBH.. doing so will require .d support for the py's [03:01] bigjools: and would require a way to provide .d for the yaml conf's too [03:01] I fear that is too much work at this late stage :( [03:01] bigjools: Indeed!! the easier way right now is simply installing to /usr/share/maas and then symlink everything to /etc/maas/ [03:01] yeah [03:02] bigjools: however, the problme with that is that user settings won't be preserved on upgrade [03:02] bigjools: i also found another bug in upgrading but with dbconfig-common, for some reason it was not preserving the password info and was re-writing the config file. [03:03] the latter is expected, but should have presrved the password, but anyways, i work arounded it by simply sourcing that file and grabbing the password from there instead of letting dbconfig-common to rewrite the config [03:04] bigjools: now to continue on the config, I think it would be the best for now... will have to check with Daviey to see what's his appreciation on this [03:04] bigjools: maybe we can simply send the configs in /usr/share/maas and add some code in the configs itself that source whatever is in /etc/maas [03:05] smoser: still around? [03:33] roaksoax: are you going to land your packaging branch? [03:33] bigjools: yeah, i'm just testing one more thing [03:34] ok I put the bug back to in-progress :) [03:34] tarmac will set it when the branch lands [03:34] cool [04:35] roaksoax: I shall force a daily build now [04:36] bigjools: sure [04:37] being a build admin has its bonuses [04:38] bigjools: indeed it is [04:38] i envy you [04:38] hahah [04:38] i used to use PPA's extensively [04:40] * roaksoax bed [04:40] we bumped the prio permanently on the daily build ppa :) [04:40] nn roaksoax [04:40] night [06:23] allenap, rvba: So it turns out that MAASClient doesn't return an object like the Django Client return. It has a 'object.code' vs 'object.status_code' and 'object.read()' rather than 'object.content' [06:23] so... mocking/stubbing/ bad for the real world :) === lifeless_ is now known as lifeless [06:50] allenap: you lied to me... :), list data is not supported by MAASClient. When it gets down into 'make_payload' you support bytes/unicode/IOBase and callable. [07:03] hey lifeless, any idea wtf is going on here? http://paste.ubuntu.com/1261414 [07:21] I think we have a nasty bug with omshell here :/ [07:24] good grief, I thought I'd seen bad code but this is hideous. http://ftp.fr.netbsd.org/cvsroot/src/usr.sbin/dhcp/common/Attic/parse.c,v [07:30] bigjools: looks like a url to me [07:30] bigjools: maybe it needs quotes ? [07:30] lifeless: I think the base64 parser in omshell blows [07:30] quotes don't help [07:30] bigjools: that is very odd [07:30] looking at the code it treats + and / specially [07:30] oh joy. [07:31] but the code is sooooo bad that it's taking me a while to work out what [08:22] jam: Sorry, I didn't realise you were talking about MAASClient. The cli supports multiple values per key, and so does the server side; it's just MAASClient that needs a little love to get it there. [08:31] allenap: yeah, I got it to work with https://code.launchpad.net/~jameinel/maas/maasclient-multipart/+merge/128176 [08:31] though it only fixes POST, GET uses urlencode() which just strs the list [08:32] allenap: is there a good way to generate a huge number of nodes for testing? Like I want to populate the DB with 1000, 10000, 100,000 nodes. [08:33] jam: There's a way. I'm not sure about a good way. [08:33] allenap: well, pressing 'add node' in the web ui is pretty bad [08:33] I can do it in SQL surgery, but I need to generate mac addresses, etc. for each. [08:33] So it makes me think I should do it in python, but via the API or just direct on the db [08:34] but how hard is it to grab maasserver.model objects and play with them [08:34] I imagine 'settings' needs to be set correctly for me to mutate the db [08:42] jam: How about: make syncdb && make harness; from maasserver.testing.factory import factory; factory.make_node() [08:42] allenap: if make harness will do it, that works for me [08:43] That was the sort of command I was looking for [09:08] w7z: mumble? [10:40] allenap: so what can I do to land the multipart list stuff? I don't actually have a 'mapping' I have an Iterable [10:40] but string is an Iterable [10:40] though apparently getattr(s, '__iter__') fails. [10:42] there's no sane way to do it without isinstance [10:42] jam: If it's not a string type, assume it's iterable of string types, and let things blow up if it's not? [10:42] allenap: well there are layering issues. The part that checks if it is a string type only returns 1 payload to attach, so you need to loop at a higher level. [10:42] or change the lower thing to always return a list of payloads [10:42] or.. [10:43] Blast. [10:43] (It also accepts files) [10:43] which are iterable, and you want to upload the whole file as one payload [10:44] jam: A file isn't a collections.Sequence... but I guess stick with list and put it in the docstring. [10:45] allenap: I could do "if isinstance(x, collections.Sequence) and not isinstance(x, basestring)" [10:47] jam: Yeah. Or change make_payload to gen_payloads, so allowing it to yield multiple things, then all the isSomething checks can be done there. [10:49] jam: as I said on the MP, I don't understand ~jameinel/maas/ignore_results, we have a global celery settings for that. [10:49] rvba: where? (my system at least was improved by nuking the mnesia schema) [10:49] rvba: if you can point me to it, I'll happily reject/revert my patch. [10:49] jam: CELERY_IGNORE_RESULT is set to True in etc/celeryconfig_common.py [10:49] as it could be me killing things repeatedly. [10:50] rvba: so right now, I have to install maas-dns in order to do 'make run' is that true? [10:50] if I don't the service fails to startup properly [10:50] jam: Something like http://paste.ubuntu.com/1261655/ [10:50] and the only log I see is the 'you should install maas-dns' [10:51] * allenap looks forward to "yield from" [10:51] jam: no, a local named instance is started when you run 'make run. [10:51] ' [10:51] rvba: well, I can't get 'make run' to talk to me, the last error in the terminal is the 'you should install maas-dns on the region controller' [10:52] I had this problem for quite a while this week. [10:52] until it magically fixed itself [10:52] jam: that message is for the packaged version, I did not bother changing that message in the dev instance. [10:52] (my best guess at this point, is I got enough queues laying around that rabbit would say 'come back later' when asked to update dns, so it wasn't crashing the startup process.) [11:13] allenap: I switched to the 'yield' form, care to approve? [11:13] jam: Sure. [11:19] jam: I'm seeing the dns error too, I'll investigate. But this is only a task failing, it should not cause trouble for the other services. [11:21] rvba: so I installed maas-dns, which installed maas, I then uninstalled it, but I now have a twistd process runinng. [11:21] jam: +1 :) [11:21] everytime I kill it, it comes back with a new pid [11:21] I'm guessing upstart is keeping it alive? [11:22] ah, maas-txlongpoll [11:22] which didn't get uninstalle [11:22] d [11:22] jam: as I said, you don't have to install maas-dns on a dev instance… also, you'll get into trouble if you install the maas package and try to play with a dev instance at the same time :/ [11:22] Gah, so buildout says it's installing testresources 0.2.5, but then uses the system one, 0.2.4. How completely useless. [11:23] allenap: buildout config as it stands takes system packages over packages it installs. [11:23] I brought it up earlier [11:23] as being useless, too. [11:24] (vs using system packages if there isn't a custom package installed) [11:24] allenap: so you have to uninstall the system package to have buildout get the right one. [11:24] and then re-install it in the future when you want to use the system package for some other project. [11:24] jam: Ah, right, thanks. At least there's a workaround. [11:26] rvba: uninstalling package 'maas' is triggering maas-txlongpoll to *start* [11:26] jam: ! [11:26] (which naturally prevents uninstalling maas) [11:28] jam: I've spotted one problem: looks like the wrong celeryconfig is loaded on a dev instance (by the region worker at least). [11:28] That's why its trying to write to /etc/bind/maas and failing. [11:42] rvba: I'm also getting failures in "_write_temp_file" in provisioningserver/utils.py [11:43] (when I create a node, it triggers writing the dns config again, and I get a failure trying to write the temp file) [11:44] rvba: so it looks like that is the N^2 behavior I was seeing. Every node added is regenerating a full DNS list. [11:44] (it is failing, but it still is pulling together the data which is O(N) and when you allocate N nodes == O(N^2) by the end) [11:48] jam: the DNS writing task should only be triggered with the node gets an IP address. Only then do we need to update the DNS config. [11:48] s/with the node/when the node/ [11:48] rvba: well I'm doing factory.make_node() in a loop [11:48] which probably gives addresses? [11:49] jam: no, there is no lease creation in there as far as I can tell. [11:49] rvba: maybe it knows the dns config didn't get written properly yet? [11:50] it is definitely looping on failing to create it. [11:50] but it may be unrelated to me creating 1k nodes. [11:51] "Sending due task 'upload_dhcp_leases'" is in the log file an awful lot. [11:51] That's a celerybeat task that is run every minute. [11:58] rvba: I'm seeing it roughly every few seconds [11:58] however, 'make_node' also creates a new nodegroup if you don't pass one in. [11:59] jam: ah! right, that's what triggering all the DNS stuff. [12:00] jam: because each nodegroup gets created with a configured interface IIRC. [12:13] jam: I've just fixed the "wrong celeryconfig" problem. A stupid mistake in the startup script. [12:13] rvba: \o/ [12:13] so creating 1000 nodegroups takes about 15min, but creating 1000 nodes is <1min. [12:13] that seems reasonable. [12:14] jam: The plan is to remove the DNS config pre-population. But post 12.10 release. [12:15] rvba: sure, but you won't be creating 100,000 nodegroups in the immediate term, so 15min for 1000 seems fine. [12:16] jam: right. [13:20] rvba: Do you know if the [/+] needs to be on both sides for the [/+]no[/+] bug to manifest, or either side? [13:21] allenap: from Julian's investigation (and the mailing list message we found), I think it's on both sides. [13:22] rvba: That's one of the weirdest bugs I've ever heard of. [13:22] allenap: yeah :/. [13:24] rvba: Do you know anything about the code that submits commissioning results (op=signal)? [13:25] allenap: not really, maybe I help you with something still? [13:25] I can* [13:31] rvba: I'm fixing the code that sets the power parameters. Now, it was broken: it was checking for power_type.upper() in map_enum(POWER_TYPES), i.e. checking that the value of power_type given over the wire matched the Python name of the enum item. [13:33] allenap: what's wrong exactly? [13:34] rvba: Well, that works for IPMI, because the Python name == uppercase(enum value). [13:34] rvba: But not for the others. [13:35] rvba: I mean, we should be expecting the enum *values* over the wire, right? [13:36] allenap: right! [13:48] rvba: Okay, I'm not mad then. The reason I want to ask someone about it is to avoid breaking scripts somewhere else. tl;dr this is what tdd looks like without t. [13:49] allenap: haha :). I think Julian mentioned IDD recently. === dpb_ is now known as Guest72485 [14:02] allenap: hm,in his comment in the code, Julian said "if '/' or '+' appear either side of the word 'no'" [14:04] allenap: rarg, I've tested it and it's either side indeed, not both sides. [14:05] rvba: Gah. Top marks for you though, for confirming. === matsubara-afk is now known as matsubara [14:10] mgz: any progress on search? [14:11] jam: getting there [14:13] rvba: Do you have any time this afternoon to review the extra changes I had to make to https://code.launchpad.net/~allenap/maas/anon-power-setting/+merge/128127? [14:13] allenap: sure, I'll do it right now. [14:13] rvba: Thanks. [14:15] rvba howdy¡¡ i pushed the branch and selected u as reviewer [14:15] did you see it¿ [14:15] ? [14:15] roaksoax: I did :) [14:15] rvba cool then :) [14:15] thanks [14:16] roaksoax: I'll get to it in ~30 minutes. [14:16] rvba awesome thanks [14:17] allenap thanks for taking care of the power settings for enlistment [14:20] mgz: so I just did a quick 'wrap process_node_tags in lsprof' and the results are: out of 107s under profiling, 7.8s is parsing and processing the XML, 67s is downloading the hardware details, 32s is uploading the system_ids results, and 2.3s is getting the system_ids list. [14:20] I think the MAASClient handling of repeated tags is a bit slow (I think it encodes each value as a separate multipart message section. [14:20] uploading 12,000 values is a lot, but not 32s a lot, given that we can read that many tags in 2.3s. [14:21] roaksoax: No worries. It hasn't landed yet... when I picked up that rock I found some bugs :) [14:22] so, it is the passing data around overhead... still seems very high [14:22] And then getting the hardware details is *way* too expensive as well, given if it was a raw DB query, we could get the results in... checking. [14:23] I get the first result here in about 10s. [14:23] 32s to get everything. [14:23] mgz: 32s just to get the content out seems a bit too expensive as well. [14:24] time xpath_exists is still 6.8s, time in lxml on the current code is only 7.8s. [14:25] but "select hardware_details from maasserver_node" > /dev/null is 32s. [14:25] mgz: Is there a lot of quoting that Postgres is having to do? [14:26] ...no, but maybe it's re-serialising? it shouldn't need to, but the 9.1 support does seem a little unpolished. [14:28] mgz: time to select from an xml column into a new text table is 4.1s. [14:28] there might be a magic cast that will help? [14:28] what does adding ::text do if anything? [14:29] mgz: time select content from alt_hardware_details >/dev/null => 32s. [14:29] gr. [14:29] might be sane to just ditch some of the db changes and pull from the original location :) [14:31] mgz: interestingly, using 'bytea' instead of text is *slower* [14:31] 1m24s [14:33] heh, I think that's probably django [14:36] mgz: this is in psql [14:36] no django involved [14:36] psql -c "select ..." [14:36] ...that's very suprising then [14:36] mgz: so something very strange when 'select DATA from...' is slower than 'select xpath_exists(DATA)' on the same content. [14:37]