[00:12] arosales, interesting re affinity group bug [00:13] arosales, did red squad implement affinity groups recently? it wasn't an issue last week [00:15] sinzui: no [00:15] sinzui: only minor jumps of .2 at this stage [00:15] sinzui: planning to fix this later [00:15] sinzui: has the world stopped burning? [00:18] sinzui, i though it looked for the lastest compatible minor release [00:20] whoops thumper ^ [00:21] looking at upgradejuju initVersions [00:22] thumper, per environ/tools.FindTools ... f minorVersion = -1, then only majorVersion is considered. [00:22] s/f/if [00:23] hazmat: whether or not it looks for the latest tools is irrelevant, the code doesn't promise to work [00:23] I have been told loudly and often that we only support single jumps at this stage [00:23] and not from dev -> dev [00:23] only from released -> current dev [00:23] or old release -> current release [00:24] or last release -> current release [00:24] well old_release -> current_release is the question.. and if the code doesn't prevent it, then the default behavior is effectively not supported [00:25] ie 1.12 upgrade will go straight to 1.16 or 1.18.. [00:25] :) [00:25] hazmat: well, wether is does or not is a bit moot [00:26] either its supported and its fine, or its not supported, and we have default behavior which is potentially dangerous for prod envs. [00:26] thumper, then we can talk about downgrading ;-) [00:26] well, I've been told it isn't supported, so we should check to see if it dies :) [00:27] thankfully a test trivial with a downgrade.. [00:28] downgrade ? [00:28] are you mad [00:29] the only way is up, baby [00:29] davecheney, i filed a bug and told it was meant to work for downgrade.. [00:29] but you can --version=previous_version [00:29] unfortunately can't go back to 1.12 since its not in simple streams.. [00:33] which also sort of explains the upgrade problems [00:34] hazmat, ack thanks. I do have cgroup-lite installed. [00:34] (1.8) [00:36] gary_poster, yeah.. i looked over the tracebacks again.. didn't seem related, not sure what's up there [00:37] gary_poster, it did seem possibly networking related, but unclear.. [00:37] gary_poster, could you pastebin ifconfig from the host [00:38] hazmat http://paste.ubuntu.com/6177815/ [00:38] totally sane.. [00:38] thumper, gary_poster i'd ask hallyn about this one [00:39] hazmat, I was thinking the same. [00:40] sinzui, thumper so with deploys coming from the new streams structure, effectively the only version of juju known to trunk is 1.15 atm.. no real testing of multi-version leaps atm unless using an old client [00:41] hazmat: i don't think downgrades have ever been supportde [00:41] i've certainly never heard that requirement, and it has never been a constraint [00:43] davecheney, fair enough.. this notion of 'support' only known to juju-core devs is interesting.. my concern is primarily what does the tool allow end users. if minor increment skipping isn't supported, why does the tool allow it.. much less do it by default.. if downgrading isn't supported, how about a warning at least.. if not remove the capability [00:44] hazmat: how can you downgrade ? [00:45] maybe this was added when I wasn't looking [00:45] davecheney, juju upgrade-juju --version=lower_version [00:46] meh, that is like --to, if you do that, all the safties are demoved [00:46] removed [00:46] davecheney, https://bugs.launchpad.net/juju-core/+bug/1227991 [00:46] <_mup_> Bug #1227991: upgrade-juju can downgrade.. and causes hook executions [00:46] yup, certainly a footgun [00:49] davecheney, --to doesn't remove safeties from juju, just the charms, and its quite safe for container spec [00:50] although honestly i like the capability.. what i dislike is a default behavior which only juju-core devs knows is not supported, namely skipping versions on upgrades. [00:50] ie juju upgrade-juju.. resulting in an unsupported behavior [00:51] * hazmat files a bug for future ref === gary_poster is now known as gary_poster|away [00:58] ah ffs [00:59] lxc has changed its behaviour for another thing [00:59] fuckity fuck fuck [00:59] * thumper looks at the docs [00:59] * thumper goes to write another branch [01:06] thumper: what's the cut off date for 1.16.0? [01:14] axw: soft cut off is thursday I think [01:14] and then it gets harder :) [01:14] thumper: okey dokey, ta [01:15] public holiday came at a bad time :\ [01:15] * thumper shrugs [01:15] * thumper goes to fix the lxc bug [01:17] thumper: sounds like the same story as last time [01:17] there is a date [01:17] but it's optional [01:17] depending on your frame of reference [01:18] davecheney: I would say that we only want bug fixes after that [01:18] and not try to drop features [01:20] drop == include [01:20] or drop == exclude ? [01:20] drop == land ? [01:21] why why did I make my admin-secret so secret [01:21] such a pain [01:21] y didn't i make it 'bob' [01:26] not to land features [01:28] understood [01:28] * thumper needs hacking music [02:00] axw, davecheney: I'm confused [02:00] I use os.Symlink to make a symlink [02:00] but it doesn't set the symlink mode flag [02:00] if I go os.Stat(path), fileinfo.Mode().IsRegular() it returns true [02:00] why? [02:01] thumper: not sure [02:01] let me try [02:01] I'm writing a checker for IsSymlink [02:01] also not sure.. [02:01] so in my test test, I make a symlink , then test it [02:01] and it says nah [02:02] * thumper is confused [02:02] c.Assert(symlinkPath, jc.IsSymlink) [02:02] ... obtained string = "/tmp/gocheck-5577006791947779410/8/a-symlink" [02:02] ... /tmp/gocheck-5577006791947779410/8/a-symlink is not a symlink: &{name:a-symlink size:0 mode:384 modTime:{sec:63516189566 nsec:62250446 loc:0x6c71c0} sys:0xc2000975a0}, true [02:02] anyone seen this one.. https://bugs.launchpad.net/bugs/1233457 [02:02] <_mup_> Bug #1233457: service with no units stuck in lifecycle dying

[02:03] file, err := ioutil.TempFile(c.MkDir(), "") [02:03] c.Assert(err, gc.IsNil) [02:03] symlinkPath := filepath.Join(filepath.Dir(file.Name()), "a-symlink") [02:03] err = os.Symlink(file.Name(), symlinkPath) [02:03] c.Assert(err, gc.IsNil) [02:03] cts needs a fix, and afaics the only thing to do is munge the db.. [02:03] how about those pastebins [02:03] hazmat: sure... [02:04] thumper: I think you want os.Lstat [02:04] ah... [02:04] yeah [02:04] Stat is checking the thing that is being pointed to right? [02:04] yup [02:04] thumper, python is the same btw [02:04] os.lstat vs os.stat [02:04] yeah, that's it [02:05] hazmat: no, not seen that before [02:05] although maybe I have, but I don't recall [02:06] we have some lifecycle issues [02:12] thumper, so what's the downside of just yanking that document from mongo.. [02:12] hazmat: unsure [02:13] hazmat: nobody has ever tried [02:13] afaics as long as its cleared out of units, relations, relation scopes, its fine. [02:14] thumper, probably some txn issues as well [02:14] yeah... that's the problem.. prod customer env.. doing the crazy.. i guess i need a full db dump to verify this is sane.. [02:15] but against a sample env, it does look okay.. [02:16] the txn observers complicate things, but nothing triggers on this observation [02:16] thumper, hmm.. let me ask a different way.. do you know what processes the cleanup of services in lifecycle:dying [02:16] would have to be the provisioner [02:16] ie could wondering if i could just bounce the bootstrap jujud to process it [02:17] hazmat: no, there is a lifecycle doc written by fwereade in the tree, but no I don't really know [02:17] hazmat: try it [02:17] can't hurt [02:17] davecheney, why the provisioner? [02:17] its not related to the provider [02:17] hazmat: it's the only one left [02:17] unit agent moves the unit to dying [02:17] there is no service agent [02:17] so it must be the provisioner [02:18] davecheney, fair enough [02:18] davecheney, or at least a job on the machine agent [02:18] on bootstrap [02:18] ie could just as easily be api [02:18] or something else [02:19] davecheney, you reasonably sure that bouncing it won't pertub the system? === julianwa_ is now known as julian === julian is now known as juliawa [02:20] hazmat: it could be the machine agent [02:20] but that is a problem [02:20] because when the last machine is removed [02:20] who will remove the service ? [02:20] the machine agent moves the unit from dying to dead [02:20] that is all === juliawa is now known as julianwa [02:21] hi julianwa [02:21] hi davecheney hazmat [02:21] hazmat: yes [02:21] bouncing it is fine [02:21] julianwa, so the suggestion is juju ssh 0 ... and then [02:21] it is designed for this [02:22] julianwa, sudo service jujud-machine-0 restart [02:22] hazmat: ok, will try in few minutes [02:22] julianwa, that might resolve it as forces the agent to process current state, shouldn't pertub the system at all [02:23] hazmat: the bootstrap agent is probably restarting all the time [02:23] julianwa, if that doesn't work.. then i'll have to fallback to the manual munge the db, which i'd rather avoid if possible, as its not supported. [02:23] davecheney, huh [02:23] we don't let the process die, 'cos that will cause upstart to put it in the sin bin [02:23] but the process does restart the worker frequently [02:24] davecheney, if that's the case then the persistent error here over days, would have already been resolved.. [02:24] hazmat: yes [02:24] so a restart is unliekly to fix the problem [02:24] but it's also unlikly to make it any worse [02:25] davecheney, upstart will restart as long as its not a persistent suicide [02:25] davecheney, how often does juju kill workers.. my understanding is that it only does it on error in the worker [02:25] ie. its not a periodic process [02:25] hazmat: depends on the error [02:25] with maas, it's quite frequent [02:26] with other providers, it tends to be less frequent [02:26] davecheney, ie.. if its not a periodic process... then no error.. means no restart [02:26] so a manual restart still has value [02:26] hazmat: sure [02:26] davecheney, is the scope of the worker all the environ jobs [02:26] bottom line [02:26] nobody knows what will fix it [02:26] but a restart can't hurt [02:26] indeed [02:27] and its better than touching juju's db by hand.. which is the next step.. so worth a shot [02:28] aha [02:28] hazmat: I'd highly recommend not hacking the juju db by hand [02:28] davecheney, thumper we have the underlying issue it looks like [02:28] jujud died perm [02:29] thumper, except when nesc :-) like migration [02:29] hazmat: only if you mean pyjuju -> go juju [02:30] 12:28 < hazmat> jujud died perm [02:30] ^ can you please expand on this [02:30] davecheney, its not running [02:30] what does the log file say ? [02:32] thumper, ack on recommend, and yes that's the migration i'm working on... in this case service is pretty innoucous though, it has no agent manipulating state. [02:37] an observers to trigger on txn, in fact its the lack of triggers there that forces the manual [02:38] hazmat: already restart jujud-machine-0 on bootstrap node [02:38] hazmat: nothing changed [02:38] julianwa, did it stay up.. ie is it in the ps aux | grep juju output [02:39] julianwa, could you copy/send /var/log/juju/machine-0.log for analysis [02:39] thumper: [02:39] 2013-10-01 01:03:32 DEBUG juju.rpc.jsoncodec codec.go:172 -> {"RequestId":9,"Response":{"Results":[{"StringsWatcherId":"4","Changes":null,"Error":null}]}} [02:39] 2013-10-01 01:03:32 DEBUG juju.worker.logger logger.go:45 reconfiguring logging from "=DEBUG" to "=WARNING" [02:39] ^ when did this land ? [02:39] my juju environment has stalled [02:39] it hasn't stalled [02:39] and I'm not getting any output from the agents because we log nothing at WARNING level [02:39] it just isn't giving info and debug [02:40] if you want, you should bootstrap with debug [02:40] ok, is it possible to change this ? [02:40] or read my big long email [02:40] ok, will re-read [02:40] juju set-env logging-config='=DEBUG' [02:40] on the running environment [02:40] will set it back [02:41] juju set-env logging-config='=INFO;juju=DEBUG;juju.rpc=INFO' will reduce noise [02:41] or even [02:41] juju set-env logging-config='juju=DEBUG;juju.rpc=INFO' will reduce noise [02:41] yummy [02:41] thumper, how bout a default of that [02:42] export JUJU_LOGGING_CONFIG='juju=DEBUG;juju.rpc=INFO' [02:42] thumper, what's the current default? [02:42] =WARNING [02:42] k [02:42] the logging config you bootstrap with is passed on [02:42] hazmat: thumper i'll do that [02:43] but i'm concerned that the debugging that I needed has probably been thrown away [02:43] i'll rebootstrap and try again [02:46] davecheney, hazmat: perhaps we should change the default logging level if you are using a debug release [02:46] as that is most likely to be developers [02:46] who are more interested in logs [02:46] well, debug info anyway [02:46] thumper: i think the default sohuld be export JUJU_LOGGING_CONFIG='juju=DEBUG;juju.rpc=INFO' [02:46] hazmat: machine-0.log.tar.gz under chinstrap:/home/julianwa [02:46] the main problem is the log spam from the rpc code [02:46] everything else is crucial [02:46] davecheney: alternatively, we set the rpc spam to TRACE [02:47] so DEBUG just works [02:47] sure [02:47] i don't care about the specificas [02:47] everyone hates the rpc spam [02:47] all i know is I have an environment that is broken [02:47] and i cannot debug it beucase the logging that i needed was discarded [02:48] all the agents are idle [02:48] but i have no logs to debug the problem [02:48] davecheney: I'll submit a fix tomorrow [02:48] * davecheney destroys environment and trys again [02:50] julianwa, thanks [02:52] julianwa, that's odd i get unexpected EOF extracting [02:52] and output cuts off mid line [02:53] but it doesn't look the agent was doing anything for a while [02:53] julianwa, when you restarted it, did the process stay up? [02:55] hazmat: FWIW, I'm seeing upstart issues starting upstart for the local provider now [02:55] it seems that upstart sees it very fast [02:55] and starts it before we call start [02:56] a race it seems between us going: are you running? no? ok lets start you, fail - already running [02:57] hazmat: what's you mean stay up? [02:58] hazmat: did you mean process still have same pid? [02:59] julianwa, sure [02:59] julianwa, i'm trying to verify its running [02:59] its unclear from the log why it died, just want to make sure that it is in fact alive [03:00] hazmat: yes. it's running with new pid [03:00] julianwa, good, and status still has mysql dying... [03:00] ah fark [03:00] right [03:00] julianwa, so unto manual db work.. [03:01] one moment [03:03] * thumper headdesks [03:06] thumper, do you have an alternative? [03:06] * thumper headdesks [03:06] * thumper headdesks [03:06] julianwa, are you using juju-deployer? [03:06] hazmat: no [03:06] k [03:08] hazmat: for some reason, my local provider is completely screwed [03:08] thumper, hmm.. works okay for me. what do you see? i still have a sleep in Install and a sudo in start [03:08] re upstart.go [03:09] telling me that the environment isn't up when I can see that it is [03:09] bootstrap worked [03:09] status says "you aren't bootstrapped" [03:10] thumper, would you mind pastebin status -v --debug [03:10] just curious what it looks like [03:10] hazmat: -v isn't needed if you use --debug [03:10] really [03:10] thumper, i used to assume that then someone told/showed me otherwise [03:11] no, it's bollocks [03:11] davecheney, wasn't that you? [03:11] not important [03:11] thumper, yeah.. --debug would be interesting to see [03:11] $ juju status --debug 2>&1 | pastebinit [03:11] http://paste.ubuntu.com/6178119/ [03:12] thumper, is your provider storage running?.. its basically failure to resolve the juju db i'd assume [03:13] your not even getting to state/open [03:13] not very helpful eh? [03:14] so env/conn is the failure point [03:14] well maybe.. [03:14] actually possibly before then [03:15] hmm... [03:15] for some reason it wasn't started [03:15] but it said it was [03:15] wtf [03:16] * hazmat returns to db surgery [03:16] * thumper is still confused [03:17] seems to be some weirdness there [03:37] hazmat: i did recommend -v always [03:37] but the meaning of -v has changed recently [03:38] it hasn't changed yet, but will one day soon [03:38] or perhaps it did way back, and I changed it [03:39] -v and --debug uses to be synonyms [03:39] not anymore [03:39] or somethign [03:40] julianwa, i need a sanity check before i give the modify db script.. could you run this http://paste.ubuntu.com/6178181/ from the client machine, you need to pass -e env_name and -o output_file.json .. and then copy the file over to chinstrap so i can verify what needs modification. [03:41] julianwa, you'll probably need to setup a virtualenv to get a recent version of mongodb if the client is precise [03:47] hazmat: setup a virtualenv on the server? [03:55] julianwa, on the client [03:55] sorry net dropped for bit, battery died [03:55] julianwa, virtualenv --system-site-packages jujufix && source jujufix/bin/activate && easy_install pymongo [03:56] then python script_from_pastebin -e env_name -o output.json [03:58] * thumper off to do a demo of juju [03:58] so I fully expect everything to fall in a heap [03:58] thumper, good luck [03:58] cheers [04:01] julianwa, that script is read only, it just grabs a copy of the db into a json [04:02] mostly so i can inspect and verify all refs to 'mysql' before doing an update script [04:11] we don't have a client to create virtualenv. the client is maas server. [04:11] hazmat: ^^ [04:11] julianwa, that's fine [04:12] julianwa, you can setup the virtualenv there [04:12] the virtualenv will keep the pymongo package isolated [04:12] from the system [04:12] you do need to install python-virtualenv package [04:24] ie. on the maas server [04:26] and then you can remove the virtualenv dir when the work is done [04:36] julianwa, sorry i just got told you may not have outbound net access [04:36] might have to do go for this.. [04:37] except not sure i have time for that tonight [04:46] julianwa, k.. i'm done for the night. can pick up via email. [04:49] hatch: thanks will exec script and get back to you [04:49] sorry hazmat ^^ [04:49] julianwa, no worries [07:41] mornin' all [07:41] morning [07:43] morning rogpeppe, dimitern [07:44] axw: hiya [07:44] axw: just glancing at q [07:45] axw: https://codereview.appspot.com/14197043 [07:45] axw: is the whole thing really on one line? [07:46] rogpeppe: yep, the base64 encoder doesn't add newlines if that's what you're thinking of [07:46] axw: hmm, that seems wrong - head should not have to read 10MB lines [07:47] rogpeppe: hmm yeah, I guess head is going to buffer all that isn't it. [07:47] axw: and... is there a pty at the other end of that ssh connection? [07:47] axw: indeed [07:47] rogpeppe: no pty allocated [07:47] axw: well at least we won't overflow the input buffer [07:48] rogpeppe: I can change it to a here doc. [07:48] hm [07:48] isn';t that going to be the same? [07:48] axw: that would be better, but for other reasons [07:50] rogpeppe: other reasons being? [07:50] axw: it's robust when the input doesn't happen to be exactly the right length [07:50] rogpeppe: this will always be one line [07:51] axw: currently yes, but i think it should probably be split [07:51] axw: so that it can be streamed all the way [07:51] axw: it's slightly odd that encoding/base64 doesn't provide an option to do that [07:52] rogpeppe: indeed, I thought there was a mandated line length for some usages of base64... [08:06] rogpeppe: I was thinking about https://codereview.appspot.com/14011046/ again before; I *think*, once everything's behind the API, we should be able to solve this cleanly by adding something into the environ config [08:07] i.e. a "bootstrapped" flag, which is checked to see if ssh/http storage is used [08:07] initially false (while bootstrapping), then set to true before adding into state [08:07] then the API server will always load it out as true [08:07] axw: i wondered about that [08:08] axw: i'm not sure though [08:08] axw: currently i assume that Bootstrap doesn't change any config options that need storing locally [08:10] axw: how about something like this for splitting lines (untested): http://paste.ubuntu.com/6178727/ ? [08:12] rogpeppe: sure, but then what on the server-side? head still buffers, as does a here doc [08:12] axw: no, a here doc doesn't buffer (and neither does head, other than line-at-a-time) [08:13] er yeah of course, line buffered.. :) ok, I thought the here doc was fully buffered [08:13] ok, I can make that change [08:15] axw: seems to work ok: http://play.golang.org/p/oYPyqeej5_ [08:24] fwereade, ping [08:25] dimitern, pong [08:25] axw: their unbuffered nature is almost the entire reason that heredocs exist (otherwise you could just use single quotes); think of shell archives. [08:26] fwereade, I updated that passwords CL, if you can take a look? https://codereview.appspot.com/14036045/ [08:26] dimitern, cheers [08:26] rogpeppe: thanks, I didn't think about that too hard [08:26] fwereade, ah, sorry didn't see you had comments recently - will look at them now [08:27] dimitern, they're basically a repriseof ourdiscussions, thought Ishould record them [08:28] fwereade, I couldn't find a way to access the password from the agent conf, except if there's a method on the interface [08:28] fwereade, but now at least, there's no newPassword and I do read it back to get the saved value [08:29] dimitern, hmm, I see, don't love it but I guess that's why we had that Password method in the first place [08:29] dimitern, do you think there's any way we can make its testiness clearer [08:30] dimitern, like if we added an unexported passwordForTests method to the interface, and then had an export_test func that called it? [08:31] fwereade, the problem is, I need it outside of the agent module, so export_test won't do [08:32] dimitern, jujud/bootstrap_test? [08:32] fwereade, also, originally the method was PasswordHash(), but that's not going to work in my case - I need the actual plain text password [08:32] dimitern, or more? [08:32] dimitern, yeah, I saw that, entirely unbothered there :) [08:32] fwereade, there and in machine_test, yeah [08:33] fwereade, hmm you know what? [08:33] fwereade, I do have OpenState and OpenAPI in config [08:33] dimitern, I don't see it in machine_test [08:34] fwereade, I can use these instead in assertCanConnectToState and the other [08:34] dimitern, hey, yeah, just load a config and check those work (or don't) as expected [08:34] fwereade, ok, and I'll revert to PasswordHash then [08:34] dimitern, and in bootstrap_test I think the try-to-load-a-config thing should work too [08:34] fwereade, you mean drop PasswordHash altogether? [08:34]