[00:00] we might be able to do a version check? [00:01] nop, that will not work [00:11] ah, we could only restrict API access once UpgradedToVersion >= 1.21 [00:36] anyone able to help me with a juju bootstrap issue? I've got a maas+juju environment setup where when I bootstrap it, all I see in the debug logs is a endless loop of trying to ssh into the bootstrap node, even though I can connect manually [00:52] menn0: ping [00:58] interesting, before destroying the instance, it complains about /var/lib/juju/nonce.txt does not exist [01:00] this is looking like LP#1314682 [01:01] waigani: pong [01:01] heyhe [01:01] hah! creating /var/lib/juju/nonce.txt and things start moving forward [01:02] menn0: so I'm thinking splitting the migration over two versions is probably best [01:03] waigani: not restricting the API while upgrades are in progress is doable [01:03] menn0: I looked into a few workarounds, but they add a whole bunch of kluge [01:03] ok ppl gotta go sleepto be up early in the AM, cheers [01:03] l8r perrito666 [01:03] cu all tomorrow AM [01:03] menn0: yeah, so that is what i got halfway into coding but ... [01:04] menn0: it adds boilerplate that we could avoid if we simply add restricted logins in the next version [01:05] waigani: true. what is the downside of not restricting logins [01:06] menn0: it means that MESS is not complete [01:06] that is all [01:06] so it would be one step at a time towards MESS [01:07] maybe let's do that for now and check with thumper tomorrow [01:07] menn0: but 1.22 could have restricted API and we are back on track [01:07] it's the least invasive thing to do [01:08] so removing that check fixes the issue? [01:08] menn0: I'm just testing that now.. [01:08] what's supposed to create the /var/lib/juju/nonce.txt on a juju unit? [01:08] waigani: sweet [01:08] waigani: I've got to get some lunch ready for the fam. back in a bit. [01:08] menn0: yep, np [01:09] waigani: I assigned the bug to myself earlier on this morning. you should grab it. [01:09] ok [01:14] bradm: just saw your questions - the juju bootstrap process sets up a cloud init script which creates the nonce [01:14] bootstrap process = start instance [01:15] it you look in /var/log/cloud-init-output.log you should see the bash command to echo to the nonce.txt file [01:16] wallyworld: ok, let me rebootstrap it and have a look [01:16] wallyworld: if I create the file by hand, things start moving again, but errors out when it tries to start juju-db [01:17] bradm: that implies the cloud init script has not been run [01:18] wallyworld: this is a maas setup, not sure if that means anything [01:18] bradm: yeah, we've just seen this on maas [01:19] still got the nonce problem? [01:19] cause the node is acquired and doesn't run cloud init [01:19] bigjools: bradm does [01:19] old version? [01:19] LP#1314682 sounds vaguely familiar with the symptoms [01:19] bigjools: of maas? its 1.5.2+bzr2282-0 [01:20] no it's a fix in maas's clients [01:20] they supply the nonce to the API [01:20] that bug concluded that the node which was acquired didn't run cloud init [01:20] might be wrong? [01:20] this is kinda important to fix, its for bootstack. [01:21] did you find any errors in the logs? [01:21] ok, so my bootstrap node is back up, lets see what it has [01:22] 2014-09-18 01:19:35,042 - util.py[WARNING]: Failed running /var/lib/cloud/instance/scripts/part-001 [3] [01:22] 2014-09-18 01:19:35,045 - cc_scripts_user.py[WARNING]: Failed to run module scripts-user (scripts in /var/lib/cloud/instance/scripts) [01:22] 2014-09-18 01:19:35,045 - util.py[WARNING]: Running scripts-user () failed [01:22] that doesn't look good [01:23] thats some custom scripts we have to create the nova instance dirs if it has a /dev/sdb [01:23] let me knobble that for now. [01:24] do failures there make the whole cloud init script bail out? [01:25] looks like it must be [01:25] I'll comment these out and rebootstrap [01:25] it has some checks to see if the device is there or not, I can't see why its failing immediately though [01:33] ok, the cloud init scripts seem to have finished properly, although I now can't ping the host [01:33] but I can see it creating the nonce.txt, so there's something else going on here [01:33] progress is good, thanks guys. [01:35] must be something wrong in the preseeds [01:43] waigani: where are things now? can I help with anything? [01:43] menn0: I'm currently hunting down where the "environemnt not found" err is coming from [01:43] ah, what, it does a ifdown eth0? what the? [01:45] and indeed, thats what breaks it, if I do a ifup eth0 things start working [01:47] wallyworld, bigjools: any idea why cloud-init is doing a ifdown eth0? and what I can do to stop it? :) [01:48] dunno, I thought it only did what the user data told it to do [01:48] bradm: that was done for maas to set up networking, but it can be disabled [01:48] let me check [01:49] I logged into the host and did a ifup eth0, I now have a successfully bootstrapped juju env \o/ [01:49] so if I can stop it doing that, we're in much better shape. [01:50] bradm: disable-network-management [01:50] set that to true in your yaml [01:50] LP#1341524 [01:50] but be aware that you may ten need to do stuff by hand that juju otherwise would try to do [01:50] bug 1341524 [01:50] Bug #1341524: juju/MAAS deployed host with bonding configured via preseed missing eth0 from bond on first boot

[01:51] bradm: what version of juju are you using? [01:51] wallyworld: 1.20.7, apparently. [01:51] cause the fix is only in 1.20.8 [01:52] which is not formally released yet [01:52] aha! [01:52] but will be available for testing tomorrow [01:52] and released early next [01:52] this is a staging environment for bootstack, we can use whatever version of juju that is appropriate [01:52] bradm: you can try 1.21-aplha1 [01:52] that is released [01:53] and has the fix also [01:53] email was sent to canonical-juju [01:53] with the details [01:53] I suspect I'm not on that list. [01:53] you'll need to use a custom tools-metadata-url [01:53] ok, i can forward the email [01:53] it was also sent to juju-dev i think [01:54] aha, found a juju 1.21-alpha1 release email [01:54] sent to juju@lists.u.c [01:54] good :-) [01:55] ooh, --keep-broken, that sounds awesome [01:55] ha [01:56] bigjools: allows for debugging when maas breaks :-P [01:56] wallyworld: so to confirm, that disable-network-management: true is only in the still to be released 1.20.8, or the currently released 1.21-alpha1? [01:57] bradm: it's in both those releses [01:57] wallyworld: excellent. [01:57] 1.21 is here now to try [01:57] wallyworld: ok, I can probably test out 1.21-alpha1 in staging, but we'll want to stick with stable in the long run [01:57] right, I can try that out now [01:57] bradm: agreed, this ws just a way of letting you get further [01:57] wallyworld: yup, perfect. [01:58] wallyworld: and this staging env is for exactly this. [01:58] \o/ [01:58] although I wonder how we got the other env up [01:58] nfi :-) [01:58] it is 1.18 I think [01:58] ah [01:58] explains it [01:59] a lot of new network stuff has been done since then [01:59] ev'll get upset if I break it ;) [01:59] yep :-) [02:01] menn0: waigani: where's Tum (sic) today? [02:01] wallyworld: being an 'entrepreneurial innovator' [02:02] oh, sounds serious [02:02] wallyworld: some workshop thing that he took the day off for [02:02] and didn't mark the calendar [02:02] haha [02:02] or reschedule his 1:1 with me [02:02] oh, that sucks [02:03] not really, i didn't want to talk to him anyway [02:03] don't need him [02:03] haha === fuzzy is now known as Fuzai [02:09] wallyworld: I can't imagine there'd be any bad side effects if I upgrade to 1.21, do a test deploy and then destroy and redeploy to 1.20.8 when its out, right? === Fuzai is now known as Ponyo [02:10] bradm: that should work [02:10] wallyworld: sounds like I know what I'm doing after lunch then. [02:11] wallyworld: thanks for the help [02:11] np :-) [02:45] bradm: hang on, id you say downgrade ? [02:45] i'm not sure what you mean when you say redeploy to $OLDER VERSION [03:01] Hey, does anyone know if there is a max character length for machine names? [03:04] davecheney: juju destroy-environment ; downgrade to 1.20.8 ; juju bootstrap [03:08] bradm: ok, that is fine [03:09] davecheney: currently this is all about refining the process, and its staging, so its not the end of the world to redeploy - wouldn't want to do it all the time though. [03:24] wallyworld: fwiw, the juju bootstrap just worked with 1.21-alpha1 and the tweaks to the environment. [03:27] bradm: awesome, great [03:27] huwshimi: juju doesn't impose one explicitly [03:28] wallyworld: OK, thanks [03:36] wallyworld: juju lets you choose machine names ? [03:36] no [03:37] wallyworld: then you didn't really answer huwshimi 's question :) [03:37] i just meant that juju has no length limit anywhere [03:37] what the providers do is up to them [03:37] wallyworld: the machine names will always be integers [03:37] yes ? [03:37] the instance id's are opaque blobs [03:38] well, we print out the int id now, but don't have to [03:38] the point is there's no char length encoded anywhere [03:38] nope, that is true [03:38] which was the point of the question, presumably from a ui perspective [03:39] ok [05:19] an update on the CI blocker (bug 1370635) [05:19] Bug #1370635: Unable to connect to environment after local upgrade on precise

[05:20] I have the cause figured out but the solution isn't clear yet [05:20] details on the bug [06:06] wallyworld_: can you please review https://github.com/juju/juju/pull/784 for me? I've refactored state.Storage, which should make the other branch a bit tidier [06:06] sure === uru_ is now known as urulama [06:21] axw: +1 with a suggestion, i gotta do scholl pickup, bbiab [06:23] wallyworld_: thanks, cya [06:37] what the? there's .epm files and .conf files in /var/log/juju with juju 1.21-alpha1 ? [06:37] er, .pem even. [06:39] davecheney, hey, still around? [06:39] wallyworld_: can't update RB #51 for some reason, I get an internal server error when trying to publish [06:39] I've fixed the issues though [06:43] filed LP#1370896 [06:46] axw: except landing is blocked \o/ [06:46] yeah :| [06:46] bradm: well that kinda sucks, that should be fixed. btw if you type bug 1370896 a magic fairy will print some extra info [06:47] Bug #1370896: juju 1.21-alpha1 has conf files in /var/log/juju on instances [06:48] bradm: those .pem files have been there for ages; logrotate.conf is new [06:48] but why in the log dir? [06:48] * axw shrugs [06:48] the .pem file is for rsyslogd [06:48] for SSL [06:48] * wallyworld_ thinks it needs fixing anyway [06:49] axw: just because its been there for ages doesn't make it right. [06:49] bradm: right, sorry, I thought you thought it was new [06:49] I just mentioned them both in the bug for completeness [06:49] okey dokey [06:49] anyway, not a big deal, just throwing out bugs for issues as I find them. [06:54] axw: i may have old code, but i noticed there's a test called TestFindToolsInControlBucket, which is now obsolete. it uses a test helper UploadToStorage(c, s.env.Storage()...). these can go away [06:54] wallyworld_: sure, I'll take a look... [06:54] ta, or i can delete them as a drive by [06:55] in the tools-stream branch [06:55] wallyworld_: nah can't go yet, not till we have the upgrade-step [06:55] wallyworld_: I've yet to write an upgrade step that moves everything from provider to environment storage [06:55] maybe i delete the Find test [06:56] cause i don't want to spend time making it work with tools-stream [06:56] okay [07:05] axw, wallyworld_: in case anyone asks, I'm stopping for a bit but will pick up with CI blocker later tonight [07:05] I haven't done enough hours today due to Real Life stuff [07:22] menn0: no worries, thanks [07:23] axw, ericsnow, http://reviews.vapour.ws/r/33/ please take a look, I really need to land this already :) [07:23] dimitern: looking [07:23] axw, thanks [07:27] morning all [07:33] is launchpad down of anyone else? [07:33] for* [08:05] morning all [08:08] morning all [08:13] morning voidspace [08:13] dimitern: did you see https://bugs.launchpad.net/juju-core/+bug/1308146 [08:13] Bug #1308146: not okForStorage error when deploying local charm

[08:13] I'm wondering if this has to do with something we're writing to the DB to cache information about local changes [08:13] cmarms [08:13] charms [08:13] vs, what we do for cs: charms [08:14] morning voidspace [08:14] jam, not sure, I have to look [08:14] dimitern: you did the stuff for PutCharm to support local charms, and for the GUI to be able to get its stuff, right? [08:14] jam, that's right [08:15] but it's been a while [08:17] dimitern: sure. Just the thought comes to might that it is probably the only thing that is directly different between local and cs charms [08:17] and involves possibly writing charm config keys to the db [08:18] jam, what I didn't get from the bug is whether the problematic charm was deployed from a local repo or from cs: [08:21] dimitern: from local:, from "cs:" doesn't have the problem [08:21] hence why I think it is the caching logic [08:22] "It's not exactly clear from the bug log when this regression occured, [08:22] and the bug suggests it may only affect local charms." [08:24] jam, right, I saw that later, sorry [08:24] jam, I'm looking through the code, it seems this happens in the last step - state.UpdateUploadedCharm() [08:25] jam, I see the problem [08:26] jam, we do replace . and $ in service settings when saving to state, but we don't do the same for the charms collection (and its config field) [08:31] jam, and it seems we never did [08:39] jam, yep, i'm afraid it looks like a "my bad" case :) I'll assign it to myself and propose a fix [08:39] dimitern: it sounds like it is just "need to escape", yeah [08:39] dimitern: great [08:47] axw: ping [08:48] voidspace: pong [08:48] axw: thanks for your review of my PR [08:48] you're welcome [08:48] axw: you commented that the "cmd" variable isn't needed any more in applyReplSetconfig [08:48] axw: it's used twice (for logging) - and is passed into applyReplSetConfig [08:49] ah, it's input [08:49] my bad [08:49] sorry, glanced over that [08:49] yeah, it looks kinda like that [08:49] resetting it in the loop was always pointless [08:49] that loop just didn't do what it looked like it was doing [08:50] which is why I fixed it [08:50] axw: anyway, cool - just checking you didn't have some other reason I missed [08:50] nope, thanks though [08:50] axw: so I'll add those doc comments and merge [08:50] thanks! [08:52] wallyworld_: I think we were meant to discuss charmstores and placement and things today - probably a bit late now [09:02] dimitern: escaping sounds like something we could backport to 1.20 [09:02] and since local charms never worked there anyway, we don't have to worry about compat for upgrade [09:04] morning [09:04] morning TheMue [09:05] dimitern, jam: I think I got a good approach for testing now. take a look at http://paste.ubuntu.com/8370850/ [09:05] the upper files simulate the different versions of an agent here [09:06] and then there are files for testing each version [09:07] instead of embedding a type I'm working with a simple struct for needed infos and versioned non-public test funcs containing the tests on anonymous facades [09:08] so if tests are added or changed in later versions the according test helper funcs can be added in those files too (and used until they are changed in future again) [09:08] the examples contain adding, changing and dropping a facade function [09:08] jam, sgtm [09:09] TheMue, will look shortly, thanks [09:10] dimitern: thanks, it's mostly based on your approach :D [09:19] TheMue: so that still makes it fairly easy to accidentally leave off TestSecondFuncOK (like you did in agent_v2_test.go) [09:19] now given your change to SecondFunc() it may be that you intended for it t odie [09:19] to die [09:19] using the "embed the earlier version but make it an invalid signature" rule [09:19] jam: this risk still exists, yes [09:20] Which I think fwereade and I thought was a bit too magical [09:21] jam: it is, because the public name still exists. did I missed a better idea for hiding it? [09:23] jam: surely we could do it in a similar way like I've done for the tests here: private versioned implementations which are only called in each new facade [09:23] TheMue: so in our API docs discussion, the idea was if you wanted something to disappear you have to factor out a common base and embed [09:23] jam: so a V2 dropping a function simply doesn't have it [09:25] jam: I've got my problem with the embedding, or, at least even here the methods have to be versioned. [09:25] jam: because we could have a doThisV0 valid until V3, but need a doThisV3 in V3 and higher [09:27] jam: implementing it as functions using one shared type would make it simpler (or at least more logical) to implement the v0 login in the v0 file, reusing it until v2 and add a new implementation of v3 in the v3 file [09:28] jam: so the implementation is more where it is expected than in one large file containing the base with all versioned changes [09:29] dimitern: btw, your idea of using a facade with only the interesting function is cool [09:30] TheMue, yeah? It could be written a bit more concise, if we define the interface before the test case, like type lifer interface { Life() params.Life } .. testLife(c *gc.C, facade lifer) [09:36] wwitzel3: a simpler patch for you: http://reviews.vapour.ws/r/54/ [09:36] I didn't see how to do the dependent-diff with Review Board and make my change depend on yours [09:39] wwitzel3: well, I guess I did find something, but reviewboard tells me: The file "utils/syslog/config.go" (revision e1eaf4a) was not found in the repository [09:40] dimitern: yes, the only problem then would be to be creative enough for the name *lol* [09:42] dimitern: but I now like your approach more than my first one. it's only important that the tests themself are external and versioned, so that they can be reused and changed over time [09:44] TheMue, yeah, that was my idea as well [09:44] jam, thanks for the review - one of the issues is already fixed earlier (the NewFacadeCaller calling NewFacadeCallerForVersion internally) [09:50] dimitern: yeah, I had written that up yesterday, but I keep hitting "Save" instead of "Publish" when doing reviews. [09:51] jam, axw, ericsnow, http://reviews.vapour.ws/r/55/ - fixes bug 1308146 [09:51] Bug #1308146: not okForStorage error when deploying local charm

[09:52] just a reminder, whole team meeting in 8min [09:52] dimitern: I haven't looked yet, but does it seem feasible to backport to 1.20? [09:55] jam, yes, changes are minimal [09:58] phew! I have a fix for the CI blocker [09:58] https://plus.google.com/hangouts/_/canonical.com/juju-core-team?authuser=1 [09:58] nice menn0 [09:58] it might not be the final fix we end up going with but it solves the issue for now [09:58] I will take it up with thumper tomorrow (it's his work that caused it) [09:59] but I don't blame him [09:59] it was head-hurtingly hard to figure this out [09:59] PR soon [10:00] menn0: dimitern: TheMue: voidspace: team meeting [10:00] jam: did you mean me? [10:01] menn0: well, it is the "entire juju team weekly meeting" , so yeah [10:01] menn0: you might not usually come to it on this week [10:01] dimitern: I was afk, will review after the meeting [10:02] axw, cheers [10:04] jam: omw [10:05] hmmm... currently failing to join [10:34] jam: that is unfair, I could do sleep even though my timezone says work [10:34] jam: did you set the parent branch in rbt after the fact? [10:34] perrito666: I'll let you pick whichever one you want [10:34] or can you only do it on first post [10:34] dimitern: are you there? [10:34] axw: yeah, I was trying to play around it [10:34] with it [10:35] rbt post -r --parent eventually worked [10:35] dimitern: we could do a quick standup now [10:35] voidspace, yep [10:35] jam: ah, thanks [10:35] once I rebased everything because git couldn't handle what I wanted [10:35] voidspace, ah, yes actually, brt [10:35] ah I was doing -u, that's why it wasn't working [10:36] axw: interesting, because it was trying to match up based on the summary? [10:36] but your summary is different because you're including different revs (presumably) [10:36] axw: I think my problem is that I was trying to merge master, but rbt post can't handle that [10:36] jam: actually no, I'm just a twit: I did -u [10:36] you have to have master in your parent branch [10:36] ah [10:36] :) [10:49] axw or wallyworld_: http://reviews.vapour.ws/r/56/ [10:50] looking [10:53] folks, how can I use mongo shell to connect to mongo on a live state server? [10:53] menn0: +1 with a suggestion, thanks for fixing [10:53] mattyw: local provider? [10:53] ^^ specifiacally looks like I need to login somehow [10:53] menn0: you managed to test it? [10:53] wallyworld_: yep [10:53] \o/ [10:53] menn0, aws - but I suppose I could use local provider which might make things easier [10:53] I created a script to repro [10:54] and used git bisect to automatically find the problem reb [10:54] mattyw: there's a plug from kapil [10:54] rev [10:54] juju-db [10:54] mattyw: I just asked because the answer is slightly different depending on the provider [10:54] * menn0 looks up the command line [10:54] mattyw: https://github.com/kapilt/juju-dbinspect [10:54] works on any provider afaik [10:55] mattyw: mongo 127.0.0.1:37017/admin --ssl --username "admin" --password "`sudo grep oldpassword /var/lib/juju/agents/machine-*/agent.conf | cut -d' ' -f2`" [10:55] wallyworld_, menn0 thanks very much - I'll try everything until I get what I want :) [10:55] mattyw: i've used the db-inspec tool - very niiiice [10:55] menn0, that's a command worth hanging on to [10:56] it essentially automates what menn0 typed [10:56] plus gives some standard commands [10:56] :) [10:56] mattyw: you'll need to sudo apt-get install mongodb-clients first [10:56] mattyw: the path to the agent.conf is different for the local provider, hence the difference [10:56] or use the plugin :) [10:57] menn0, thanks for your help - I've found what I was looking for [10:57] menn0, metrics have arrived from a deployed unit \o/ [10:57] \o/ [11:01] wallyworld_: is this ok? [11:01] logger.Warningf("state servers info has no environment UUID so retrieving it from environment state") [11:01] menn0: +1 thank you [11:01] TheMue: sorry frank, I cut you off! [11:01] TheMue: I will enjoy it all I hope [11:02] TheMue: except maybe giving the talk - still practising === fabrice is now known as fabrice|lunch [11:05] right, the CI blocker fix is testing now but I really need to go to bed [11:06] can someone please keep an eye on the merge and if it's successful mark bug 1370635 as "Fix Committed"? [11:06] Bug #1370635: Unable to connect to environment after local upgrade on precise