[00:00] admin-secret is needed in environments.yaml [00:00] thumper: nope, bootstrap complained that the secret was missing, didn't finish, so i had to add it [00:00] the problem seems to be the "fatal "machiner": unauthorized" [00:00] * thumper takes the dog out [00:04] sidnei: I got it working here, had to make two changes [00:04] sidnei: haven't seen that fatal error [00:05] I have revision 1481 [00:06] sidnei: is that during bootstrap? [00:06] ahasenack: nope bootstrap finishes successfully, juju status returns pending for the bootstrap node, tailing ~/.juju/local/log/machine-0.log i see a retry every 3s with that unauthorized [00:07] sidnei: did you bootstrap with -v? [00:07] ahasenack: y [00:07] sidnei: so you saw a bunch of "connection refused", and then eventually it worked? [00:07] ahasenack: only 2 [00:07] really? I see dozens [00:07] it retries several times per second [00:07] ahasenack: https://pastebin.canonical.com/94599/ [00:08] ahasenack: get an ssd? :) [00:08] impressive [00:08] sidnei: so juju status should work right after that, and that's where you have the problem then [00:09] yup, status says it's pending [00:09] https://pastebin.canonical.com/94600/ [00:09] sidnei: what about mongodb, you installed the one from raring? [00:09] ahasenack: im running raring [00:09] https://pastebin.canonical.com/94601/ [00:10] sidnei: right, but you have the raring package? [00:10] sidnei: I'm not on to something, I'm just comparing notes [00:10] ahasenack: sorry, duh. im on saucy not raring [00:10] oh, ok [00:10] $ apt-cache policy mongodb [00:10] mongodb: [00:10] Installed: (none) [00:10] Candidate: 1:2.4.3-1ubuntu1 [00:10] that's different :) [00:11] I have mongo 2.2.4 [00:11] yet there's a mongod running [00:11] I don't know if a new major mongo db version could cause this [00:11] ah, mongodbserver [00:11] could be worth asking around [00:11] i guess davecheney would know [00:12] wait, installed none? [00:12] Installed: 1:2.4.3-1ubuntu1 [00:12] mongodb-server is the package name, not mongodb [00:12] so that changed [00:12] mongodb 1:2.2.4-0ubuntu1 amd64 [00:12] it's what I have [00:12] and, well, also mongodb-server [00:12] mongodb is just a meta package probably [00:13] maybe version 2.4 has a new acl or something [00:13] Security Improvements [00:13] New Modular Authentication System with Support for Kerberos [00:14] fun [00:14] they talk about mongodb enterprise [00:14] not sure if that's what we have [00:14] but this sounds relevant [00:14] Role Based Access Control and New Privilege Documents [00:14] MongoDB 2.4 introduces a role based access control system that provides more granular privileges to MongoDB users. See User Privilege Roles in MongoDB for more information. [00:14] (from http://docs.mongodb.org/manual/release-notes/2.4/ btw) [00:15] ah that'd probably be it [00:16] we need the ssl enabled mongo [00:16] ssl is enabled [00:16] even in sidnei's case, or else bootstrap wouldn't have finished/worked [00:16] it clearly didn't work fully, but at the bootstrap stage it didn't get any errors [00:17] indeed [00:20] ahasenack: please for the love of $DEITY, don't open the 'lets use a new version of mongodb' can of worms [00:21] davecheney: why would I do that? :) [00:21] sidnei is the one using a newer version :) [00:21] but looks like you have until october to get it working with mongo 2.4 ;) [00:21] since it's in saucy [00:22] ah, here's some info in syslog: https://pastebin.canonical.com/94602/ [00:22] davecheney: indeed, saucy has 2.4.3 [00:23] sidnei: that's javascript, right? [00:23] no idea. reminds me of couchdb though *wink* [00:23] sidnei: [00:23] "With authentication enabled, eval will fail during the operation if you do not have the permission to perform a specified task. [00:23] Changed in version 2.4: You must have full admin access to run." [00:24] from http://docs.mongodb.org/manual/reference/command/eval/ [00:24] is that just to get the date from mongo? [00:24] sidnei: yup, and P, Q, and R have something different [00:24] oh, lovely [00:25] davecheney: so does go [00:25] ahasenack: yup, lets not light two fires at once [00:34] hopefully it'll be a trivial fix. not looking forward to downgrading :) [00:38] sidnei: I was just poking around, maybe this could be used instead of javascript and eval: http://docs.mongodb.org/manual/reference/method/Date/ [00:38] sidnei: but I don't know how to login on mongo yet, I tried "mongo" with username and key I found in /var/log/syslog, but didn't work [00:39] i wanted to see what that output looks like, then hack it in state/presence/presence.go in the clockDelta() thing, which is what calls the eval [00:46] ok, gotta go [00:46] cya [01:23] thumper, future optimization for local provider.. serge & smoser were discussing being able to use lxc-clone with the cloud-image, ie re run cloudinit post clone, it will make lxc provisioning significantly faster, just a copy or fs snapshot (lvm/btrfs). [01:24] hazmat: let's get it working simply first, no? [01:26] thumper, hence the future label. it seems to be working pretty well so far. [01:27] although i'd note bootstrap via sudo -E for folks using JUJU_HOME/JUJU_ENV [01:28] wut? [01:28] hmm.. [01:28] oh yeah [01:28] sudo strips env vars without -E [01:28] HOME is passed through [01:28] but others aren't [01:29] hmm... I wonder if sudo -E would pick up the right juju for me [01:29] that would make things a little easier [01:35] thumper, one other random.. --force-machine to 0 on deploy.. might do unexpected things, i got a series error just due to mismatch there, but wasn't sure if there was a sanity check against workloads on 0 with local if series did match [01:35] hazmat: machine 0 doesn't support units [01:35] hazmat: so would get a different error later [01:39] cool [01:44] wallyworld__: ping [01:44] hi [01:44] wallyworld__, davecheney: ? https://codereview.appspot.com/11455044 [01:44] wallyworld__: hangout? [01:45] sure [01:45] https://plus.google.com/hangouts/_/36d42d48a7f8998aa3789caa0e0e93bead3afb24?hl=en [01:45] wallyworld__, davecheney: also two others pending reviews for minor things [01:59] davecheney: would love some +1s :) [02:00] thumper: looking [02:01] ta [02:01] davecheney: one is to add the admin-secret [02:01] davecheney: the other is a small pipeline, one to add gc and jc prefixes instead of importing into local namespace [02:01] followed by a branch to make containers auto restart [02:02] thumper: why the \n in the maas provider ? [02:03] davecheney: because I noticed that when I went juju init [02:03] there wasn't a gap between maas and local [02:03] and all the others did [02:04] davecheney: perhaps we should change the default provider in the init to be local? [02:04] although I'd like it to mature a bit more [02:04] get some more real world testing first [02:05] heh, didn't notice that marco did this too [02:06] thumper: sounds like the tests don't care [02:30] davecheney: https://codereview.appspot.com/11491043/ maybe? [02:44] thumper: LGTM [02:47] davecheney: ta [02:54] davecheney: regardless of the upgrade to 2.4, which may or not be a side effect, why is an eval used to get the server time? seems like there's some ill side-effects of that: http://docs.mongodb.org/manual/reference/command/eval/ "By default, eval takes a global write lock before evaluating the JavaScript function. As a result, eval blocks all other read and write operations to the database while the eval operation runs." [02:55] and then there's "Changed in version 2.4: You must have full admin access to run." (re: eval) [02:55] so maybe that's what broke indeed [03:03] sidnei: not sure [03:03] i don't konw that part very well [03:03] is it part of our transactoin magic ? [03:03] davecheney: nope, it's in clockDelta() in presence [03:04] sidnei: that is both facepalm and totally correct [03:04] you can't rely on the time inside a vm being correcet [03:04] shit [03:04] you can't rely on real hardware keeping good time [03:05] so the only choice is to use the time on the db server [03:05] at least it maybe consistent with itself [03:05] yeah, the question is if there's a way to get the time withot eval. seems that from 2.2 on you could get isMaster().localTime [03:06] and there's also serverStatus.localTime [03:06] * davecheney waves hands [03:06] if it 'aint broken, something something, profit ! [03:06] eh [03:07] ok, battery is flat [03:07] time for lunch [03:18] alright, commented out clockDelta body, hardcoded a return 0, nil. status is now started, and deploy kicked off [03:19] * sidnei files bug [03:31] https://bugs.launchpad.net/juju-core/+bug/1202480 [03:31] <_mup_> Bug #1202480: raring: bootstrap local environment works, machiner gets unauthorized [03:33] https://bugs.launchpad.net/juju-core/+bug/1202481 is a funny one [03:33] <_mup_> Bug #1202481: oddly formatted directory name in deployer === thumper is now known as thumper-afk [03:34] * thumper-afk back for meetings (lotsa meetings) [03:34] in about 4 hours [04:20] fwereade_: if you happen to be around, https://codereview.appspot.com/11501043 [04:21] * sidnei eods === tasdomas_afk is now known as tasdomas [07:29] can everyone please familiarise themselves with the agenda for tonights call [07:29] https://docs.google.com/a/canonical.com/document/d/1eeHzbtyt_4dlKQMof-vRfplMWMrClBx32k6BFI-77MI/edit# [07:35] davecheney: doing so [07:37] davecheney: i don't see go 1.0.3 mentioned on that list - i'm using it and everything builds ok and ec2 works === thumper-afk is now known as thumper [07:40] not sure about azure though [07:40] dimitern: good point [07:40] it's not available in PPA [07:40] but as we're grasping at straws [07:40] yeah [07:41] dimitern: added [07:41] cheers [07:50] good morning to you europeans [07:51] davecheney: do the go-curl developers know that it is broken under go 1.1? [07:59] thumper: yes, see agenda [08:34] * thumper looks for mgz [08:34] thumper: ping [08:34] TheMue: hey [08:34] TheMue: so... all my changes around tools are in trunk now [08:34] thumper: sent you a mail regarding the autosync stuff [08:35] yeah, I've not looked at the doc sorry [08:35] thumper: yep, already analyzing them [08:35] thumper: np [08:36] thumper: it's just the place where i note my findings [08:36] thumper: so if you have anything to add feel free [08:36] ok... [08:36] thumper: thx [08:36] TheMue: AFAIK, the major point is to make it easy for prod-stack [08:37] if we do that, we fix a lot of issues [08:38] thumper: we've got a first solution with sync-tools --source /my/path/to/tools [08:38] thumper: but that's not enough and convenient [08:39] TheMue: but anyone can do upload tools now [08:39] TheMue: if it grabs the binary [08:39] no build env needed any more [08:39] thumper: exactly [08:39] TheMue: here is an interesting idea [08:39] ... [08:39] * TheMue listens [08:40] * thumper thinks [08:40] ;) [08:40] hmm... [08:40] not as easy [08:40] the problem isn't the cli [08:40] but the bootstrap node [08:40] a setting maybe? [08:40] in the config [08:41] thought about that too [08:41] to say whether we should auto upload on bootstrap [08:41] making it explicit helps, otherwise the detection of the need is not simple [08:42] that could help [08:42] it's semi-automatic [08:44] beside the --source for sync-tools I already thought about a --source for bootstrap [08:46] I think an explicit setting saying "this cloud can't see remote tools", plz upload [08:46] don't guess [08:46] then it is a one off config item [08:46] we would need the sync still [08:46] to do an upgrade, no? [08:46] or does upgrade do an --upload-tools too? [08:46] or probably should [08:47] * thumper goes to have a cuppa tea and watch arrow [08:47] back to chat to mgz later === thumper is now known as thumper-afk [08:50] upgrade afaik only on demand [08:51] thumper-afk: thx for sharing your thoughts [08:55] hazmat: Hi! I'm trying to test a deployment, and it's hanging here: http://bazaar.launchpad.net/~hazmat/juju-deployer/refactor/view/head:/deployer.py#L1142 [08:56] hazmat: afaics, that loop never ends (my deployment times out in that loop), and it's checking for a non-existent key (the unit has 'agent-state', but not 'state'). [08:59] (well, only ends via timeout - anyway, I'm hacking around it atm, but wasn't sure if it was a WIP - in which case, sidnei, I'm guessing we shouldn't be using it yet). [09:29] anyone here used juju with azure? [09:29] i was just trying to set it up and it's not obvious to me where to find the various pieces of management info [09:34] rvba: ^ [09:35] rogpeppe: sure, I can help you with that, just one sec… [09:36] rvba: actually, i've just found some of the info by scrolling down and finding "Settings" [09:36] rogpeppe: I've written a small tutorial… let me forward it to you. [09:36] rvba: ta [09:53] fwereade_: it worked! s.BackingState.Sync() + a few modifications, I even reverted the in := w.in stuff and the flip-flopping to make sure with the fix the test pass and without it doesn't, i.e. the events are not lost [09:53] dimitern, sweet! [09:57] looks to me that azure-in-all.go did in fact land btw, so we will want to revert that for release [10:00] rvba: where do you get the azure command from? [10:01] rvba: or... perhaps i don't need to run it. [10:01] rogpeppe: you don't need to run it indeed. [10:01] fwereade_: https://codereview.appspot.com/11506043 - others as well? this fixes the StringsWatcher [10:03] rvba: where do i get the cert for the management-certificate-path entry in environments.yaml? [10:04] rogpeppe: you can just generate it with: http://paste.ubuntu.com/5887043/ [10:05] Then upload the public part to Azure. [10:05] rvba: hmm, might be nice if that was in the instructions [10:05] Settings > Management certificates [10:05] rvba: the public part being the .cer file, presumably? [10:06] rogpeppe: yes [10:06] rogpeppe: that's right… these instructions I've sent you are a draft really, they were meant for our team which has already setup an Azure account with a certificate. We need to polish that up for general consumption. [10:07] rvba: yeah, i've been making some notes about places that tripped me up which i'll pass on to you [10:07] ta === thumper-afk is now known as thumper [10:09] mgz: ping [10:10] thumper: hey, sorry I missed you earlier, forgot to get into irc after rebooting back to ubuntu [10:10] mgz: yeah, got time to chat? [10:10] hangout? [10:10] er... [10:10] or just irc? [10:10] that would mean rebooting again :) [10:10] you can't do hangouts in ubuntu? [10:10] mgz: dude, get more computers \o/ [10:11] mgz: does mumble work in ubuntu? [10:11] I have many, not all of which are that portable :) [10:11] or skype? [10:11] mumble is good [10:11] * thumper fires up mumble [10:11] is canonical still running the mumble server? [10:12] I think I just need to hop... [10:12] ^yeah, details are on the internal wiki [10:12] it eventually connected [10:12] * thumper has to remember what his push to talk button was [10:13] * thumper is in the cloud engineering kitchen [10:14] rvba: what about the storage account key? [10:14] talking thing isn't working [10:14] * thumper pokes some more [10:14] audio is such fun [10:15] rogpeppe: once you've created the storage account, click on it, then click on "manage access keys" at the bottom of the screen to get the key. [10:15] rvba: ah, thanks - that menu at the bottom is good for evading the eyes :-) [10:15] rogpeppe: yeah :) [10:27] dimitern, https://codereview.appspot.com/11506043/ reviewed [10:27] fwereade_: thanks [10:28] rvba: do you have to manually create the environment's container? [10:28] rogpeppe: yes [10:28] We could automate this part but haven't done it yet. [10:29] rvba: i assumed so, but was just checking i hadn't done something wrong [10:30] fwereade_: I was thinking of AssertChange, but good naming is an art, and it didn't sound right [10:30] fwereade_: :) so I guess I can make it AssertChanges and leave the comments I added [10:35] rvba: is it possible to create the container from the portal? (i'm probably missing another menu option again...) [10:36] rogpeppe: yes: click on the storage, then the "CONTAINERS" tab, then "Add" at the bottom. [10:37] rvba: ah, i'd missed that the storage name was a link [10:44] dimitern, AssertOneChange/AssertOnlyOneChange perhaps? [10:44] it's a bit of a nasty global rename [10:44] dimitern, but it's probably clearer than what I suggested first [10:44] now *that's* a doman name juju-azurea44xm2tkgd51kt2l7zx3228isi0olwasqcpiccatd24pmj3lf5zj9.cloudapp.net [10:45] rogpeppe: yeah, we'll reduce the size of the random part :) [10:46] fwereade_: no, I decided to go with your original suggestion - AssertChange + AssertOneChange, sgtm [10:47] fwereade_: I tend to avoid very long idents when there are doc comments anyway :)\ [10:54] fwereade_: updated https://codereview.appspot.com/11506043/ - rogpeppe: can you take a look as well please? [10:56] dimitern: will do [10:59] dimitern: is there a reason you can't do : for {data, ok := <-in; if !ok { break }; out <- data.(*params.StringsWatchResult).Changes} ? [10:59] rogpeppe: why, how is this better? [11:00] dimitern: it makes the flip logic much more obvious - why use a select? [11:00] dimitern: it's considerably shorter and simpler for one thing [11:01] rogpeppe: it's not more obvious for me at least [11:01] rogpeppe: i've never seen channels being used like that in a loop [11:02] rogpeppe: hmm [11:02] dimitern: how is this not obvious? http://paste.ubuntu.com/5887170/ [11:03] dimitern: it's a classic way to transfer data from one channel to another [11:03] rogpeppe: really? didn't know [11:03] dimitern: read a value; write the value [11:04] rogpeppe: but you have to use w.in and w.out in the loop, right? and instead of the panic you can have return nil after the loop [11:04] dimitern: with the select, it's not obvious that only one arm is ever active at a time [11:04] dimitern: yes [11:04] rogpeppe: if you know the mechanics of a select block, it is (that's by definition) [11:05] dimitern: no - you have to follow the logic of when in and out can be non-nil [11:06] dimitern: BTW i think you need to select on tomb.Dying when writing to the out channel [11:06] rogpeppe: so for instead of select in this case i agree is probably better [11:06] rogpeppe: i can see it now [11:06] dimitern: well, the original uses for *and* select [11:06] rogpeppe: but for more complicated cases where you're not just reading from one and directly writing to another as select gives you more flexibility [11:06] dimitern: sure [11:06] dimitern: this is a nice simple case though [11:07] rogpeppe: except for a small thing actually [11:07] dimitern: i know what you're going to say :-) [11:07] rogpeppe: the initial event won't be sent with that code until there is another change [11:08] dimitern: then swap the statements [11:08] rogpeppe: hmm, not really [11:08] rogpeppe: the initial event comes as an argument to the loop, it's not read from a channel [11:09] dimitern: data.(*params.StringsWatchResult).Changes [11:09] oops [11:09] dimitern: http://paste.ubuntu.com/5887184/ [11:09] rogpeppe: that will probably work yes [11:10] rogpeppe: i'll try it [11:10] rogpeppe: it's a pity if we don't as well change the notifywatcher's loop the same way though [11:11] dimitern: that's slightly different - it's ok if we're always ready to receive on w.in there [11:13] rogpeppe: there are no changes to send there, so we can always send the initial event [11:13] dimitern: yeah [11:17] dimitern: you probably want something more like this actually: http://paste.ubuntu.com/5887201/ [11:18] rogpeppe: why wait on the tomb? the commonWatcher is already doing this [11:19] rogpeppe: I think it's more like http://paste.ubuntu.com/5887206/ [11:19] dimitern: how else do we get out of the loop if we're trying to write to the out channel and the watcher is stopped? [11:19] rogpeppe: oops that was for the strings watcher [11:19] rogpeppe: http://paste.ubuntu.com/5887210/ that's for the notifywatcher [11:20] rogpeppe: aha! so that's why we need select - see :) [11:20] dimitern: yeah we need select when sending the out value but not when reading the in value [11:21] rogpeppe: the interesting thing is, it still works, without the select on the tomb [11:21] rogpeppe: just run all tests - all pass [11:21] dimitern: i bet i could make it deadlock [11:21] rogpeppe: so let's keep the original select blocks then please [11:22] dimitern: your original suggestion had the same problem [11:22] dimitern: (the one in the CL) [11:22] rogpeppe: how so? [11:22] rogpeppe: can you show me how to deadlock it? if the tests aren't detecting this case we need a better test [11:22] dimitern: yes [11:23] dimitern: you could deadlock it by creating a new watcher, reading the first change, making a change, waiting for the loop to receive a change, then stop the watcher without reading from the out channel [11:24] fwereade_: how do you feel about my lasy 2 pastes above? [11:24] dimitern: this doesn't have that problem: http://paste.ubuntu.com/5887201/ [11:25] dimitern, rogpeppe, meeting, will have to read backproperly in amo [11:25] rogpeppe: i'm not really convinced yet [11:25] dimitern: what your issue with it? [11:25] s/what/what's/ [11:26] rogpeppe: we have a goroutine in commonLoop that waits on w.tomb.Dying() [11:26] dimitern: yes, and? [11:26] rogpeppe: and it exists the loop when it happens [11:26] dimitern: there are two loops [11:27] dimitern: if the loop in StringsWatcher.loop is blocked trying to send on out, how does it know that the goroutine in commonLoop has finished? [11:27] rogpeppe: hmm.. [11:28] rogpeppe: it *can* know if it waits for it [11:28] rogpeppe: but i suppose waiting on the tomb is the same thing really [11:28] dimitern: yes. and your code wasn't waiting for it [11:28] rogpeppe: nor was john's notifywatcher [11:29] dimitern: because out != nil => in == nil [11:29] dimitern: actually john's notifywatcher was [11:29] dimitern: because it was always ready to receive on in [11:29] rogpeppe: hmm [11:30] rogpeppe: i don't see how mine is different? because i'm reading changes, rather than always sending empty structs? [11:30] dimitern: because of line 215 [11:31] rogpeppe: so if I move that at the end of the case it'll work better? [11:32] dimitern: no [11:32] rogpeppe: why? [11:32] dimitern: to make it work, you'd have to read on tomb.Dying in the select loop [11:32] rogpeppe: but the notifywatcher is not doing that and yet you say it's correct? [11:33] dimitern: in the notify watcher it's reading on w.in which is always non-nil [11:33] dimitern: the point is that when in is nil, you're trying to send on out without allowing any way to know when the tomb is killed [11:33] rogpeppe: that's only because there are always changes to send (empty) [11:34] rogpeppe: but in my case that's also true [11:34] rogpeppe: meeting? [11:34] dimitern: there aren't always changes to send in the notifyWatcher - otherwise it would always be sending [11:34] standup time [11:34] ah yes [11:35] I can't make standup I'm afraid, will be back shortly though [11:44] rogpeppe: lp:~jameinel/juju-core/upgrader-api-worker lp:~jameinel/juju-core/long-timeouts [12:00] * TheMue => lunch [12:06] rogpeppe: btw, I was told that Ian landed a change which affects how the MAAS and the azure provider get the instance id of a machine (previously, that was in environprovider). So I need to test this. [12:15] rogpeppe: I got an awful crash when starting jujud: http://paste.ubuntu.com/5887330/ (bottom of the file). Does that ring any bell? [12:15] rvba: interesting [12:16] rvba: is this on tip? [12:16] rogpeppe: yes, with a couple of tweaks in the Azure provider (that's what I'm testing) but the crash seems unrelated to my changes. [12:16] rogpeppe: apparently, it crashed precisely in the new code added by Ian. [12:19] rvba: hmm, the stack trace doesn't seem to make sense to me [12:19] rvba: it seems to imply it's crashing in yamlBase64Value.String [12:20] rvba: ah, no, i was looking at an old version [12:20] I'm using r 1487. [12:20] rvba: ha! [12:20] rvba: the problem is obvious - the previous line is missing a "return" [12:20] fmt.Errorf("cannot load state from URL %q: %v", stateInfoURL, err) [12:21] rogpeppe: ha! [12:24] noodles775, this is with the python support? [12:24] noodles775, or with juju-core? [12:24] noodles775, re deployer [12:30] hazmat: python support (PyEnvironment.wait_for_units) === flaviamissi_ is now known as flaviamissi [12:31] hazmat: I could be reading incorrectly, but it looks like we shouldn't be using juju-deployer/refactor yet. [12:43] rogpeppe: are you able to boostrap a node (with that bug fixed)? Even with a manual fix for the "return" problem I'm getting an error, the stateInfoURL is the empty string apparently… [12:43] rvba: i failed to bootstrap correctly, although the node was created. [12:44] rvba: it failed to download the tools (although it seemed to upload them ok) [12:45] rogpeppe: same here: http://paste.ubuntu.com/5887414/ [12:47] i saw this error: http://paste.ubuntu.com/5887427/ [12:48]