| davecheney | worker/instancepoller/aggregate.go:67:22: error: reference to undefined identifier ‘ratelimit.New’ bucket := ratelimit.New(gatherTime, 1) | 00:21 |
|---|---|---|
| davecheney | i have the latest github.com/juju/ratelimit | 00:21 |
| bodie_ | same test failure on my part again | 00:25 |
| bodie_ | http://paste.ubuntu.com/7117154/ | 00:26 |
| bodie_ | hrm | 00:26 |
| bodie_ | broken pipe | 00:26 |
| waigani | davecheney: godeps -u dependencies.tsv | 00:33 |
| === thumper-afk is now known as thumper | ||
| thumper | wallyworld_: the reason we don't use version.readSeries is because it conflates the host with what we are going to start using tools | 00:46 |
| wallyworld_ | ok | 00:46 |
| wallyworld_ | is there no way to avoid "some-series" | 00:46 |
| thumper | sure, can use "precise" | 00:46 |
| thumper | however | 00:46 |
| thumper | in order for that method to work correctly, they need to pass in series | 00:47 |
| thumper | so it should use that when it happens | 00:47 |
| thumper | that function is unused as yet | 00:47 |
| wallyworld_ | ok | 00:47 |
| wallyworld_ | just make sure you tell them..... | 00:47 |
| thumper | ack | 00:47 |
| thumper | davecheney: ping | 00:55 |
| thumper | davecheney: I'm getting the same failure for juju-mongodb that I was getting for mongodb-server on power | 00:55 |
| thumper | it seems to think there is no journal present | 00:55 |
| thumper | is mongod changing the user from root? | 00:56 |
| thumper | davecheney: you had the local provider working, did you do anything special for mongod? | 00:58 |
| thumper | davecheney: I want to check in prior to filing a bug | 01:00 |
| thumper | o/ axw | 01:00 |
| axw | hey thumper | 01:00 |
| davecheney | thumper: all I did was the symlink | 01:03 |
| davecheney | to be fair | 01:03 |
| davecheney | tests don't pass on ppc64el | 01:03 |
| thumper | davecheney: I get it failing to even start | 01:03 |
| thumper | davecheney: complaining about no journal | 01:03 |
| * thumper does an update | 01:04 | |
| thumper | lets see if that changes anything | 01:04 |
| thumper | nope | 01:05 |
| davecheney | whaaaaaaaaaaaa ? | 01:07 |
| davecheney | when you deploy the local machine | 01:07 |
| davecheney | sorry, bootstrap local env | 01:07 |
| thumper | umm... not long ago | 01:07 |
| davecheney | worker/instancepoller/aggregate.go:67:22: error: reference to undefined identifier ‘ratelimit.New’ bucket := ratelimit.New(gatherTime, 1) | 01:07 |
| thumper | I grabbed my branch | 01:07 |
| davecheney | ^ any idea how to fix this one | 01:07 |
| thumper | rebuilt | 01:07 |
| davecheney | then I can try to have a look | 01:07 |
| thumper | run godeps? | 01:08 |
| davecheney | ubuntu@winton-02:~/src/launchpad.net/juju-core$ go get -u -v github.com/juju/ratelimit | 01:08 |
| davecheney | github.com/juju/ratelimit (download) | 01:08 |
| davecheney | github.com/juju/ratelimit | 01:08 |
| davecheney | what ? | 01:08 |
| davecheney | \ | 01:08 |
| davecheney | head isn't buildable / | 01:08 |
| davecheney | where is the source for godeps ? | 01:08 |
| thumper | launchpad.net/godeps | 01:08 |
| davecheney | sorry | 01:08 |
| davecheney | dumb question | 01:08 |
| davecheney | ubuntu@winton-02:~/src/launchpad.net/juju-core$ godeps -u github.com/juju/ratelimit | 01:09 |
| davecheney | godeps: cannot parse "github.com/juju/ratelimit": open github.com/juju/ratelimit: no such file or directory | 01:09 |
| davecheney | oh for fuxcks sake | 01:09 |
| davecheney | why the fuck doesn't it default to something sane | 01:10 |
| davecheney | ubuntu@winton-02:~/src/launchpad.net/juju-core$ pstree -p | grep -C2 mongo | |-{jujud}(31735) | `-{jujud}(31742) |-mongod(31700)-+-{mongod}(31705) | |-{mongod}(31706) | |-{mongod}(31707) | |-{mongod}(31708) | 01:14 |
| davecheney | urgh | 01:14 |
| davecheney | thumper: bzr up'd | 01:14 |
| davecheney | and apt-get updated | 01:14 |
| davecheney | still works for me | 01:14 |
| davecheney | thumper: have you removed the mongodb-server package ? | 01:14 |
| thumper | davecheney: no | 01:14 |
| davecheney | that'll be the problem | 01:15 |
| thumper | why? | 01:15 |
| davecheney | first in the path maybe ? | 01:15 |
| thumper | nah, full path specified | 01:15 |
| davecheney | yes, but /usr/bin/mongod => mongodb-server <-- borken! | 01:15 |
| davecheney | thumper: hang on, let me test that scenario | 01:18 |
| thumper | davecheney: my upstart script specifies /usr/lib/juju/bin/mongod | 01:19 |
| davecheney | thumper: i'm still trying to reproduce the failure | 01:19 |
| * thumper is reading mongo source files ... | 01:20 | |
| davecheney | that'll only make it worse | 01:20 |
| hazmat | davecheney, my eyes are bleeding already | 01:22 |
| hazmat | davecheney, the assert seems to be from here.. https://github.com/mongodb/mongo/blob/v2.6/src/mongo/util/logfile.cpp#L241 | 01:22 |
| davecheney | thumper: i've just bootstrapped | 01:23 |
| davecheney | root 32316 2.1 0.5 445460 43088 ? Ssl 01:19 0:00 /usr/bin/mongod --auth --dbpath=/home/ubuntu/.juju/local/db --sslOnNormalPorts --sslPEMKeyFile /home/ubuntu/.juju/local/server.pem --sslPEMKeyPassword xxxxxxx --bind_ip 0.0.0.0 --port 37017 --noprealloc --syslog --smallfiles | 01:23 |
| davecheney | worked fine | 01:23 |
| hazmat | hmm | 01:23 |
| hazmat | davecheney, are you on the same emulation thingy thumper is on? | 01:23 |
| thumper | davecheney: so what is different for you and me? | 01:23 |
| thumper | this seems to be an os alignment bug | 01:23 |
| bodie_ | I could really use a second pair of brain cells on this | 01:24 |
| davecheney | ii juju-mongodb 2.4.9-0ubuntu2 ppc64el MongoDB object/document-oriented database for Juju | 01:24 |
| davecheney | ii mongodb-server 1:2.4.9-1ubuntu1 ppc64el object/document-oriented database (server package) | 01:24 |
| thumper | hazmat: davecheney is on a different machine | 01:24 |
| davecheney | bodie_: sure, mine are lonley | 01:24 |
| bodie_ | it's in TestWriteFailure in environs/sshstorage | 01:24 |
| bodie_ | http://paste.ubuntu.com/7117154/ | 01:24 |
| bodie_ | no idea what's supposed to happen, or what's going wrong, except the broken pipe bit | 01:25 |
| thumper | davecheney: can I get you to try my branch? | 01:25 |
| bodie_ | replicable | 01:25 |
| bodie_ | it's on a remote | 01:25 |
| davecheney | thumper: sure | 01:25 |
| thumper | lp:~thumper/juju-core/juju-mongodb | 01:25 |
| davecheney | bodie_: you might ahve to turn up the debug on the remove sshd | 01:25 |
| davecheney | bodie_: my guess is /bin/ssh is | 01:25 |
| thumper | davecheney: build it, and make sure that juju-mongodb is installed | 01:25 |
| davecheney | a. crashing | 01:25 |
| davecheney | b. being hung up on | 01:25 |
| thumper | bootstrap a simple local provider | 01:25 |
| thumper | davecheney: and it should start with no tweaking at all | 01:26 |
| davecheney | OT: hazmat where can I get a trusty install image that is < 2.5 Gb | 01:26 |
| thumper | davecheney: in theory | 01:26 |
| bodie_ | could it be caused by disabling root login? | 01:26 |
| davecheney | bodie_: absolutely | 01:26 |
| bodie_ | hm | 01:27 |
| davecheney | check /var/log/secure.log | 01:27 |
| davecheney | or audit.log | 01:27 |
| davecheney | or wherever the AUTH syslog target points too | 01:27 |
| davecheney | bodie_: the bug is that that test is discarding the output of stderr | 01:28 |
| hazmat | davecheney, hmm.. i just use cloud-images most of the time.. their minimal and small.. but you mean like a boot installer.. if you need super small and have the bandwidth.. there's always net install | 01:28 |
| davecheney | hazmat: i just want to download it and try it | 01:29 |
| davecheney | do I really need a 2.5 Gb iso ? | 01:29 |
| bodie_ | hrm ? ... I'm redirecting stderr into the test output.. so you're saying it didn't go in because it was being discarded in the first place? | 01:29 |
| hazmat | davecheney, http://cdimage.ubuntu.com/daily-live/current/ look like 950mb | 01:29 |
| davecheney | hazmat: ta, that'll do nicely | 01:29 |
| hazmat | davecheney, kvm with cloud image.. gets you down to even smaller | 01:29 |
| davecheney | bodie_: by default stderr and stdout go to dev null unless otherwise requested | 01:30 |
| hazmat | davecheney, 240mb http://cloud-images.ubuntu.com/trusty/current/ | 01:30 |
| davecheney | hazmat: ta muchly | 01:30 |
| hazmat | davecheney, bottom of that page for the raw downloads | 01:30 |
| davecheney | much better solution | 01:30 |
| bodie_ | I called it like so: go test ./... > output.txt 2>&1 & | 01:30 |
| hazmat | davecheney, also.. virtualbox images here.. http://cloud-images.ubuntu.com/vagrant/trusty/current/ | 01:30 |
| davecheney | bodie_: this isn't something you've done | 01:31 |
| bodie_ | ok | 01:31 |
| davecheney | this is a bug inside the test code | 01:31 |
| davecheney | hazmat: a bounty of resources | 01:31 |
| davecheney | hazmat: that error | 01:31 |
| davecheney | da fuq | 01:31 |
| wallyworld_ | thumper: pingy | 01:31 |
| davecheney | that isn't a journal problem | 01:32 |
| davecheney | this is somethig to do with the 64mb pages option | 01:32 |
| thumper | coming | 01:32 |
| davecheney | thumper: this may be the problem | 01:32 |
| davecheney | Linux winton-02 3.13.0-8-generic #28-Ubuntu SMP Mon Feb 17 08:22:39 UTC 2014 ppc64le ppc64le ppc64le GNU/Linux | 01:32 |
| davecheney | thumper: what you got ? | 01:32 |
| davecheney | i reckon you've got a different kernel | 01:32 |
| thumper | uname -a? | 01:32 |
| davecheney | thumper: yup | 01:32 |
| thumper | Linux rockne-02 3.13.0-15-generic #35-Ubuntu SMP Mon Mar 3 15:56:08 UTC 2014 ppc64le ppc64le ppc64le GNU/Linux | 01:33 |
| davecheney | hmm | 01:33 |
| davecheney | ok, that is another angle to try | 01:33 |
| waigani_ | thumper: hangouts is not loading for me :( | 01:39 |
| waigani_ | wallyworld_: you lot still in hangout? I can't even load https://plus.google.com/ | 01:41 |
| wallyworld_ | waigani_: yeah, we're here | 01:41 |
| waigani_ | wtf? | 01:41 |
| waigani_ | :( | 01:41 |
| wallyworld_ | talk about you :-P | 01:41 |
| davecheney | waigani_: thumper is using the internet | 01:41 |
| davecheney | you'll have to wait your turn | 01:41 |
| wallyworld_ | lol | 01:42 |
| waigani_ | lol - bastards | 01:42 |
| thumper | davecheney: can you run 'getconf PAGESIZE' | 01:49 |
| thumper | davecheney: I get 65536 | 01:50 |
| davecheney | ubuntu@winton-02:~$ getconf PAGESIZE | 01:50 |
| davecheney | 4096 | 01:50 |
| davecheney | bingo | 01:50 |
| wallyworld_ | i get 4096 also | 01:50 |
| thumper | arse | 01:50 |
| davecheney | thumper: https://docs.google.com/a/canonical.com/spreadsheet/ccc?key=0Aje1qBKEwuLVdDIxdG9NeWF2REJjemdLbllzbGotamc&usp=drive_web#gid=0 | 01:50 |
| wallyworld_ | thumper: patch getconf :-) | 01:50 |
| davecheney | line 10 | 01:50 |
| thumper | wallyworld_: on rockne? | 01:50 |
| wallyworld_ | yeah | 01:51 |
| davecheney | thumper: wallyworld_ the 4k/64k thing is provided by the kvm emulation | 01:51 |
| davecheney | it's totally possible that different vm's get different views of the underlying hardware | 01:51 |
| wallyworld_ | but why does a log funtion vare? | 01:51 |
| wallyworld_ | care | 01:51 |
| wallyworld_ | we could just patch it to see if it works as an experiment | 01:52 |
| davecheney | wallyworld_: simpler/safer just to apt-get update a new kernel image | 01:52 |
| wallyworld_ | davecheney: what does your uname -r say? | 01:53 |
| davecheney | 3.13.0-8-generic | 01:53 |
| davecheney | i'm a week or two behind | 01:53 |
| wallyworld_ | i wonder that thumper's is? | 01:53 |
| wallyworld_ | what | 01:53 |
| davecheney | wallyworld_: check scrollback | 01:54 |
| wallyworld_ | ah. -15 | 01:54 |
| davecheney | wallyworld_: i thnk that assert is checking that the log file is aligned to the os page size | 01:55 |
| davecheney | probably because they use mmap a lot | 01:55 |
| davecheney | ^ just a guess | 01:55 |
| davecheney | but probably not totally wrong | 01:55 |
| wallyworld_ | makes sense i guess | 01:55 |
| davecheney | mongo does a lot of dumb shit for performance without proof that the simple way was a problem to begin with | 01:55 |
| wallyworld_ | seems like they could do better though | 01:55 |
| wallyworld_ | s/mongo does a lot of dumb shit for performance without proof that the simple way was a problem to begin with/mongo does a lot of dumb shit | 01:56 |
| davecheney | fassert( 16142, _fd >= 0 ); | 01:56 |
| davecheney | what the bollocks is this ? | 01:56 |
| davecheney | every assert has a unique number | 01:56 |
| davecheney | but this isn't a constant ? | 01:57 |
| thumper | davecheney: it'll be a number they can grep the source for :-) | 01:57 |
| wallyworld_ | fassert(42, 'the meaning of life') | 01:57 |
| * thumper takes a stab in the dark | 01:57 | |
| davecheney | thumper: i hope youre joking | 01:57 |
| wallyworld_ | it's mongo, of course not | 01:58 |
| * thumper guesses not | 01:58 | |
| davecheney | fassert( 2 , 2 != 2 ) // totally unique, bro | 01:58 |
| waigani_ | chrome does not load plus.google.com anymore, yet firefox does ?!? You'd think, if anything, it would be the other way around | 01:58 |
| thumper | davecheney: can you try my branch? | 02:04 |
| thumper | I was told that there was a second VM for me, but it looks like not... | 02:04 |
| thumper | permissing denied (public key) | 02:05 |
| thumper | wallyworld_: added cloudinit test for trusty state server | 02:27 |
| wallyworld_ | thanks :-) | 02:27 |
| thumper | np | 02:27 |
| davecheney | wallyworld_: bug: 2014-03-19 03:04:00 DEBUG juju.environs.simplestreams simplestreams.go:995 metadata: &{map[com.ubuntu.juju:14.04:ppc64:{ 1.17.6.1 ppc64 map[20140319:0xc21042bde0]} com.ubuntu.juju:12.04:ppc64:{ 1.17.6.1 ppc64 map[20140319:0xc21042b720]}] map[] Wed, 19 Mar 2014 03:03:55 +0000 products:1.0 com.ubuntu.juju:released:tools} | 03:04 |
| davecheney | thumper: root 11980 0.7 0.4 1076640 37628 ? Ssl 03:04 0:00 /usr/lib/juju/bin/mongod --auth --dbpath=/home/ubuntu/.juju/local/db --sslOnNormalPorts --sslPEMKeyFile /home/ubuntu/.juju/local/server.pem --sslPEMKeyPassword xxxxxxx --bind_ip 0.0.0.0 --port 37017 --noprealloc --syslog --smallfiles | 03:04 |
| davecheney | branch looks fine, LGTM | 03:04 |
| davecheney | urgh, bugs in the server package | 03:04 |
| davecheney | Removing mongodb-server (1:2.4.9-1ubuntu1) ... | 03:05 |
| davecheney | arg: remove | 03:05 |
| davecheney | mongodb stop/waiting | 03:05 |
| wallyworld_ | davecheney: me not understand? | 03:05 |
| davecheney | the middle line | 03:05 |
| davecheney | some output that shouldn't leak to the user | 03:05 |
| wallyworld_ | that's debug | 03:05 |
| * thumper goes to drink coffee | 03:05 | |
| thumper | davecheney: thanks for testing | 03:05 |
| davecheney | thumper: looking good | 03:06 |
| davecheney | sorry your vm has brain issues | 03:06 |
| davecheney | ttps://bugs.launchpad.net/ubuntu/+source/mongodb/+bug/1294455 | 03:06 |
| _mup_ | Bug #1294455: mongodb-server leaks debug information during remove <mongodb (Ubuntu):New> <https://launchpad.net/bugs/1294455> | 03:07 |
| davecheney | not important | 03:07 |
| bodie_ | so I tried upping the log level on sshd to see about that broken pipe, and it caused a bunch of other test failures for some reason. | 03:07 |
| bodie_ | I guess my next step is to try a 12.04 vm, and if that doesn't work, i don't know | 03:08 |
| davecheney | bodie_: you said you disabled root login | 03:08 |
| davecheney | that is probably the root cause | 03:08 |
| bodie_ | yeah, still breaks with root login enabled | 03:08 |
| davecheney | ok | 03:08 |
| davecheney | bodie_: are you canonical | 03:08 |
| davecheney | ? | 03:08 |
| bodie_ | contracting w/ canonical, so, sorta | 03:08 |
| bodie_ | hai o/ | 03:08 |
| davecheney | right, you have my sympathies | 03:08 |
| wallyworld_ | davecheney: as soon as outgoing ssh and access to port 10170 is sorted out, you can pull trunk and should be able to bootstrap ec2, hp cloud etc environments | 03:09 |
| davecheney | i have no idea how you have managed to get yourself into this knot | 03:09 |
| bodie_ | i suspect it's a reality tv show | 03:09 |
| davecheney | i recommend a 12.04 vm | 03:09 |
| davecheney | that is what the landing bot users | 03:09 |
| davecheney | wallyworld_: OH CRAP, i forgot about the api port | 03:09 |
| davecheney | this is a nightmare | 03:09 |
| wallyworld_ | davecheney: i asked for the api port in the rt | 03:09 |
| bodie_ | hey man, I'm getting paid to write Go and work from home. I'LL. TAKE. IT. | 03:09 |
| davecheney | wallyworld_: could I try it now | 03:09 |
| bodie_ | I just really want to get rolling. the testing stuff is driving me #go-nuts. | 03:10 |
| wallyworld_ | davecheney: you could, but it won't bootstrap cause it will get stuck on ssh | 03:10 |
| wallyworld_ | unless your vm is special | 03:10 |
| davecheney | wallyworld_: could I bootstrap from my amd64 machine to a precise/386 image, or from an armhf machine to an amd64 precise image | 03:10 |
| davecheney | that is effectively the same thing | 03:10 |
| wallyworld_ | yes | 03:10 |
| wallyworld_ | yes you could | 03:10 |
| davecheney | good, i'll test that while I wait | 03:10 |
| wallyworld_ | so long as the tools are there | 03:10 |
| wallyworld_ | or, so long as they are packaged and available locally | 03:11 |
| davecheney | wallyworld_: i'll choose a version that I know tools exists for | 03:11 |
| davecheney | bodie_: same, https://twitter.com/davecheney/status/446075333360361472 | 03:11 |
| wallyworld_ | davecheney: if you have the tarballs, you can use the --metadata-source bootstrap are and they will be uploaded | 03:11 |
| wallyworld_ | arg | 03:11 |
| wallyworld_ | just fyi | 03:12 |
| davecheney | wallyworld_: whoa there big fella, i'm not that keen | 03:12 |
| axw | davecheney: I came to the realisation the other day that I basically never wear shoes anymore | 03:12 |
| axw | my feet never felt so free | 03:12 |
| axw | thumper: https://codereview.appspot.com/77340046 when you're not too busy | 03:27 |
| axw | I'd kinda like to get it in for 1.17.x so we don't have backwards compat nightmares | 03:28 |
| * axw goes to pick up his daughter | 03:28 | |
| thumper | kk | 03:31 |
| * thumper will look now | 03:31 | |
| davecheney | wallyworld_: I don't thnk we should/need to ask for a firewall hole for 17017 | 03:44 |
| davecheney | that must be proxyable | 03:44 |
| davecheney | and we should prxy it | 03:44 |
| davecheney | it's http after all | 03:44 |
| wallyworld_ | hmm, yeah | 03:45 |
| davecheney | so the proxy will have to allow | 03:45 |
| davecheney | CONNECT machine-0:17017 HTTP/1.1 | 03:46 |
| wallyworld_ | 17070 i think you mean | 03:46 |
| wallyworld_ | i'll update the rt | 03:47 |
| davecheney | yes | 03:47 |
| davecheney | i don't know what i'm talking abouit | 03:47 |
| davecheney | jsut kind of wildly stabbing at the keys | 03:47 |
| thumper | davecheney: so, with my branch and your VM, did the local provider start and work with no tweaks? | 03:49 |
| davecheney | yes | 03:50 |
| thumper | \o/ | 03:50 |
| === vladk|offline is now known as vladk | ||
| thumper | davecheney: I've made your tweak suggestion | 03:50 |
| davecheney | hang on, lemmie get some proof | 03:50 |
| thumper | davecheney: I was kinda relying on us refactoring that code in the next six months due to other OS support | 03:50 |
| thumper | but was a good call | 03:50 |
| thumper | so I changed it | 03:50 |
| thumper | o/ vladk | 03:50 |
| vladk | thumper: hi | 03:51 |
| davecheney | O_o! http://paste.ubuntu.com/7117795/ | 03:51 |
| davecheney | thumper: http://paste.ubuntu.com/7117798/ | 03:51 |
| davecheney | winning! | 03:51 |
| thumper | yeah baby, yeah! | 03:52 |
| thumper | davecheney: make sure you write that one up for the end of day email! | 03:52 |
| thumper | wallyworld_: so for power -> ec2 we are just waiting firewall poking? | 03:52 |
| wallyworld_ | thumper: believe so. of course my code will first first time, guarantee it | 03:53 |
| thumper | coolio | 03:53 |
| wallyworld_ | thumper: before we ship 1.18, i want to get my current branch in if possible, which is proper detection of supported arches on each provider, based on what images are available via simplestreams | 03:54 |
| thumper | wallyworld_: line it up | 03:54 |
| wallyworld_ | thumper: still wip | 03:55 |
| davecheney | wallyworld_: root 11980 0.1 0.4 1076640 41212 ? Ssl 03:04 0:05 /usr/lib/juju/bin/mongod --auth --dbpath=/home/ubuntu/.juju/local/db --sslOnNormalPorts --sslPEMKeyFile /home/ubuntu/.juju/local/server.pem --sslPEMKeyPassword xxxxxxx --bind_ip 0.0.0.0 --port 37017 --noprealloc --syslog --smallfiles | 03:56 |
| davecheney | out of the box | 03:56 |
| davecheney | noice | 03:56 |
| wallyworld_ | davecheney: you mean the password being logged? | 03:57 |
| davecheney | wallyworld_: eh ? | 03:58 |
| wallyworld_ | davecheney: sorry, i didn;t grok your paste | 03:58 |
| davecheney | mongo obscures the passwd on the cmdline itself | 03:58 |
| davecheney | i didn't do that | 03:58 |
| davecheney | marcoceppi: what would it take to promulgate a trusty/mysql charm ? | 04:00 |
| davecheney | thumper: does debug-log work for the local provider ? | 04:05 |
| thumper | davecheney: not until it is handled over the api | 04:06 |
| davecheney | ubuntu@winton-02:~/src/launchpad.net/juju-core$ juju debug-log | 04:06 |
| davecheney | Warning: Permanently added 'localhost' (ECDSA) to the list of known hosts. | 04:06 |
| davecheney | Permission denied (publickey). | 04:06 |
| davecheney | ERROR rc: 255 | 04:06 |
| davecheney | ok | 04:06 |
| thumper | davecheney: there is an all-machines.log | 04:06 |
| davecheney | yeah, i just forgot | 04:06 |
| thumper | np | 04:06 |
| davecheney | ubuntu@winton-02:~/src/launchpad.net/juju-core$ tail -f ~/.ju-bash: cannot create temp file for here-document: No space left on device | 04:06 |
| davecheney | ^C | 04:06 |
| davecheney | uh oh | 04:06 |
| davecheney | somethign has gone batshit | 04:06 |
| axw | thumper: thanks, yeah I tested with local and canonistack - just about to test with azure | 04:07 |
| davecheney | ok cock | 04:08 |
| davecheney | root@winton-02:~# du -sh /var/lib/mongodb/* | 04:08 |
| davecheney | 3.1G /var/lib/mongodb/journal | 04:08 |
| davecheney | 64M /var/lib/mongodb/local.0 | 04:08 |
| davecheney | 17M /var/lib/mongodb/local.ns | 04:08 |
| davecheney | 0 /var/lib/mongodb/mongod.lock | 04:08 |
| davecheney | 4.0K /var/lib/mongodb/_tmp | 04:08 |
| davecheney | mongo has gone batshit | 04:08 |
| davecheney | oh hang on | 04:08 |
| davecheney | this is the system mongo db | 04:08 |
| davecheney | not outs | 04:08 |
| davecheney | ours | 04:08 |
| davecheney | ok | 04:08 |
| davecheney | i'll just delete that | 04:08 |
| thumper | haha | 04:08 |
| davecheney | it probably is still batshit insane | 04:09 |
| davecheney | lemmie raise a bug for that | 04:09 |
| * thumper tries to land that branch again | 04:09 | |
| davecheney | If you're happy and you know it, land your branch ♬ | 04:11 |
| davecheney | root@winton-02:~# du -sh /var/lib/mongodb | 04:14 |
| davecheney | 8.0K /var/lib/mongodb | 04:14 |
| davecheney | great, now I can't trigger the problem | 04:14 |
| davecheney | asshole | 04:14 |
| thumper | davecheney: branch landed | 04:29 |
| === vladk is now known as vladk|offline | ||
| dimitern | morning all | 07:38 |
| dimitern | mgz, jam, relatively quick review ? https://codereview.appspot.com/77340044/ | 07:39 |
| rogpeppe | mornin' all | 08:05 |
| rogpeppe | dimitern: 1000 line diff is "relatively quick"? :-) | 08:05 |
| axw | morning | 08:13 |
| axw | relative to a 2000 line diff it is ;) | 08:14 |
| dimitern | :) | 08:16 |
| dimitern | rogpeppe, most of that are tests | 08:17 |
| rogpeppe | dimitern: so we're not supposed to read tests as closely then? :-) | 08:18 |
| dimitern | :D | 08:18 |
| rogpeppe | axw: hiya | 08:18 |
| dimitern | i'll remember that one | 08:18 |
| rogpeppe | dimitern: i confess that i probably *don't* read tests as closely as production code. | 08:18 |
| dimitern | rogpeppe, well then ;) give it a try ? https://codereview.appspot.com/77340044/ | 08:19 |
| rogpeppe | dimitern: currently reviewing https://codereview.appspot.com/77500045 | 08:20 |
| dimitern | rogpeppe, sure, np | 08:20 |
| dimitern | fwereade, so it turned out there *will be* a sprint in Malta after all.. just not for us ;) | 08:21 |
| dimitern | last week of may for the apps & client guys | 08:21 |
| axw | I'd rather go to Malta than Las Vegas... hmm that is quite the first world problem | 08:22 |
| fwereade | dimitern, ha, yeah | 08:22 |
| rogpeppe | lol | 08:22 |
| davecheney | ya'all are writing the the wrong Go application | 08:22 |
| fwereade | dimitern, I will continue to quietly agitate for malta sprints | 08:23 |
| rogpeppe | axw: do we now have a valid environment configuration in the state from the word go now? | 08:39 |
| axw | rogpeppe: yup | 08:39 |
| rogpeppe | axw: cool. we can probably simplify various pieces of agent logic because of that | 08:39 |
| axw | rogpeppe: yeah, there are definitely some things I haven't gotten around to simplifying. e.g. workers waiting for valid config | 08:40 |
| rogpeppe | axw: that's what i was thinking of | 08:40 |
| === vladk|offline is now known as vladk | ||
| axw | rogpeppe: it is still possible to get an invalid config *after* bootstrap, but that's very close to being fixed | 08:41 |
| axw | rogpeppe: there's a small amount of work required to ensure config changes in state are transactionally validated&updated | 08:41 |
| vladk | morning all | 08:42 |
| axw | morning vladk | 08:42 |
| rogpeppe | axw: ah yes, because if you've got two set-environments concurrently, the combination might not be valid, i guess | 08:42 |
| rogpeppe | vladk: hiya | 08:42 |
| axw | rogpeppe: yup. waigani has been working on fixing it. there's a bit more to go, but it's still racy for the moment | 08:43 |
| dimitern | mgz, ping | 08:57 |
| dimitern | axw, fwereade, rogpeppe, jam, i'd really appreciate a review so i can land https://codereview.appspot.com/77340044/ | 09:03 |
| === Ursinha is now known as Ursinha-afk | ||
| rogpeppe | dimitern: sorry, still on that other review | 09:03 |
| fwereade | dimitern, I'll take a look | 09:03 |
| axw | about to go afk, will take a look later if it's not reviewed | 09:04 |
| dimitern | fwereade, ta! | 09:04 |
| === Ursinha-afk is now known as Ursinha | ||
| rogpeppe | fwereade, jam: do you think we need to elide passwords from service-set configuration attributes in the log? | 09:24 |
| fwereade | dimitern, reviewed, let me know what you think | 09:38 |
| dimitern | fwereade, cheers, looking | 09:39 |
| dimitern | fwereade, i'm not quite following you on that one https://codereview.appspot.com/77340044/diff/1/agent/agent.go#newcode135 | 09:41 |
| dimitern | fwereade, how do you suggest to improve the doc comment? | 09:42 |
| fwereade | dimitern, so, we've got a global mutex for config changes | 09:42 |
| dimitern | fwereade, I agree it's not perfect, but it's a step in the right direction | 09:42 |
| dimitern | fwereade, yeah | 09:42 |
| fwereade | dimitern, right, yeah, as I say prgress not perfection | 09:42 |
| fwereade | dimitern, just add a note to configMutex explaining the situation if there isn't already one | 09:43 |
| dimitern | fwereade, that it's used to guard against modifying a created config? | 09:43 |
| fwereade | dimitern, that it sort of implies a singleton config, but all we really have is an accidental-singleton config | 09:44 |
| dimitern | fwereade, ok, got it | 09:45 |
| jam1 | rogpeppe: can I get a follow up review on: https://codereview.appspot.com/76120044/ I simplified it | 09:45 |
| rogpeppe | jam1: looking | 09:45 |
| rogpeppe | jam1: what was the problem with my suggested (much simpler still) approach? | 09:48 |
| jam1 | rogpeppe: the only difference is I use package vars (that I can change in a threadsafe manner), and do some logging. | 09:49 |
| rogpeppe | jam1: utils.OverrideGOMAXPROCSFuncs seems ugly to me | 09:49 |
| rogpeppe | jam1: and it doesn't seem necessary | 09:49 |
| rogpeppe | jam1: i'd prefer to keep mocking inside package boundaries when reasonable to do so | 09:50 |
| jam1 | rogpeppe: so the alternative is to make them public variables, which doesn't seem great, vs a public function to set them. | 09:50 |
| jam1 | rogpeppe: how do you suggest telling if GOMAXPROCS was called? by calling it once to force it to a given value, and then calling it again to observe that the runtime has changed? | 09:50 |
| rogpeppe | jam1: we could use the approach already used inside cmd/jujud: var useMultipleCPUs = utils.UseMultipleCPUs | 09:51 |
| jam1 | (runtime.GOMAXPROCS(1); run the agent; finalN := runtime.GOMAXPROCS(0); c.Assert(finalN, gc.Equals, 2) ) ? | 09:51 |
| rogpeppe | jam1: (see newDeployContext for an example) | 09:52 |
| jam1 | fairy nuff | 09:52 |
| jam1 | we miss a bit of end-to-end of a test, but we do test the two layers | 09:52 |
| axw | fwereade: I reworked the DistributionInstances CL. I'm now looking at reworking (replacing) the UnitAssigner one. Based on our discussion last night, I'm going to add a method, SupportsUnitPlacement, to the new EnvironCapability interface that wallyworld introduced | 09:52 |
| dimitern | fwereade, re https://codereview.appspot.com/77340044/diff/1/agent/agent.go#newcode176 | 09:53 |
| jam1 | rogpeppe: TBH I don't really give a ****, I just want GOMAXPROCS getting called. | 09:53 |
| dimitern | fwereade, the _DELETE_ key is used to specify values to delete | 09:53 |
| axw | fwereade: then I'll move EnvironCapability into state, and check it during AddMachine/Unit.AssignTo* | 09:53 |
| fwereade | axw, I'm going to pre-send a single comment on one of your reviews, please read it and ping me :) | 09:53 |
| rogpeppe | jam1: yeah, sorry. i just didn't like the complexity i saw there. | 09:53 |
| axw | fwereade: sure | 09:53 |
| axw | NOT LGTM? ;) | 09:54 |
| rogpeppe | jam1: particularly as it cluttered the public interface of utils | 09:54 |
| fwereade | axw, ;p | 09:54 |
| fwereade | dimitern, yeah, and it makes me sad | 09:54 |
| rogpeppe | jam1: FWIW your test wasn't properly end to end either | 09:54 |
| fwereade | dimitern, it feels likewe're kind of abusing that Paramsstruct because it has some fields that overlap with what we really want | 09:54 |
| dimitern | fwereade, initially I tried having "" to signal delete, but it's not quite enough if you just need to add/set an empty field | 09:55 |
| fwereade | dimitern, it seems like it'd be clearer and less vulnerable to hilarious screwups to use a different struct for the migrate func | 09:55 |
| dimitern | fwereade, ah, you're saying we need a different struct than AgentConfigParams ? | 09:55 |
| fwereade | dimitern, eg changing tag/nonce is likely to be fatal | 09:55 |
| axw | fwereade: why would StateServer not be true for new state servers? are you thinking about repurposed machine agents? | 09:56 |
| dimitern | fwereade, *any* reckless change is likely to be fatal :) | 09:56 |
| fwereade | dimitern, granted, but I don;t see much reason to allow changes that basically can't ever make sense | 09:56 |
| fwereade | axw, the trouble is that MachineConfig.StateServer really means MachineConfig.Bootstrap IIRC | 09:57 |
| fwereade | axw, maybe everything already ignores StateServer if passed into StartInstance? | 09:57 |
| axw | fwereade: it's only used in environs/cloudinit | 09:58 |
| dimitern | fwereade, ok, what then, to summarize? change the params struct and remove some of the things you can change? or just change the struct | 09:58 |
| fwereade | axw, if so that's sort of ok but still very vulnerable to accidental screwups | 09:58 |
| fwereade | dimitern, wait, was that struct new in that CL? I thought it was repurposed from creation-time | 09:58 |
| fwereade | dimitern, I think we want a separate struct | 09:59 |
| dimitern | fwereade, AgentConfigParams was chosen because it fits nicely with the NewAgentConfig() taking params and constructing one | 09:59 |
| fwereade | dimitern, but maybe I missed something? | 09:59 |
| fwereade | dimitern, yeah, my contention is that they're different jobs and want different params | 09:59 |
| dimitern | fwereade, yeah, but if we're going to have mostly the same fields? | 10:00 |
| fwereade | dimitern, stuff like "" meaning "leave alone" in one context, and "set to empty" in another, etc | 10:00 |
| dimitern | fwereade, ok, I'll look into it and repropose | 10:00 |
| fwereade | dimitern, how many fields are you actually changing though? | 10:00 |
| dimitern | fwereade, potentially DataDir, LogDir, Jobs and Values | 10:01 |
| fwereade | dimitern, I'd really prefer a struct with those fields (and a separate DeleteValues field, rather than a magic key) | 10:01 |
| dimitern | fwereade, changing DataDir is as dangerous as changing Tag or Nonce, but it's required for this migration | 10:01 |
| dimitern | fwereade, will do | 10:02 |
| fwereade | dimitern, understood, this is a sharp knife we're building | 10:02 |
| axw | fwereade: I was thinking I could change the DistributionInstances code to return the instances for state server machines. You still need to know whether it's going to be a state server at provisioning time (I had planned to use MachineConfig.StateServer to decide whether to do this) | 10:02 |
| fwereade | axw, I think we will know that, but we don't currently express it well | 10:03 |
| axw | ... because instances can't be moved between Cloud Services later | 10:03 |
| fwereade | rogpeppe, can I get your input on this discussion with axw? see comments in https://codereview.appspot.com/73910043/ and discussion here | 10:03 |
| axw | fwereade: alternatively, base it on the jobs in state for the machine being provisioned | 10:03 |
| rogpeppe | fwereade: looking | 10:03 |
| fwereade | axw, yeah, ISTM that the DistributionInstances path is the rightone | 10:03 |
| axw | fwereade: if MachineConfig.StateServer really means "BootstrapMachine" then I think that's the way to go | 10:04 |
| fwereade | axw, I think it pretty much does *but* you have done the most stuff to bootstrap lately, and I have only *read* that code rather than *actively worked with it* so your instincts may well be more finely tuned | 10:05 |
| axw | fwereade: that CL is going to have to change a bit anyway, because labels won't be useful anymore | 10:05 |
| fwereade | axw, if you can say for sure I'm wrong there then I will gladly defer to your superior perspective :) | 10:05 |
| axw | fwereade: it is now, but that's only because HA isn't merged yet :) | 10:05 |
| fwereade | axw, yeah, indeed :) | 10:06 |
| axw | fwereade: that code was all kinda circumspect | 10:06 |
| axw | I figured it'd have to change a bit | 10:06 |
| fwereade | axw, there's a whole load of stuff that sync-bootstrap has rendered irrelevant-shading-to-crazy, and it'd be lovely to get time to rationalise all that one day | 10:06 |
| axw | indeed | 10:07 |
| fwereade | axw, but while it's not actively hurting us it remains a copious-free-time deal ;) | 10:07 |
| axw | fwereade: does the EnvironCapability thing sound reasonable? | 10:09 |
| axw | I think it's basically what you were saying last night | 10:09 |
| fwereade | axw, that name sounds good, but I sorry, which CL? | 10:10 |
| axw | fwereade: none, I was referring to a message before | 10:12 |
| axw | <axw> fwereade: I reworked the DistributionInstances CL. I'm now looking at reworking (replacing) the UnitAssigner one. Based on our discussion last night, I'm going to add a method, SupportsUnitPlacement, to the new EnvironCapability interface that wallyworld introduced | 10:12 |
| axw | <axw> fwereade: then I'll move EnvironCapability into state, and check it during AddMachine/Unit.AssignTo* | 10:12 |
| * fwereade hunts frantically for the EnvironCapability code | 10:13 | |
| axw | fwereade: environs/interface.go | 10:14 |
| axw | fwereade: landed yesterday I think | 10:14 |
| * fwereade sees it there and wonders how his grep failed | 10:14 | |
| axw | fwereade: we could also use this for enabling/disabling the firewaller job, for example | 10:15 |
| fwereade | axw, yeah, that sounds sane to me | 10:15 |
| fwereade | axw, thanks | 10:15 |
| axw | goodo | 10:15 |
| axw | thanks | 10:15 |
| axw | fwereade: out of interest, why would/should StateServer not affect cloudinit? won't that be the place for installing juju-mongodb/mongodb-server still? | 10:19 |
| fwereade | axw, it seemed rather better to let the new state servers know all their special info over the api connection rather than jamming important info into cloudinit (where other things can read it) | 10:21 |
| axw | I think that might be the only thing to do though, the rest requires a secure connection - so probably a InstallMongo flag would be more appropriate | 10:21 |
| fwereade | axw, also, it'd be nice to be able to promote a machine to a state server at some date in the future | 10:21 |
| axw | yeah, makes sense | 10:21 |
| fwereade | axw, even if we don't do so today | 10:22 |
| axw | fwereade: sure, except for on azure of course :) | 10:22 |
| fwereade | axw, quite so :) | 10:22 |
| fwereade | axw, goddamn azure :) | 10:22 |
| axw | heh | 10:22 |
| fwereade | axw, anyway I think the DistributionInstances thing is good *anyway* | 10:22 |
| axw | the new manual provider | 10:22 |
| fwereade | axw, because we'll want to distrubute state servers where possible everywhere, right? | 10:23 |
| axw | yeah I think so too, I was just curious | 10:23 |
| axw | fwereade: yes it'll be useful for ec2 | 10:23 |
| perrito666 | good morning everyone | 10:23 |
| axw | fwereade: bbl, gotta take care of my little one. thanks for the chat. | 10:24 |
| fwereade | axw, cheers | 10:24 |
| jam1 | rogpeppe: https://codereview.appspot.com/76120044 updated | 10:26 |
| jpds | jam1: Hey. | 10:32 |
| jam1 | hi jpds | 10:32 |
| jpds | What happened to "juju image-metadata" ? | 10:32 |
| jam1 | jpds: it is 'juju metadata generate-image' | 10:32 |
| jam1 | or 'juju metadata validate-image' | 10:32 |
| jpds | And how do I get index and imagemetadata.json into my S3 stuff? | 10:33 |
| jam1 | jpds: in 1.16 or in 1.17+ ? | 10:33 |
| jam1 | We've been polishing it a bit in 1.17 | 10:33 |
| jpds | jam1: 1.16. | 10:33 |
| jam1 | :( | 10:33 |
| jpds | jam1: OK, so I have the stuff; where am I suppose to put it? | 10:34 |
| jam1 | jpds: you haven't bootstrapped yet, correct? | 10:34 |
| jpds | jam1: Trying to. | 10:34 |
| jam1 | (in 1.17 you can juju bootstrap --metadata-source /path/on/local/disk) but that won't help you for 1.16 | 10:34 |
| jpds | Excellent. | 10:35 |
| jam1 | I'm digging out 1.16 to see if I have answers for you | 10:36 |
| jam1 | jpds: as a rough approximation, you need to put it somewhere preserving the directory structure (streams/v1/index.json) and then point "image-metadata-url=PATH_TO_ROOT" | 10:38 |
| jpds | I'm just going to upgrade to 1.17. | 10:40 |
| jam1 | jpds: works for me :) | 10:40 |
| voidspace | grabbing a cuppa before standup | 10:43 |
| jpds | jam1: It's still complaining that index file has no data for cloud. | 10:47 |
| jam1 | jpds: do you have the backtrace of commands that you've run and their output? | 10:47 |
| jpds | jam1: Yes. | 10:47 |
| jam1 | You need to make sure that your "generate-image" command has the settings that match what you are bootstrapping. | 10:47 |
| natefinch | standup | 10:48 |
| jam1 | natefinch: thanks for the reminder, switching machines | 10:48 |
| jam1 | brt | 10:48 |
| rogpeppe | jam1: reviewed | 10:50 |
| jam | mgz: poke | 10:51 |
| dimitern | fwereade, updated https://codereview.appspot.com/77340044/ | 11:19 |
| dimitern | fwereade, should be ready to land now i think | 11:29 |
| perrito666 | so, to what degree of poking am I allowed to go to obtain a review? :p | 11:33 |
| * perrito666 starts mailing plastic fingers | 11:34 | |
| jam | wallyworld: if you're around, any ideas why "juju bootstrap" would be trying to look for "com.ubuntu.cloud:server:12.04:i386" ? | 11:34 |
| jam | Is this a case of "if local Arch == i386, boot a machine with i386" ? | 11:34 |
| jam | This is with released 1.17.5 | 11:34 |
| jam | so not trunk. | 11:34 |
| wallyworld | um | 11:34 |
| jam | wallyworld: this is custom image metadata, where the target arch should only contain amd64 | 11:34 |
| jam | but the image metadata lookup fails with: | 11:35 |
| jam | index file has no data for product name(s) ["com.ubuntu.cloud:server:12.04:i386"] | 11:35 |
| wallyworld | jam: is the client machine precise i386? | 11:35 |
| jam | wallyworld: I think we should add a line at startup of "juju" processes which logs version.Current | 11:36 |
| wallyworld | that would be helpful | 11:37 |
| wwitzel3 | natefinch: http://paste.ubuntu.com/7119308/ | 11:37 |
| fwereade | dimitern, LGTM | 11:38 |
| fwereade | perrito666, hassle everybody mercilessly | 11:39 |
| dimitern | fwereade, tyvm | 11:43 |
| dimitern | fwereade, and I did test it live again :) | 11:44 |
| axw | rogpeppe: thanks for the review. you're right about AS not being changeable, but I'm going to change the use of StateServer based on discussion with fwereade anyway | 11:48 |
| axw | rogpeppe: it'll unify some of the logic between state/non-state grouping, and it'll also be useful for ec2 AZs later | 11:49 |
| axw | (where we need to have a set of instances so we can manually distribute) | 11:49 |
| perrito666 | fwereade: what a reckles advice :p | 11:53 |
| rogpeppe | axw: sgtm | 11:54 |
| wwitzel3 | rogpeppe: in the cmd/jujud/bootstrap_test.go , there are some tests to initBootstrapCommand, which calls InitializeState. That, after our changes from yesterday calls environs.New(envCfg). It works fine when running, but the tests are now failing with an "environment is not prepared" error from environs.New | 11:59 |
| wwitzel3 | rogpeppe: any thoughts on either refactoring the code to make those tests not fail OR refactoring the tests so that the environment is prepared by the time environs.New is called? | 12:00 |
| rogpeppe | wwitzel3: i think the tests *should* be passing config from a prepared environment | 12:01 |
| rogpeppe | wwitzel3: if they're not, they need to be fixed | 12:02 |
| wwitzel3 | rogpeppe: ok, so the testConfig getting passed in to initBootstrapCommand ? | 12:03 |
| rogpeppe | wwitzel3: probably | 12:04 |
| wwitzel3 | rogpeppe: or I guess, how would I ensure that the config being passed in is from a prepared environment? Is there an example of that in a test somewhere? | 12:05 |
| wwitzel3 | rogpeppe: testConfig right now is just a a dummy.SampleConfig that gets run through the b64yaml helper. | 12:05 |
| rogpeppe | mgz: ping | 12:52 |
| rogpeppe | wwitzel3: you can get the prepared environment config from the dummy environment that is started up in most tests | 12:53 |
| rogpeppe | fwereade, jam, mgz: could we talk about API addresses for a moment or two? | 12:53 |
| fwereade | rogpeppe, got a call in 7 mins, but if it's quick, sure | 12:53 |
| rogpeppe | fwereade: https://plus.google.com/hangouts/_/canonical.com/juju-core ? | 12:54 |
| rogpeppe | fwereade: might be quickest | 12:54 |
| mgz | rogpeppe: I'll be there | 12:55 |
| rogpeppe | mgz: thanks, that's very useful | 13:10 |
| mgz | rogpeppe: that sounded good to me, | 13:10 |
| mgz | and hangouts hate me | 13:10 |
| mgz | so, carry on :) | 13:10 |
| wwitzel3 | natefinch: I pushed tests for the initiateReplicaSet and some other minor fixes to my branch | 13:15 |
| wwitzel3 | natefinch: I am still trying to figure out this environment not prepared stuff | 13:15 |
| natefinch | wwitzel3: ok, I just got back, I'll merge in and see what I can see | 13:17 |
| ahasenack | hi guys, https://bugs.launchpad.net/juju-core/+bug/1271144 is still happening, I added some log files to it | 13:29 |
| _mup_ | Bug #1271144: br0 not brought up by cloud-init script with MAAS provider <cloud-installer> <landscape> <local-provider> <lxc> <maas> <regression> <juju-core:Fix Released by rogpeppe> <https://launchpad.net/bugs/1271144> | 13:29 |
| ahasenack | can someone reopen the bug? | 13:29 |
| bodie_ | morning fine fellas | 13:29 |
| rogpeppe | ahasenack: i *thought* we'd fixed it | 13:30 |
| sparkiegeek | rogpeppe: so did we ;) | 13:30 |
| ahasenack | rogpeppe: I just tried with 1.17.5, still no br0 and lxc deployments to bootstrap fail | 13:30 |
| rogpeppe | ahasenack: but it was very hard for us to test that our fix worked | 13:30 |
| rogpeppe | ahasenack: because we don't have access to a maas to test on | 13:30 |
| ahasenack | rogpeppe: I reproduced it both with a real metal maas server, and with maas running on kvm | 13:31 |
| rogpeppe | ahasenack: do you know which revno corresponded to 1.17.5 ? | 13:32 |
| ahasenack | let me see if the branch is tagged | 13:32 |
| ahasenack | is juju-core still in lp? That's where I'm looking | 13:32 |
| ahasenack | rogpeppe: juju-1.17.5 2422 | 13:32 |
| wwitzel3 | yeah, after the fix, I wasn't able to reproduce the error on my vbox maas | 13:33 |
| ahasenack | current is 2443 | 13:33 |
| ahasenack | rogpeppe: we can help debug whatever you need | 13:33 |
| ahasenack | that was just running juju tags, I'm assuming that is the official tag for 1.17.5 | 13:33 |
| sparkiegeek | wwitzel3: did you tear down and re-deploy each time? | 13:34 |
| ahasenack | er, bzr tags | 13:34 |
| sparkiegeek | wwitzel3: emminently reproducible here :) | 13:34 |
| rogpeppe | wwitzel3: can you run with this? | 13:34 |
| jam | wwitzel3: a quick question, were you testing LXC on machine-0 or on one of the other machines? | 13:36 |
| bodie_ | hmm... i'm getting breakage on a fresh go get launchpad.net/juju-core | 13:39 |
| bodie_ | log/syslog/config.go:228: method slConfig.CACertPath is not an expression, must be called | 13:39 |
| bodie_ | and a few others | 13:39 |
| bodie_ | I don't imagine broken code would get merged | 13:39 |
| bodie_ | yeah tons of stuff is broken. the only thing I can understand that might explain this is a bad connection.... | 13:42 |
| jam | bodie_: a fresh go get probably got updates from 3rd party branches that are newer than we are using | 13:43 |
| jam | (go get grabs the tip of all dependencies) | 13:43 |
| jam | we have a dependencies.tsv to state what version we are using | 13:43 |
| jam | you should be able to "go get launchpad.net/godeps", and then do "godeps -u dependencies.tsv" | 13:44 |
| jam | which should set your tree to be the exact version of dependencies we should use for building. | 13:44 |
| bodie_ | tried that, and install still complained about tons of random breakage | 13:46 |
| bodie_ | I just double-checked on my workstation to confirm, and nothing went wrong | 13:46 |
| bodie_ | but it's totally reproducible on my remote | 13:46 |
| bodie_ | starting to think the gopher gods have it in for me | 13:46 |
| natefinch | bodie_: godeps doesn't actually update your branches, it just tells you which ones are broken. I'm not sure what the bzr command is to tell those branches to move to the right commit. | 13:47 |
| jam | bodie_: *potentially* it is a different version of go libraries? | 13:47 |
| jam | natefinch: godeps -u dependencies.tsv does the right thing *if* you've already downloaded the full histories | 13:47 |
| jam | it won't fetch new data | 13:47 |
| jam | but if you have it locally, it will set the branch to the right revision | 13:47 |
| natefinch | jam: ahh, so you need to bzr pull first, then run godeps | 13:47 |
| bodie_ | but go get already bzr pulls doesn't it? | 13:48 |
| jam | natefinch: well, potentially "go get -u" to do the recursive fetch of all dependencies, but that potentially overwrites any local commit you have in a dependency. | 13:48 |
| jam | bodie_: "go get" or "go get -u" ? | 13:48 |
| jam | -u is needed to update in place | 13:48 |
| natefinch | bodie_: go get puts you at head... we need some non-head stuff | 13:48 |
| bodie_ | go get -u -... | 13:48 |
| bodie_ | well i'm just saying, if you already go got, godeps should have the latest bzr content | 13:48 |
| jam | bodie_: right, go get -u launchpad.net/juju-core/... ; godeps -u dependencies.tsv should leave you in a working state | 13:49 |
| natefinch | jam: it seems like having head of all our branches not building is a big mistake for exactly this reason. ideally godeps should only be needed for release branches that need older stuff | 13:49 |
| jam | natefinch: you can't develop a 3rd party lib and never land it independently of trunk | 13:49 |
| natefinch | jam: oh, yeah, I see... if you're working on a feature branch, and the feature needs updates to external packages.... yeah ok. \ | 13:50 |
| bodie_ | I might have just screwed up here by starting off with go get -u -v when I never ran go get without -u | 13:50 |
| bodie_ | giving this a try | 13:50 |
| wwitzel3 | jam: I was deploying the lxc to machine-0 and I was doing a full destroy each time. I have it running now. | 13:51 |
| natefinch | jam: this is why Gustavo's versioning of packages is good... prevents exactly this problem by just making a new version that the new code references. | 13:51 |
| jam | natefinch: right, the recent one is that axw has been improving gwacl, but that meant gwacl tip was incompatible with juju-core for a while. | 13:51 |
| jam | natefinch: "ish" | 13:51 |
| jam | except Gustavo has broken us several times | 13:51 |
| jam | because he *thought* his change was compatible | 13:51 |
| niemeyer | jam: have I? | 13:51 |
| jam | when it really wasn't | 13:52 |
| jam | niemeyer: you broke the test suite | 13:52 |
| niemeyer | jam: Really!? | 13:52 |
| bodie_ | fight fight! | 13:52 |
| jam | when you added timeouts for mongo | 13:52 |
| niemeyer | jam: I'm now aware about that | 13:52 |
| jam | because we were already doing timeouts in juju-core | 13:52 |
| jam | and your timeouts and our timeouts doubled up | 13:52 |
| natefinch | jam: heh, yeah, it's definitely not always possible to know if a change will break... but there are changes you *know* will break people | 13:52 |
| niemeyer | jam: Well, sorry.. but that's a completely different kind of "breakage" | 13:52 |
| niemeyer | jam: What else have I broken? | 13:52 |
| jam | niemeyer: our test suite started failing because of an update to mgo. | 13:53 |
| jam | I don't know of a case where stuff stopped compiling | 13:53 |
| niemeyer | jam: That's good.. | 13:53 |
| niemeyer | jam: What else? | 13:53 |
| jam | niemeyer: the test suite has broken at least one other time, and we pinned to rev 241 for a while. | 13:53 |
| jam | I don't remember the exact reason offhand | 13:53 |
| niemeyer | jam: Okay, phew.. glad the "several times" was an excited overstatement then | 13:54 |
| niemeyer | jam: In terms of timeouts, why did it break again? Just out of curiosity.. | 13:55 |
| bodie_ | oh god | 13:55 |
| bodie_ | digitalocean's mirror has go1 | 13:55 |
| jam | niemeyer: we had a test that asserted we retried an appropriate number of times, but we ended up having mgo retrying on top of juju retrying | 13:55 |
| jam | something like that | 13:55 |
| * bodie_ slowly claws skin off face | 13:55 | |
| niemeyer | jam: Oh man.. okay.. | 13:55 |
| bodie_ | I'm going to go yell at someone. brb. | 13:55 |
| niemeyer | jam: That's *totally unspecified* | 13:55 |
| niemeyer | jam: If you assume mgo will connect a specific number of times, updates *may* break it | 13:56 |
| jam | niemeyer: so, more context as I'm remembering. mgo used to flood connections to Mongo, and we needed it to slow down a bit. We did the slow-down in Juju code, but then mgo started slowing down, and then we were too slow, IIRC. | 13:57 |
| niemeyer | jam: Sure, that's okay.. | 13:57 |
| niemeyer | jam: I reckon the update broke the test logic | 13:57 |
| mgz | jam: the socket timeout issue was the most painful, that gave us random failures | 13:57 |
| niemeyer | jam: But the test logic was truting on unspecified behavior.. I simply cannot promise to not touch that kind of logic | 13:57 |
| mgz | rogpeppe will remember the details | 13:58 |
| niemeyer | trusting | 13:58 |
| niemeyer | mgz: There was an actual bug there in this case | 13:58 |
| niemeyer | So yeah, that kind of breakage is on me.. will pay a beer to the debugger, sorry. | 13:58 |
| niemeyer | But that's unrelated to natefinch's point. | 13:58 |
| mgz | rogpeppe will look forward to that :) | 13:59 |
| niemeyer | mgz: Yeah, I owe him more beers than I can pay | 13:59 |
| niemeyer | So, the point remains: the versioning of stable APIs is a good method to avoid API breaks. | 13:59 |
| rogpeppe | voidspace: lp:~rogpeppe/juju-core/524-state-api-hostports | 14:01 |
| * rogpeppe agrees with niemeyer | 14:02 | |
| === marcoceppi is now known as marcoceppi-mobil | ||
| bodie_ | niemeyer++ | 14:04 |
| bodie_ | okay, now I just think I'm going insane | 14:11 |
| bodie_ | ubuntu 12.04 golang is version 1?! | 14:11 |
| bodie_ | this can't be right | 14:11 |
| niemeyer | bodie_: Why not? | 14:12 |
| natefinch | bodie_: this is why I just always build go from source, plus you need to do that to enable cross compilation anyway | 14:12 |
| bodie_ | I don't know... I guess it was my mistake to assume something would be a proper version on 12 LTS | 14:13 |
| jam | natefinch: bodie_: precise is just that old | 14:13 |
| jam | you need to "add-apt-repository ppa:juju/golang-go" | 14:13 |
| jam | and we'll give you 1.1/1.2 (I don't quite remember now) | 14:13 |
| niemeyer | bodie_: Hmm.. 1 looks like a proper version...? :) | 14:14 |
| niemeyer | bodie_: 12 was 2 years ago | 14:14 |
| natefinch | niemeyer: almost all go code written these days won't compile in go 1. | 14:15 |
| niemeyer | natefinch: These days is two years after two years ago.. :) | 14:15 |
| natefinch | niemeyer: yes... and there's been plenty of time to update the golang package to 1.1 | 14:15 |
| niemeyer | natefinch: Distributions don't get updated with new code just because it got released | 14:16 |
| niemeyer | natefinch: LTS means long term *support* | 14:16 |
| bodie_ | just asked the same question in #go-nuts and they pointed me to niemeyer's godeb. heh | 14:16 |
| niemeyer | natefinch: Security fixes and the such will get applied | 14:16 |
| bodie_ | (a couple of minutes ago) | 14:16 |
| niemeyer | natefinch: If you want new code, just update to a new release | 14:16 |
| bodie_ | I feel special | 14:16 |
| jam | niemeyer: natefinch: yeah, generally Ubuntu doesn't change your toolchain underneath you in the official archives. | 14:16 |
| jam | it would have been possible to potentially add a "golang-go-1.1" sort of package. | 14:17 |
| jam | but not to change the "golang-go" package out of 1.0 status. | 14:17 |
| bodie_ | yeah, cause that's the version it was at, at that point in time. I get that. | 14:17 |
| bodie_ | it's just annoying to realize... lol | 14:17 |
| jam | bodie_: :) | 14:17 |
| jam | well, I would have pushed to be compileab | 14:17 |
| bodie_ | spent an hour trying to figure out why everything was broken until I checked the version -_- | 14:17 |
| jam | compilable on default tools of Precise for a bit longer. | 14:18 |
| niemeyer | bodie_: That's reasonable :) | 14:18 |
| jam | but 1.1 was just too much better than 1.0 | 14:18 |
| bodie_ | I don't think making a habit of making sure you have a properly updated toolchain is a bad idea | 14:21 |
| bodie_ | I just get lazy and assume I'll have the things I need from repos | 14:21 |
| bodie_ | didn't even cross my mind to check the version until I noticed the output of go get -v was so different | 14:22 |
| natefinch | bodie_: you probably should run trusty. It's what most of us are developing on at this point | 14:22 |
| bodie_ | maybe I'll make a mention of this in CONTRIBUTING... | 14:22 |
| bodie_ | oh I'm running trusty on my workstation | 14:22 |
| bodie_ | not fun | 14:22 |
| bodie_ | this is why I'm making a 12.04 vm | 14:22 |
| natefinch | no? I have no problems with it | 14:23 |
| bodie_ | it's been literally one thing after another for an entire week including the weekend | 14:23 |
| bodie_ | maybe I'm just being dumb, maybe not everything is totally shipshape on 14.04 | 14:23 |
| bodie_ | ruling out the latter | 14:23 |
| bodie_ | you also built your mongo on a different release | 14:24 |
| bodie_ | the packaged mongo doesn't work with juju | 14:24 |
| bodie_ | the packaged juju-mongo doesn't work with juju | 14:24 |
| natefinch | bodie_: true. I guess if mongo doesn't even build on trusty, that's a problem. Though there was a fix to the mongo problem last night. You just need to install juju's mongodb on trusty now | 14:24 |
| bodie_ | GCC 4.8.2 doesn't work with mongo source | 14:24 |
| bodie_ | okay, cool :) | 14:24 |
| bodie_ | maybe I'll do that after I get this 12.04 vm running. lol | 14:24 |
| natefinch | bodie_: if you want. It seems like you're much more likely to run into problems running an old version than a new version | 14:25 |
| bodie_ | pretty stoked that Go is moving to a Go-compiled Go. pretty awesome :D | 14:26 |
| bodie_ | yeah, hopefully I can just make it work on this box. | 14:26 |
| bodie_ | otherwise to the VM I go | 14:26 |
| bodie_ | also had issues with a 13.10 vm last night .... | 14:26 |
| bodie_ | ssh breaking | 14:26 |
| bodie_ | I don't even... | 14:26 |
| bodie_ | I'm about ready to decide I should have gone into a career as a gardener | 14:27 |
| ppetraki | has anyone tried to download a bundle lately from the charm store? I've tried to export several different ones and all I'm getting is | 14:28 |
| ppetraki | envExport: | 14:28 |
| ppetraki | services: {} | 14:28 |
| ppetraki | relations: [] | 14:28 |
| ppetraki | series: precise | 14:28 |
| wwitzel3 | ahasenack, jam: run through it a few times now, still unable to reproduce with my local maas. tear and clean instance each time .. http://paste.ubuntu.com/7120061/ | 14:29 |
| ahasenack | wwitzel3: try getting rid of the squid cache | 14:29 |
| wwitzel3 | ahasenack: be happy to if I knew how | 14:31 |
| ahasenack | if [ ! -d /var/cache/squid-deb-proxy/00 ]; then | 14:32 |
| ahasenack | $SQUID -z -f /etc/squid-deb-proxy/squid-deb-proxy.conf | 14:32 |
| ahasenack | fi | 14:32 |
| ahasenack | maybe trashing that directory and restarting | 14:32 |
| ahasenack | the initscript will create it again | 14:33 |
| arosales | ppetraki: also may want to try #juju-gui | 14:37 |
| bodie_ | hmmmm | 14:37 |
| bodie_ | getting the same weirdness as I was last night on my 13.10 | 14:38 |
| bodie_ | http://paste.ubuntu.com/7120110/ | 14:38 |
| bodie_ | anyone know what might be up? I tried debugging sshd last night, but I just got more breakage | 14:38 |
| wwitzel3 | ahasenack: removed the folder and turned off the squid-deb-proxy service, giving it another try now | 14:38 |
| ahasenack | wwitzel3: I'm not sure you can leave it off, juju might expect it to be running | 14:39 |
| wwitzel3 | ahasenack: true | 14:41 |
| wwitzel3 | ahasenack: removed the squid-deb-proxy directory, rebooted, and the br0 interface comes up .. my maas is providing DNS and DHCP. Is yours not? Maybe that is the difference? | 14:47 |
| ahasenack | wwitzel3: ours is | 14:47 |
| ahasenack | wwitzel3: I tried another maas, an older one on precise (v1.4), and I also nuked the squid cache there. There it worked this time | 14:47 |
| ahasenack | wwitzel3: so we only have one maas where this is happening right now, before nuking the cache there I think we need to know what is going on | 14:48 |
| axw | bodie_: those tests don't run a real ssh executable | 14:50 |
| axw | bodie_: see storageSuite.sshCommand; it writes a fake ssh command to $PATH | 14:51 |
| wwitzel3 | ahasenack: I am able to replicate the behavior before rogpeppe's fix .. maybe if revert to prefix, bootstrap, and populate the cache, then upgrade to 1.17.5 and try to bootstrap without removing the cache .. if I can replicate it that way. | 14:51 |
| wwitzel3 | ahasenack: I will try that to see if i can get in to the same error pattern you're experiencing | 14:51 |
| ahasenack | wwitzel3: ok, I'll try again on that maas machine to see if it wasn't just an archive fluke, but without touching the squid cache | 14:51 |
| wwitzel3 | ahasenack: sounds good | 14:52 |
| axw | fwereade: https://codereview.appspot.com/77820044 - when you have some time | 14:52 |
| axw | (please) | 14:52 |
| === marcoceppi-mobil is now known as marcoceppi | ||
| sparkiegeek | wwitzel3: can you paste your cloud-init-output.log from a successful run? | 15:01 |
| sparkiegeek | (cc: ahasenack ^^) | 15:01 |
| sparkiegeek | wwitzel3: I see it trying to install bridge-utils immediately *before* an apt-get update | 15:01 |
| === marcoceppi is now known as marco-traveling | ||
| sparkiegeek | wwitzel3: also line 24 from http://paste.ubuntu.com/7120061/ looks like template fail :) | 15:02 |
| ahasenack | sparkiegeek: wwitzel3: http://pastebin.ubuntu.com/7120231/ | 15:04 |
| ahasenack | wwitzel3: that's /var/lib/cloud/instance/cloud-config.txt on the bootstrap node | 15:05 |
| bodie_ | axw..... ok... so how am I supposed to tackle this | 15:05 |
| ahasenack | it's the same on the other bootstrap node where it worked | 15:06 |
| ahasenack | could be "luck". Would love to see if it's just a missing apt-get update | 15:06 |
| ahasenack | oh, btw | 15:06 |
| ahasenack | sparkiegeek: wwitzel3: where it's failing is trusty, and where it worked for me not is precise | 15:06 |
| sparkiegeek | ahasenack: "not is precise" ? | 15:07 |
| ahasenack | s/not/now/ | 15:07 |
| ahasenack | weirdest typo that happens to everyone | 15:07 |
| sparkiegeek | :) | 15:07 |
| ahasenack | so what I mean, is that it's using different archives | 15:08 |
| axw | bodie_: sorry, I'm not sure what the root cause is. it just looks like a broken test, but I can't tell at the moment | 15:09 |
| natefinch | axw: isn't it like 2am there? | 15:09 |
| axw | natefinch: nah, just after 11pm | 15:09 |
| bodie_ | it's a broken test, I think that's as far as I've gotten. heh | 15:09 |
| axw | bodie_: I meant as opposed to broken code under test | 15:10 |
| natefinch | axw: ahh, what time zone is that? I don't have it in my world clock list | 15:10 |
| bodie_ | ko | 15:10 |
| bodie_ | ok* | 15:10 |
| bodie_ | probably caused by my custom shell | 15:10 |
| axw | natefinch: UTC+8, aka AWST | 15:11 |
| bodie_ | although that doesn't make sense either | 15:11 |
| natefinch | axw: cool. I'll add that to my list. Gotta keep track of everyone's time zone so I know when it's reasonable to bug them :) | 15:11 |
| axw | :) | 15:11 |
| perrito666 | yey, for he who loves reviews now we have a 2 for 1 great offer, review https://codereview.appspot.com/77850043 and get to also review its dependency https://codereview.appspot.com/77490043 | 15:11 |
| perrito666 | do not miss this opportunity dear juju hackers | 15:12 |
| axw | bodie_: it's explicitly executing as bash, though it's possible that your ~/.bashrc is influencing | 15:12 |
| natefinch | perrito666: I'll look | 15:13 |
| mgz | axw: that's my best guess too, but I'm a little surprised .bashrc gets triggered | 15:13 |
| axw | mgz: that's *a* bug, we should disable it wiht --norc in the test code | 15:14 |
| axw | some of the tests do that, but evidently not this one | 15:14 |
| axw | bodie_: try sticking "--norc" after "#!/bin/bash" in sshCommand, see if that does anything | 15:14 |
| bodie_ | ok | 15:15 |
| axw | fwereade: yeah, Base doesn't quite sit well with me but I can't think of a better name :\ | 15:22 |
| axw | EnvironDefaults? | 15:22 |
| rogpeppe | axw: in i prefer entities that do one thing. "Base" types can so easily become a bin to chuck all kinds of random stuff in | 15:24 |
| rogpeppe | s/in/in general/ | 15:24 |
| bodie_ | woot, got broken pipe test passing. | 15:25 |
| bodie_ | sshstorage | 15:25 |
| bodie_ | on on to the next one | 15:25 |
| axw | bodie_: how? --norc? | 15:25 |
| axw | rogpeppe: hmm... point taken. maybe it can be replaced with multiple types with default behaviour for specific things | 15:26 |
| rogpeppe | axw: that would seem better to me | 15:26 |
| rogpeppe | axw: that means that if you add some new methods, you'll need to explicitly add the default behaviour to all providers, but i actually think that's an advantage | 15:27 |
| rogpeppe | axw: as it forces providers to think about whether the default behaviour is actually appropriate for that provider | 15:28 |
| axw | yeah | 15:28 |
| axw | fair enough. I will take another look at that part tomorrow. | 15:28 |
| rogpeppe | axw: thanks | 15:30 |
| axw | I'm off, g'night folks | 15:31 |
| bodie_ | axw, I got it working by changing my shell back to bash | 15:32 |
| natefinch | bodie_: weird.... I was pretty sure rogpeppe was running uh... some non-bash shell, and he can run the tests. | 15:33 |
| bodie_ | totally weird | 15:33 |
| rogpeppe | natefinch: yeah, i do | 15:33 |
| rogpeppe | natefinch: my $SHELL | 15:34 |
| rogpeppe | % echo $SHELL | 15:34 |
| rogpeppe | /home/rog/other/plan9port/bin/rc | 15:34 |
| bodie_ | :) | 15:34 |
| bodie_ | hah! | 15:34 |
| bodie_ | I'm using oh-my-zsh | 15:34 |
| bodie_ | probably some plugin messing with things | 15:34 |
| bodie_ | I'm quickly realizing I need to be as absolutely stock as possible | 15:34 |
| bodie_ | THEN get my tests to pass | 15:34 |
| bodie_ | then start experimenting, I guess | 15:35 |
| mgz | bodie_: you're doing a fine job finding test suite isolation issues... | 15:35 |
| mgz | just not really want you want to be doing | 15:35 |
| natefinch | mgz, bodie_: a good point.... that actually is quite useful, just, yeah.... | 15:35 |
| bodie_ | even then, it surely would be better to get tests passing first so I at least know what's breaking what, and then start customizing my environment | 15:35 |
| * fwereade is disappearing for a bit to have a snippet of his public holiday | 15:35 | |
| bodie_ | heh, enjoy! | 15:36 |
| * fwereade will be around for a call in 4 hours at the very least | 15:36 | |
| bodie_ | ppa:juju/golang-go isn't found -- is it $1/golang ? | 15:51 |
| mgz | bodie_: yup | 15:52 |
| bodie_ | fyi, ppa is 1.1.2 | 15:54 |
| bodie_ | maybe THAT'S the problem with my 14.04? it's running 1.2.1? | 15:55 |
| bodie_ | oy. | 15:55 |
| bodie_ | natefinch, go version? | 15:55 |
| natefinch | bodie_: I'm running 1.3, anything >= 1.1 should be fine | 16:08 |
| natefinch | bodie_: er 1.2 | 16:08 |
| natefinch | it would be a trick to be running 1.3 | 16:08 |
| bodie_ | I would like to do that as well lol | 16:09 |
| voidspace | rogpeppe: do you have me on mute? | 16:09 |
| voidspace | rogpeppe: or can you just not hear me? | 16:10 |
| voidspace | rogpeppe: or are you ignoring me? | 16:10 |
| voidspace | hah | 16:10 |
| rogpeppe | mgz: a review for you? https://codereview.appspot.com/77820043 | 16:11 |
| wwitzel3 | ahasenack: http://paste.ubuntu.com/7120556/ , also I was not able to reproduce the error state by bootstrapping with the pre-fix version, then destroying and bootstrapping with the fix. | 16:13 |
| mgz | rogpeppe: looking | 16:13 |
| rogpeppe | mgz: ta | 16:13 |
| mgz | rogpeppe: now actually reviewing it | 16:29 |
| rogpeppe | mgz: ta#2 | 16:29 |
| mgz | rogpeppe: lgtm. | 16:34 |
| rogpeppe | mgz: ta#3! | 16:34 |
| rogpeppe | mgz: the document has only been around for a few revisions. it's not used by anything else | 16:35 |
| mgz | as in, wasn't in the last 1.17 release either? that's perfectly fine then. | 16:35 |
| natefinch | Am I the only one that thinks this formatting is terrible? | 16:41 |
| natefinch | func StartInstance( | 16:41 |
| natefinch | env environs.Environ, machineId string, | 16:41 |
| natefinch | ) ( | 16:41 |
| natefinch | instance.Instance, *instance.HardwareCharacteristics, error, | 16:41 |
| natefinch | ) { | 16:41 |
| natefinch | .... | 16:41 |
| natefinch | } | 16:41 |
| mgz | yeah, it's dire | 16:43 |
| ahasenack | wwitzel3: how could we inject some debugging into that cloud-init script? I would love to see the contents of sources.list and sources.list.d/* at that point, and/or maybe just try adding an apt-get update before it tries to install bridge-utils | 16:43 |
| ahasenack | wwitzel3: like http://pastebin.ubuntu.com/7120722/ (untested) | 16:44 |
| mgz | ahasenack: can be done. | 16:44 |
| mgz | rogpeppe: I wonder if SetAPIHostPorts should actually error out on a zero port | 16:48 |
| rogpeppe | mgz: perhaps | 16:48 |
| rogpeppe | mgz: there are so many other ways that the addresses can be wrong though, it seems like it's not really worth doing that one thing | 16:49 |
| mgz | yeah, maybe, not sure what the right level to be checking these things is really | 16:49 |
| jamespage | is it possible to change the timeout on the bootstrap command? it appears to be 10 mins right now - I can think of physical server deployments where this is not long enough :-) | 16:59 |
| natefinch | jamespage: yep, there's a config value you can set in the environments.yaml | 17:01 |
| mgz | jamespage: bug 1257649 (see status) | 17:01 |
| _mup_ | Bug #1257649: ssh timeout for bootstrap could be configurable <config> <maas-provider> <micro-cluster> <juju-core:Fix Released by dimitern> <https://launchpad.net/bugs/1257649> | 17:01 |
| jamespage | natefinch, mgz: phew | 17:01 |
| natefinch | jamespage: juju help bootstrap will show you how, I believe | 17:01 |
| === vladk is now known as vladk|away | ||
| rogpeppe | mgz, fwereade: currently we store instance ids in the control bucket. can you think of a strong reason to continue to do that for the api addresses, rather than storing the addresses themselves? | 17:09 |
| rogpeppe | if we store addresses, then we won't need an Environ to be able to work out the API server addresses - just read-only access to the control bucket | 17:09 |
| rogpeppe | although, i guess we will need to know where the control bucket is | 17:10 |
| mgz | yeah, we also don't promise that one is globably readable either I think | 17:11 |
| mgz | but putting addresses in it ('as well') should be fine as a basic optimisation | 17:11 |
| rogpeppe | mgz: i'm wondering if we need to maintain the instance ids at all | 17:11 |
| rogpeppe | mgz: ah, of course we do... | 17:11 |
| rogpeppe | mgz: for destroy-environment --force | 17:12 |
| mgz | yup. | 17:12 |
| mgz | weeeel... | 17:12 |
| mgz | you can implement that in other ways | 17:12 |
| mgz | just nuking all machines with matching names is how we generally do it I think | 17:13 |
| rogpeppe | mgz: good point | 17:13 |
| rogpeppe | mgz: so... maybe we don't need to do it | 17:13 |
| rogpeppe | mgz: it would make some logic simpler, and it opens the door to having long-term useful .jenv files without provider credentials | 17:14 |
| === DrabMakyo is now known as ShinyMakyo | ||
| === vladk|away is now known as vladk | ||
| === vladk is now known as vladk|offline | ||
| rogpeppe | here's the branch that changes the peergrouper to publish the api addresses in the state: https://codereview.appspot.com/77600048 | 17:27 |
| rogpeppe | reviews appreciated | 17:28 |
| rogpeppe | mgz, dimitern, fwereade, natefinch, wwitzel3: ^ | 17:28 |
| wwitzel3 | rogpeppe: will take a look in a bit | 17:30 |
| rogpeppe | wwitzel3: ta | 17:30 |
| perrito666 | wow, launchpad just sent me an email stating it errord trying to send an email from a comment I added, how rude | 17:34 |
| perrito666 | :p | 17:34 |
| bodie_ | oh my god | 17:34 |
| bodie_ | I think my tests may finally be passing | 17:34 |
| mgz | you're special now :) | 17:34 |
| jamespage | mgz, I'm not getting a all-machines.log aka debug-log with the MAAS provider and 1.17.5 on trusty - is that something new? | 17:39 |
| jamespage | can't find a bug that looks like it | 17:39 |
| mgz | ugh, probably, we did change several things around rsyslog | 17:41 |
| ahasenack | wwitzel3: around? | 17:41 |
| mgz | jamespage: the big change was bug 1281071 | 17:42 |
| _mup_ | Bug #1281071: juju internal use of rsyslog should use ssl/tls for aggregation <juju-core:Fix Released by axwalk> <https://launchpad.net/bugs/1281071> | 17:42 |
| wwitzel3 | ahasenack: yep | 17:42 |
| ahasenack | wwitzel3: it worked with this patch: http://pastebin.ubuntu.com/7121040/ | 17:42 |
| mgz | ...but that was in 1.17.4? | 17:42 |
| ahasenack | wwitzel3: apt-get update being the fix for me, the rest was just debugging | 17:42 |
| ahasenack | Setting up bridge-utils (1.5-2ubuntu7) ... | 17:43 |
| ahasenack | stop: Unknown instance: | 17:43 |
| ahasenack | networking stop/waiting | 17:43 |
| ahasenack | installed bridge-utils, restarted networking, br0 is up | 17:43 |
| mgz | jamespage: only obvious change since then is the adding of the rsyslog-gnutls package | 17:44 |
| wwitzel3 | ahasenack: ok, I will figure out a way to get that in there | 17:44 |
| wwitzel3 | ahasenack: have to ping some people to see if there is a prefered way to make that happen or if we can just do as you did and add the apt-get update line. | 17:45 |
| jamespage | mgz, yeah - I've seen that work OK on OpenStack with 1.17.4 | 17:45 |
| ahasenack | wwitzel3: I'm building debs in my junk ppa, others can try too (I see jamespage was affected too) | 17:45 |
| mgz | jamespage: we probably need a new bug filed, if you can find anything on a unit about rsyslog and the forwarding, that would be relevent | 17:46 |
| jamespage | mgz, doing that now | 17:46 |
| jamespage | https://bugs.launchpad.net/juju-core/+bug/1294776 | 17:46 |
| _mup_ | Bug #1294776: No debug-log with MAAS, 14.04 and juju-core 1.17.5 <juju-core:New> <https://launchpad.net/bugs/1294776> | 17:46 |
| mgz | thanks! | 17:46 |
| wwitzel3 | mgz: you have an option on the br0 fix? just added an apt-get update before the explict apt-get install bridge-utils that was rogpeppe's fix? | 17:47 |
| wwitzel3 | mgz: since we have it confirmed fixing the issue for ahasenack | 17:47 |
| wwitzel3 | s/option/opinion | 17:47 |
| rogpeppe | wwitzel3: ah, so apt-get update is necessary? | 17:48 |
| ahasenack | it was for me, not for wwitzel3 | 17:48 |
| rogpeppe | ahasenack: weird | 17:48 |
| ahasenack | could be a difference in the images that were used to install the node? | 17:48 |
| rogpeppe | ahasenack: do you know what's actually going on there? | 17:48 |
| ahasenack | rogpeppe: apt-get install bridge-utils was not finding bridge-utils, that's all I know | 17:48 |
| rogpeppe | ahasenack: ahhh | 17:48 |
| ahasenack | I checked sources.list, they look fine | 17:49 |
| ahasenack | so I tried apt-get update | 17:49 |
| ahasenack | later during the bootstrap process apt-get update is ran | 17:49 |
| rogpeppe | ahasenack: so i guess some images ship with a sources.list that doesn't include bridge-utils | 17:49 |
| ahasenack | and later again, bridge-utils is installed by something else, and then it works | 17:49 |
| ahasenack | rogpeppe: well, no, sources.list is ok, if it weren't a simple apt-get update wouldn't have fixed it | 17:49 |
| ahasenack | rogpeppe: maybe some images were generated after apt-get update was run, and some were generated without apt-get update having ever been run | 17:50 |
| ahasenack | I can check /var/lib/apt if you want, probably | 17:50 |
| rogpeppe | ahasenack: sounds worrying :-) | 17:50 |
| ahasenack | it's always good to start with a fresh apt-get update run anyway | 17:50 |
| ahasenack | you do it in the sync bootstrap | 17:51 |
| ahasenack | "just to be sure" | 17:51 |
| ahasenack | Adding apt repository: deb http://ubuntu-cloud.archive.canonical.com/ubuntu precise-updates/cloud-tools main | 17:51 |
| ahasenack | Running apt-get update | 17:51 |
| ahasenack | Running apt-get upgrade | 17:51 |
| ahasenack | more because you add another sources.list, sure | 17:51 |
| rogpeppe | to me it all seems like things to slow down our already slow instance start time, but perhaps that's just me | 17:52 |
| rogpeppe | i do see that we need to do it | 17:52 |
| rogpeppe | but i look at docker and thing "a better way is surely possible" | 17:52 |
| rogpeppe | s/thing/think/ | 17:52 |
| mgz | I'm a little nervous about fixes when we don't understand why... | 17:53 |
| mgz | cloud-init does update/upgrade as one of its earliest steps | 17:53 |
| mgz | well before we get to our runcmd | 17:53 |
| mgz | why do we need it twice? | 17:53 |
| rogpeppe | mgz: +1 | 17:54 |
| wwitzel3 | rogpeppe, mgz, ahasenack: when I look at the order that things run on my maas, we issue an apt-get update before attempting to install bridge-utils | 17:55 |
| ahasenack | mgz: where does it do it? | 17:55 |
| ahasenack | wwitzel3: where's the evidence? There are two places where bridge-utils gets installed | 17:55 |
| ahasenack | wwitzel3: in my case, the first one fails | 17:56 |
| ahasenack | my first one: | 17:56 |
| ahasenack | Reading state information... | 17:56 |
| ahasenack | E: Unable to locate package bridge-utils | 17:56 |
| ahasenack | stop: Unknown instance: | 17:56 |
| ahasenack | networking stop/waiting | 17:56 |
| ahasenack | + install -D -m 644 /dev/null /var/lib/juju/nonce.txt | 17:56 |
| ahasenack | that comes from that piece of code I patched to include apt-get update | 17:56 |
| ahasenack | it failed | 17:56 |
| ahasenack | then later, | 17:56 |
| ahasenack | The following NEW packages will be installed: | 17:57 |
| ahasenack | bridge-utils | 17:57 |
| ahasenack | and it goes on and installs it | 17:57 |
| ahasenack | I don't know what prompted that one | 17:57 |
| mgz | ahasenack: the cloud-config specifies apt_update as true | 17:57 |
| ahasenack | I think it was bootstrap | 17:57 |
| ahasenack | right, it's in that part where it installs package by package | 17:57 |
| ahasenack | git | 17:57 |
| mgz | ahasenack: check the cloud-config file in /var/lib/cloud or whereever it is | 17:58 |
| ahasenack | mgz: that apt_update true kicks in after | 17:58 |
| mgz | o_O | 17:58 |
| ahasenack | mgz: that file does not have apt-get update | 17:58 |
| ahasenack | only after I added it via the patch | 17:58 |
| mgz | oh god, is this an ssh race | 17:58 |
| mgz | wwitzel3: is the bridge-utils install done via runcmd or via the ssh-in-and-do-shit step? | 17:59 |
| ahasenack | first attempt, that failed, is via runcmd | 17:59 |
| ahasenack | it's in that patch, I pasted it | 17:59 |
| ahasenack | http://pastebin.ubuntu.com/7121040/ | 18:00 |
| ahasenack | I just added -y, and apt-get update before | 18:00 |
| mgz | runcmd should absolutely be *after* apt_upgrade | 18:00 |
| ahasenack | but apt-get install bridge-utils was there already | 18:00 |
| mgz | smoser!!!! | 18:00 |
| ahasenack | and that's the one failing | 18:00 |
| bodie_ | welp, got my tests to pass | 18:00 |
| smoser | yes | 18:01 |
| mgz | wait, I think we already had this conversation | 18:01 |
| smoser | runcmd is after cloud-init implmented apt-get install | 18:01 |
| mgz | did when runcmd happens get changed? | 18:01 |
| mgz | ahasenack is seeing odd things | 18:01 |
| ahasenack | mgz: where do we set apt_update True? | 18:02 |
| smoser | running 'apt-get update' is not "just to be sure", its really a requirement. if you want to avoid missing packages in -updates. | 18:02 |
| mgz | environs/cloudinit/cloudinit.go AddAptCommands | 18:03 |
| wwitzel3 | mgz: I have it being installed twice (bridge-utils) once during the ssh-in .. it is one of the first things to happen after SSH, then later another install is attempted, but at that point it is already at the latest version. | 18:03 |
| ahasenack | wwitzel3: you have it as part of cloud-init too | 18:03 |
| mgz | smoser: the question is, why do we have to run it *again* in runcmd when the cloud-config has apt_update set | 18:03 |
| ahasenack | wwitzel3: check /var/lib/cloud/instance/cloud-config.txt | 18:04 |
| ahasenack | mgz: how does the cloud-config "file" generated by provider/maas/environ.go get to play with the one from environs/cloudinit/cloudinit.go ? | 18:04 |
| voidspace | rogpeppe: sorry, I thought the call was at 7pm | 18:04 |
| voidspace | rogpeppe: it's a EuroPython committee meeting call | 18:04 |
| rogpeppe | voidspace: np | 18:05 |
| rogpeppe | voidspace: it's 7pm CET, i guess | 18:05 |
| voidspace | yep | 18:05 |
| ahasenack | mgz: and, remember, this issue of br0 not coming up only happens with the bootstrap node, not with other nodes that are brought up by juju later on | 18:05 |
| ahasenack | hints at different cloud-inits | 18:05 |
| smoser | mgz, well there are race conditions that are unavoidable in apt | 18:06 |
| mgz | ahasenack: well, does you /var/lib/cloud/instance/clopud-config.txt have the update key or not? | 18:06 |
| smoser | but i doubt your'e hitting that | 18:06 |
| smoser | do you see evidence of apt-get update having run in /var/log/cloud-init-output.log ? | 18:06 |
| ahasenack | mgz: without my patch, no apt-get update in it. | 18:06 |
| mgz | smoser: yeah, seems more like some odd bug in our code if you've not done anything evil recently | 18:06 |
| mgz | ahasenack: okay, that's the bug then. | 18:07 |
| smoser | mgz, trusty ? precise ? | 18:07 |
| smoser | for sure neither one *should* hvae had a regression to this affect. | 18:07 |
| ahasenack | smoser: I don't think it's an issue with cloud-init. | 18:07 |
| smoser | and look in /var/log/cloud-init.log | 18:07 |
| smoser | look for WARN in /var/log/cloud-init.log | 18:07 |
| ahasenack | there is no evidence that apt_update: True was passed to it. | 18:07 |
| smoser | well, look at /var/log/cloud-init-output.log | 18:07 |
| smoser | it will clearly show you if it was urn | 18:08 |
| ahasenack | I did | 18:08 |
| smoser | (and you'll also see mention of it in /var/log/cloud-init.log if it ran it) | 18:08 |
| ahasenack | it's clear that it is following the commands juju gave it via http://pastebin.ubuntu.com/7121040/ | 18:08 |
| ahasenack | and there was no apt-get update in that | 18:08 |
| mgz | so, somehow wayne has that, and you don't | 18:08 |
| ahasenack | luck I guess | 18:09 |
| ahasenack | also, we use the fast-path-installer, which dumps a cloud image on the node | 18:09 |
| ahasenack | wwitzel3: do you use that too? | 18:09 |
| mgz | I'm assuming he does not | 18:10 |
| wwitzel3 | ahasenack: nope | 18:10 |
| ahasenack | ok, that narrows it down a lot | 18:11 |
| ahasenack | if you use the debian installer, then it's more or less guaranteed that apt-get update was run | 18:11 |
| ahasenack | and /var/lib/apt is still "fresh" | 18:11 |
| ahasenack | whereas the fast-path-installer uses a cloud image, that I think was downloaded to the maas server only once when it was first installed | 18:12 |
| ahasenack | or it downloads it from the internet, I don't know | 18:12 |
| wwitzel3 | ahasenack: yeah the first 3 lines when I bootstrap after the ssh keys are generated are adding the cloud-tools repository, then an update, and then an upgrade | 18:12 |
| ahasenack | wwitzel3: ok, the bug happens before that | 18:12 |
| ahasenack | wwitzel3: you only see it in the cloud-init-output.log file | 18:12 |
| ahasenack | it happens before juju bootstrap ssh'ed in | 18:13 |
| ahasenack | wwitzel3: if you want, switch to the fastpath installer and try again, maybe you'll hit the bug too | 18:14 |
| wwitzel3 | ahasenack: yeah, my cloud-config.txt doesn't have a apt-get update in it, but my bridge is setup fine probably because update has already been run | 18:14 |
| ahasenack | that's my conclusion | 18:15 |
| ahasenack | mgz: ^^^ wwitzel3's cloud-config.txt doesn't have apt-get update | 18:15 |
| ahasenack | like mine | 18:15 |
| ahasenack | he got lucky :) | 18:15 |
| ahasenack | I didn't :) | 18:15 |
| mgz | okay, so we need to find out why that's not getting in, and fix it. | 18:16 |
| ahasenack | hazmat: remember the br0 not coming up bug on the bootstrap node? There were two cloud-init configs involved, right? Because br0 does get up in services deployed to new machines, just not in the bootstrap one | 18:16 |
| ahasenack | hinting that the one used for bootstrap is different than the rest | 18:17 |
| ahasenack | there was a long discussion here in the channel when that was first filed | 18:17 |
| hazmat | ahasenack, yes.. there was.. i thought it was addressed with 1.17.5 | 18:17 |
| === vladk|offline is now known as vladk | ||
| ahasenack | hazmat: nope, still happening | 18:17 |
| hazmat | ahasenack, nutshell was on bootstrap node previously bridge-utils wasn't installed because of a divergent code path for bootstrap when constructing cloud-init | 18:17 |
| ahasenack | hazmat: in my case, a simple 'apt-get update' just before 'apt-get install bridge-utils' worked | 18:18 |
| ahasenack | hazmat: so the fix was to just add "apt-get install bridge-utils" in bootstrap's cloud-init? | 18:18 |
| hazmat | ahasenack, ah.. the package is old.. | 18:18 |
| ahasenack | hazmat: right | 18:18 |
| natefinch | wwitzel3: btw, when you get time to get back on the stuff we were working on, I merged from trunk and fixed some tests. Still have the environment not prepared failure, but that's the only one left right now. | 18:18 |
| ahasenack | hazmat: I think it was assumed that apt_update: was set to True in cloud-init | 18:18 |
| ahasenack | hazmat: which it might as well be, but not for bootstrap's cloud-init | 18:18 |
| === Ursinha is now known as Ursinha-afk | ||
| hazmat | ahasenack, i'd file a bug.. i gotta run out for a min.. bbiab | 18:19 |
| ahasenack | hazmat: sure, it's filed, we were "remembering" it | 18:19 |
| ahasenack | that should be enough info to fix it | 18:20 |
| === Ursinha-afk is now known as Ursinha | ||
| wwitzel3 | natefinch: great, I will pull those changes and take a look | 18:33 |
| voidspace | g'night all | 18:33 |
| rogpeppe | i'm done too | 18:33 |
| rogpeppe | happy evenings, all | 18:33 |
| wwitzel3 | later rogpeppe | 18:35 |
| perrito666 | natefinch: could you give a second look to https://codereview.appspot.com/77850043/ ? I think it is much better now | 18:45 |
| perrito666 | tx :) | 18:45 |
| natefinch | perrito666: LGTM | 18:47 |
| perrito666 | natefinch: thank you | 18:52 |
| natefinch | wwitzel3: looks like the dummy environ expects some configuration to be set when you get the state out of it. Looks like it expects a state-id to be set | 18:53 |
| wwitzel3 | natefinch: is there an example of that? | 18:53 |
| natefinch | wwitzel3: not surte | 18:58 |
| wwitzel3 | k, I'll take a look around, brb | 18:58 |
| === vladk is now known as vladk|offline | ||
| === vladk|offline is now known as vladk | ||
| === Ursinha is now known as Ursinha-afk | ||
| === Ursinha-afk is now known as Ursinha | ||
| === vladk is now known as vladk|offline | ||
| natefinch | thumper: how's your knowledge of the dummy provider? I'm trying to update the jujud bootstrap tests to include a valid environment and it's bombing out due to an environment not being prepared, but I can't figure out how I'm actually supposed to prepare an environment for the dummy provider. | 19:47 |
| thumper | natefinch: sorry, no idea | 20:07 |
| natefinch | thumper: heh ok | 20:08 |
| bodie_ | http://paste.ubuntu.com/7121842/ | 20:16 |
| bodie_ | any ideas? | 20:16 |
| bodie_ | I'm on 14.04 | 20:17 |
| bodie_ | go1.2.1 | 20:17 |
| bodie_ | have the mongo from make install-dependencies | 20:17 |
| bodie_ | have all the right revisions via godeps -u dependencies.tsv | 20:17 |
| natefinch | bodie_: you're building trunk, right? | 20:17 |
| bodie_ | just wiped all my source code and go-got the new thing before doing all of the above | 20:17 |
| bodie_ | yeah | 20:17 |
| natefinch | bodie_: do you have mongod in /usr/lib/juju/bin/ ? | 20:18 |
| bodie_ | http://paste.ubuntu.com/7121855/ | 20:19 |
| bodie_ | so I guess I have it, but not in my PATH | 20:19 |
| bodie_ | this is just what I got via make install-dependencies | 20:19 |
| natefinch | bodie_: is this running the replicaset tests? | 20:22 |
| natefinch | bodie_: nm, I see at the bottom of the paste | 20:23 |
| bodie_ | it's not crucial, but i'd really like to get the tests passing on my local | 20:25 |
| bodie_ | do I need juju-mongo in my PATH? | 20:25 |
| natefinch | bodie_: no, I don't have it in my path | 20:26 |
| natefinch | bodie_: ha, I'm lying, I do have *a* mongod in my path, and removing it causes that exact error | 20:28 |
| natefinch | bodie_: so, yeah, add that directory to your path, should fix that error | 20:28 |
| bodie_ | nice | 20:29 |
| bodie_ | thought it looked familiar to what was going wrong when I had no mongo | 20:30 |
| perrito666 | you know, it is very hard for me to code in this and not think permanently on http://www.gamefabrique.com/games/juju-densetsu/ which I played as a kid | 20:30 |
| natefinch | that is perhaps the worst error handling I've ever seen, though. No "can't find mongo" instead a nil pointer error. | 20:30 |
| bodie_ | heh | 20:31 |
| bodie_ | yeah, the tests need some work | 20:31 |
| bodie_ | I just am keen to get rolling on actions | 20:31 |
| natefinch | perrito666: I don't know the game, but man, do those kind of graphics bring back memories | 20:31 |
| perrito666 | natefinch: It was a famicom game :) I have it ringing in my head since I heard of juju | 20:33 |
| bodie_ | yeesh. | 20:33 |
| bodie_ | *** Test killed: ran too long (10m0s). | 20:33 |
| bodie_ | FAILlaunchpad.net/juju-core/worker/uniter/debug600.003s | 20:33 |
| bodie_ | :( | 20:33 |
| thumper | sinzui: https://bugs.launchpad.net/juju-core/+bug/1294632 | 20:33 |
| _mup_ | Bug #1294632: lxc broken in trunk r2440 <bootstrap> <ci> <local-provider> <lxc> <regression> <juju-core:Triaged> <https://launchpad.net/bugs/1294632> | 20:33 |
| thumper | sinzui: have there been any packaging changes with respect to jujud? | 20:34 |
| sinzui | yeah. that is sad | 20:34 |
| thumper | sinzui: I can run the local provider locally and all good | 20:34 |
| thumper | the only difference is that I compile jujud | 20:34 |
| thumper | and the bot probably doesn't | 20:34 |
| thumper | which means it may not have a jujud locally | 20:34 |
| thumper | and hence goes looking for tools | 20:34 |
| sinzui | thumper, I did more than compile. I packaged it | 20:34 |
| thumper | sinzui: what is it testing? the package? | 20:35 |
| thumper | and which packages? | 20:35 |
| thumper | does your package contain jujud? | 20:35 |
| natefinch | bodie_: weird, those tests run in ~1 second for me. | 20:35 |
| sinzui | juju-core_1.17.6 | 20:35 |
| thumper | sinzui: the CI machine, does it do any compile from source, or just tests packages? | 20:35 |
| bodie_ | what the hell... | 20:35 |
| bodie_ | hmm, maybe there's some driver I'm missing (this is on a VM) | 20:36 |
| sinzui | thumper, I think you mean the new behaviour searches the env (PATH actually) for the jujuds and asks if they match | 20:36 |
| thumper | sinzui: no... the local provider always needs to get some tools | 20:36 |
| thumper | what it does by default is to look for a jujud that lives next to the command line juju | 20:36 |
| thumper | if it doesn't find one, it tries to compile one | 20:37 |
| sinzui | thumper, CI makes the tarball, makes the package from the tarball, then installs the package in a non-system location. Paths are updated and the tests begin. | 20:37 |
| thumper | if it can't copmile one, it looks for tools | 20:37 |
| sinzui | thumper, tests worked until the revision..tools were found before that rev | 20:37 |
| bodie_ | with juju-mongodb in my PATH now: http://paste.ubuntu.com/7121938/ | 20:37 |
| thumper | sinzui: ok, what is in the package? | 20:37 |
| sinzui | What WE RELEASE | 20:38 |
| bodie_ | (this is my 14.04 machine, not the one taking 10 minutes) | 20:38 |
| thumper | and the contents hasn't changed recently? | 20:38 |
| sinzui | thumper, nothing else failed, so we know the tools were found and also sent to the CPCs | 20:38 |
| thumper | hmm... | 20:38 |
| sinzui | packaging rules haven't changed for a few weeks | 20:38 |
| thumper | sinzui: I've assigned it to wallyworld to fix | 20:39 |
| thumper | he should have a better idea what is going on | 20:39 |
| natefinch | thumper, sinzui: this seems bad (from running the tests with the new juju-mongodb: value *mgo.QueryError = &mgo.QueryError{Code:16149, Message:"exception: cannot run map reduce without the js engine", Assertion:false} | 20:39 |
| thumper | sinzui: is there any way I can look at the CI machine? | 20:39 |
| thumper | natefinch: which test? | 20:40 |
| thumper | AFAIK, we shouldn't be using the javascript engine at all | 20:40 |
| natefinch | thumper: TestCharmStreaming from store/server_test.go | 20:40 |
| sinzui | thumper, ~/jobs/local-deploy/workspace/extracted-bin/usr/lib/juju-1.17.6/bin$ jujud version | 20:40 |
| sinzui | 1.16.6-precise-amd64 | 20:40 |
| thumper | sinzui: what about ./jujud version | 20:41 |
| thumper | it doesn't search the path, it looks for one beside the juju being executed | 20:41 |
| sinzui | ah correct, I didn't export path like the test does | 20:42 |
| natefinch | thumper: looks like mgo's MapReduce function must require the javascript engine (used in Store/store.go Store.Counters) | 20:42 |
| natefinch | sinzui: do we run the tests in CI with juju-mongodb? This seems like it should have failed there. | 20:45 |
| thumper | natefinch: CI runs precise | 20:46 |
| thumper | no juju-mongodb for precise | 20:47 |
| sinzui | thumper, you are now subscribed to lp:~sinzui/+junk/cloud-city The staging-id_rsa key will let you ssh to jenkins@54.84.137.170 | 20:47 |
| sinzui | thumper, | 20:47 |
| sinzui | PATH=/var/lib/jenkins/jobs/local-deploy/workspace/extracted-bin ./jujud version | 20:47 |
| sinzui | 1.17.6-precise-amd64 | 20:47 |
| thumper | sinzui: actually | 20:47 |
| thumper | you are right | 20:47 |
| thumper | bootstrap is broken locally | 20:47 |
| thumper | for me too right now | 20:47 |
| * thumper sends evil looks wallyworld's way | 20:47 | |
| natefinch | thumper: how can we know trusty works if we don't have CI running on it? | 20:48 |
| thumper | natefinch: we cross our fingers | 20:48 |
| natefinch | thumper: it didn't work | 20:48 |
| thumper | sinzui: can we have CI set up for trusty? | 20:48 |
| natefinch | :) | 20:48 |
| natefinch | sinzui: sorry to give you more work. I know you've been terribly bored lately. | 20:48 |
| sinzui | natefinch, mongodb-server is forced on the unittests because juju-mongodb support was reverted | 20:48 |
| sinzui | natefinch, ppc64el and arm64 always have juju-mongodb installed | 20:49 |
| sinzui | natefinch, the lxc tests to *not* use juju-mongodb...it broke deployments to the cloud | 20:50 |
| bodie_ | so is this why my 14.04 getup is failing? | 20:51 |
| bodie_ | pardon me for being obvious here, but... | 20:51 |
| bodie_ | I just want to know if I need to nuke my workstation and replace it with a bulletproof version | 20:52 |
| natefinch | bodie_: it seems juju has an incompatibility with the version of mongodb we built specifically to work with juju... | 20:52 |
| bodie_ | oy | 20:52 |
| natefinch | bodie_: when you find a bulletproof version, let us know | 20:52 |
| thumper | sinzui: so I landed a branch yesterday that forces juju-mongodb for all trusty deployments | 20:52 |
| natefinch | bodie_: we stripped out the javascript engine because it had problems on some OSes... but I guess we didn't realize one of the functions we call actually gets sent out to the javascript engine. | 20:54 |
| bodie_ | ah, right | 20:54 |
| sinzui | thumper, the aws and canonistack trusty tests are happy | 20:56 |
| thumper | interesting | 20:56 |
| thumper | sinzui: what do they do? | 20:56 |
| sinzui | Deploy a wordpress stack, then another tests deploys a stack on stable, then upgrades to the RC | 20:57 |
| natefinch | bodie_: do you want to borrow my mongod? I can zip up the whole directory and you can just drop it on disk. So long as you're on 64 bit, it should work | 20:57 |
| sinzui | thumper, I can see juju-mongodb was installed and started in th last test | 20:58 |
| bodie_ | well, it's working on my remote. I'd rather get it to legitimately work on trusty | 20:58 |
| bodie_ | ootb type thing | 20:58 |
| bodie_ | that way I know it's ... well. trusty | 20:58 |
| bodie_ | i'll just use the vps for now | 20:58 |
| natefinch | bodie_: I'm not sure there is a "legitimate" way right now, if you can't actually build mongodb on trusty | 20:58 |
| bodie_ | well there's the packaged version and the juju packaged version | 20:59 |
| bodie_ | packaged version seems to not work because it's 2.7.0-pre | 20:59 |
| thumper | sinzui: was mongodb-server installed too? | 20:59 |
| * sinzui looks | 20:59 | |
| marco-traveling | quick, what does juju set-env do | 20:59 |
| marco-traveling | because juju help is a bit ambiguous | 21:00 |
| sinzui | thumper, There is no evidence of it | 21:00 |
| thumper | hi marco-traveling | 21:01 |
| marco-traveling | hi thumper | 21:01 |
| thumper | marco-traveling: it updates the config kept in the state database | 21:01 |
| thumper | marco-traveling: that is used by everything | 21:01 |
| thumper | marco-traveling: doesn't touch anything local | 21:01 |
| marco-traveling | thumper: cool, so it doesn't update the environmet variables for a hook env | 21:01 |
| sinzui | thumper, I am confident that juju-mongodb was the only mongo installed. | 21:01 |
| natefinch | gotta run, sorry. Good luck with mongo everyone | 21:02 |
| thumper | marco-traveling: no, it updates the juju environment config | 21:02 |
| thumper | sinzui: thanks | 21:02 |
| ev | is there a preferred way for injecting environment variables such that the juju hook context can see them? I tried baking the cloud image with /etc/environment populated with http_proxy, socks_proxy, no_proxy, etc, but that didn't make its way into $charm/hooks/hooks.py | 21:06 |
| thumper | ev: not at this stage | 21:07 |
| thumper | ev: what are you after? | 21:07 |
| thumper | ev: juju is now proxy aware (for http, https, ftp and no) | 21:08 |
| thumper | ev: no socks_proxy though | 21:08 |
| ev | thumper: no_proxy support, basically | 21:08 |
| ev | I can't send traffic to swift through the proxy | 21:08 |
| thumper | ev: you can either put it in your environments.yaml, or for a running environment: 'juju set-env no_proxy=foo,bar' | 21:09 |
| ev | thumper: if you had to do this, how would you? Right now I've hacked os.environ['http_proxy'] into every single charm, which is...less than ideal. | 21:09 |
| ev | wait what?! How do I get this in my environments.yaml, and how did I miss this option | 21:09 |
| thumper | ev: 'juju set-env http_proxy=http://myproxy.com' | 21:10 |
| thumper | ev: because I'm terrible at documentation... | 21:10 |
| thumper | ev: it is very recent... | 21:10 |
| thumper | ish | 21:10 |
| * thumper goes to look at docs | 21:10 | |
| thumper | ev: sorry about that... | 21:11 |
| thumper | ev: environment now has: | 21:12 |
| thumper | http-proxy, https-proxy, ftp-proxy, no-proxy, apt-http-proxy, apt-https-proxy, apt-ftp-proxy | 21:12 |
| thumper | the apt values default to the plain values (ie. apt-http-proxy defaults to http-proxy) | 21:12 |
| thumper | but can be overridden by themselves | 21:12 |
| thumper | eg. my local provider has "apt-http-proxy: http://10.0.3.1:8000" for squid deb proxy | 21:13 |
| ev | thumper: that's excellent | 21:13 |
| ev | thank you | 21:13 |
| thumper | np | 21:13 |
| waigani | thumper: https://codereview.appspot.com/77930043 | 21:13 |
| bodie_ | hmmm | 21:23 |
| bodie_ | getting "no reachable servers" in my store_test | 21:23 |
| bodie_ | anyone know what might cause that? | 21:24 |
| bodie_ | (i.e. is the store down?) | 21:24 |
| thumper | intermittent test failure | 21:24 |
| thumper | we all get that | 21:24 |
| thumper | really annoying | 21:24 |
| thumper | feel free to fix | 21:24 |
| thumper | will trade fix for beer | 21:24 |
| * bodie_ puts up an ultraviolet bug-zapper lamp. | 21:25 | |
| bodie_ | apparently that worked | 21:36 |
| bodie_ | passed this time | 21:36 |
| thumper | I did say it was intermittent | 21:37 |
| sinzui | thumper, is you scratch out some notes about proxy changes, I will write it up for the release notes and put it into the unstable docs | 21:38 |
| sinzui | s/is/if/ | 21:38 |
| thumper | sinzui: ok, ta | 21:38 |
| ev | thumper: is there any way to tell juju to not clean up the bootstrap node on failure? I constantly find myself wanting this. | 21:47 |
| ev | so I can peek at its brain | 21:48 |
| thumper | ev: no... not that I'm aware of | 21:48 |
| ev | rubbish | 21:50 |
| ev | so if I'm seeing: | 21:55 |
| ev | Installing package: git | 21:55 |
| ev | 2014-03-19 21:54:31 ERROR juju.provider.common bootstrap.go:127 bootstrap failed: rc: 1 | 21:55 |
| ev | Stopping instance... | 21:55 |
| ev | do I have any recourse? | 21:55 |
| thumper | ev: which provider | 21:55 |
| ev | openstack | 21:56 |
| thumper | does it need an apt proxy? | 21:56 |
| ev | it has one set | 21:56 |
| ev | hmm, maybe this is on my end | 21:56 |
| thumper | ev: can you pastebin a 'juju get-env' | 21:57 |
| ev | let me just make sure spawning an instance normally works | 21:57 |
| ev | thumper: that wont work without a bootstrap :) | 21:57 |
| thumper | haha, yeah | 21:58 |
| thumper | ev: environtments.yaml config values? | 21:58 |
| wallyworld | thumper: https://codereview.appspot.com/77960043 | 22:04 |
| * thumper looks | 22:04 | |
| * wallyworld goes to make breakfast now he has fixed his regression | 22:05 | |
| waigani | thumper: https://codereview.appspot.com/77970043 | 22:07 |
| ev | thumper: it was on my end - dns was missing | 22:11 |
| thumper | ev: ok, cool | 22:12 |
| ev | suspected something was up when apt /immediately/ returned 1, instead of hung like when it's not going via the proxy :) | 22:12 |
| ev | woooo! "starting juju machine agent" | 22:12 |
| thumper | wallyworld: https://bugs.launchpad.net/juju-core/+bug/1285923 is that part of your branch? | 22:15 |
| _mup_ | Bug #1285923: provider/ec2: tests fail expecting a ppc64 image for precise <ec2-provider> <ppc64el> <juju-core:Triaged by thumper> <https://launchpad.net/bugs/1285923> | 22:15 |
| thumper | ev: \o/ | 22:15 |
| wallyworld | thumper: not that i know of, i'd need to look | 22:16 |
| wallyworld | could be | 22:16 |
| thumper | ok | 22:17 |
| bodie_ | d'you guys have an opinion on goimports? | 22:17 |
| bodie_ | I like being lazy, but I don't like being wrong and lazy | 22:18 |
| wallyworld | thumper: the arch bit is fixed. i wasn't aware of the series issue. i'll have to read that more closely to grok what the claimed issue is | 22:18 |
| thumper | wallyworld: ok | 22:18 |
| thumper | bodie_: I don't use it | 22:18 |
| thumper | not yet anyway | 22:19 |
| bodie_ | is it necessary to use this PPA to get the lbox tool? | 23:22 |
| bodie_ | ppa:gophers/go | 23:22 |
| bodie_ | when I try to apt-get update, it looks like it's down | 23:22 |
| bodie_ | can't I just go get launchpad.net/lbox ? | 23:23 |
| bodie_ | giving that a shot to see if it works | 23:25 |
| davecheney | bodie_: it's just a command | 23:31 |
| davecheney | use go get | 23:31 |
| bodie_ | yeah, that worked | 23:32 |
| bodie_ | I'm trying to use lbox now -- it's asking me to navigate to my browser, but I'm on a headless remote. is this a problem? | 23:32 |
| davecheney | yes that will be a problem | 23:33 |
| davecheney | you could try | 23:33 |
| bodie_ | like, is there any other way I can do it, or should I just rsync it to my local and do it from here? | 23:33 |
| bodie_ | bah.... | 23:33 |
| davecheney | bzr login | 23:33 |
| davecheney | you need an OAUTH key from LP | 23:33 |
| bodie_ | yeah | 23:33 |
| bodie_ | hm | 23:33 |
| bodie_ | bzr unknown command login | 23:33 |
| bodie_ | does that require cobzr? | 23:33 |
| bodie_ | er, no | 23:34 |
| bodie_ | well, can I just push my branch and then use launchpad.net to propose it? | 23:34 |
| bodie_ | bzr lp-propose-merge? | 23:36 |
| davecheney | bodie_: try copying the file ~/.lpad_oauth | 23:40 |
| bodie_ | from where to where? It's not on either machine | 23:41 |
| bodie_ | can't I at least use bzr to push to my account on launchpad.net and then merge it with my web browser on my workstation? | 23:42 |
| bodie_ | er, "propose" the merge | 23:42 |
| bodie_ | oh LOL it lets me view the site in my terminal | 23:46 |
| bodie_ | excellent | 23:46 |
| bodie_ | w3m | 23:48 |
| davecheney | sensible-browser for the win | 23:48 |
| bodie_ | hmmm | 23:49 |
| bodie_ | works til I get to here | 23:49 |
| bodie_ | bodie@juju-dev:~/go/src/launchpad.net/juju-core$ bzr launchpad-login | 23:50 |
| bodie_ | binary132 | 23:50 |
| bodie_ | bodie@juju-dev:~/go/src/launchpad.net/juju-core$ bzr register-branch fix-bson-references | 23:50 |
| bodie_ | launchpad.net password for binary132@gmail.com: : | 23:50 |
| bodie_ | bzr: ERROR: Invalid url supplied to transport: "https://binary132%40gmail.com:<my password in plaintext! yikes>@xmlrpc.launchpad.net/bazaar/": nonnumeric port | 23:50 |
| bodie_ | pardon my paste | 23:50 |
| davecheney | bodie_: that isn't your LP login | 23:51 |
| davecheney | thats your gmail login, yes ? | 23:51 |
| bodie_ | well, I used bzr launchpad-login to set it as binary132, which is my lpad login | 23:56 |
| bodie_ | so I'm not quite sure why it's asking for "launchpad.net password for binary132@gmail.com" | 23:56 |
Generated by irclog2html.py 2.7 by Marius Gedminas - find it at mg.pov.lt!