| pmatulis | is there some trickery in specifying a public SSH key with 'juju authorised-keys add <key>' ? | 00:20 |
|---|---|---|
| pmatulis | i'm seeing this: http://paste.ubuntu.com/12242048/ | 00:20 |
| beisner | fyi thedac, merge collision alert on rmq. something landed, you'll need to rebase and resolve conflicts in your cluster wip. | 00:21 |
| lazyPower | pmatulis: should be juju authorized-keys add `cat ~/.ssh/somekey.pub` | 00:36 |
| lazyPower | the import should have worked w/ LP data as well considering its using ssh-import-id | 00:37 |
| lazyPower | that broke for you? :-\ | 00:37 |
| lazyPower | pmatulis: do you see the key you attempted to add with juju authorized-keys list? | 00:37 |
| pmatulis | lazyPower: yeah, lol, you have to paste the key at the terminal. just figured that out | 00:38 |
| pmatulis | lazyPower: should really allow to point at a key file | 00:38 |
| lazyPower | pmatulis: sounds like an excellent bit-sized bug | 00:40 |
| beisner | thedac, commented @ bug 1486177. just clarifying that today's rmq merge doesn't resolve the cluster race bug. while it may resolve issues in some scenarios, not in ours. | 00:40 |
| mup | Bug #1486177: 3-node native rabbitmq cluster race <amulet> <backport-potential> <openstack> <uosci> <rabbitmq-server (Juju Charms Collection):Confirmed for thedac> <https://launchpad.net/bugs/1486177> | 00:40 |
| beisner | wolsen, niedbalski fyi ^ | 00:40 |
| wolsen | beisner, is this the one I did a review on earlier? | 00:46 |
| * wolsen checks | 00:46 | |
| wolsen | beisner, yargh, I caused that merge conflict merging another change | 00:48 |
| wolsen | beisner, the merge conflict isn't bad - I can take care of it in the merge | 00:49 |
| wolsen | beisner, yep easy-peasy | 00:52 |
| beisner | wolsen, actually i'd like to get thedac 's other cluster fixes rebased, as i'm syncing that into my amulet-test-refactor branch to confirm resolution before merge. but yeah, the conflict doesn't look to bad. but i'm a bit blocked until we do that < | 00:58 |
| wolsen | beisner, heh | 00:58 |
| wolsen | beisner, just pushed | 00:58 |
| wolsen | :) | 00:58 |
| beisner | or that! :-D | 00:58 |
| * beisner rebases | 00:58 | |
| wolsen | thedac ^^ | 00:58 |
| pmatulis | lazyPower: your request is my command: https://bugs.launchpad.net/juju-core/+bug/1490789 | 01:23 |
| mup | Bug #1490789: [juju authorised-keys add] cannot specify a key file <juju-core:New> <https://launchpad.net/bugs/1490789> | 01:23 |
| lazyPower | pmatulis: <3 | 01:24 |
| beisner | ping wolsen | 01:38 |
| beisner | wolsen, niedbalski, thedac - i'd recommend reverting rmq/next 107 and 108: http://paste.ubuntu.com/12242485 | 01:42 |
| beisner | ie. rmq/next is completely bust atm | 01:45 |
| beisner | whereas at r106, it was only ~50% bust, and only when clustering | 01:46 |
| beisner | then i'd like to get thedac 's fixes landed, then rebase my amulet-test-refactor branch, get those landed so we can reliably test proposals, and go from there. | 01:47 |
| beisner | ie. please revert and freeeeeeeze rmq/next, pending resolution of CRIT bug 1486177 | 01:48 |
| mup | Bug #1486177: 3-node native rabbitmq cluster race <amulet> <backport-potential> <openstack> <uosci> <rabbitmq-server (Juju Charms Collection):Confirmed for thedac> <https://launchpad.net/bugs/1486177> | 01:48 |
| wolsen | beisner, done | 05:05 |
| === lukasa is now known as Guest4580 | ||
| jamespage | gnuoy, landed the hugepages branch for nova-compute | 09:35 |
| gnuoy | jamespage, fantastic, thanks | 09:35 |
| gnuoy | cd - | 09:36 |
| === lukasa_ is now known as lukasa | ||
| g3naro | To get the ID of an AMI that's configured to run as a NAT instance, use a command to describe images, and use filters to return results only for AMIs that are owned by Amazon, and that have the amzn-ami-vpc-nat string in their names. The following example uses the AWS CLI: | 11:23 |
| beisner | wolsen, thx | 13:07 |
| jcastro | marcoceppi: ok so do you want to run that session? | 13:41 |
| marcoceppi | jcastro: sure | 13:52 |
| === scuttlemonkey is now known as scuttle|afk | ||
| beisner | dosaboy, fyi i flipped bug 1486177 back out of Fix Committed, see comment. | 14:25 |
| mup | Bug #1486177: 3-node native rabbitmq cluster race <amulet> <backport-potential> <openstack> <sts> <uosci> <rabbitmq-server (Juju Charms Collection):Confirmed for thedac> <https://launchpad.net/bugs/1486177> | 14:25 |
| dosaboy | beisner: ack, lets slip that log addition in | 14:55 |
| beisner | dosaboy, the plan is to defer that name resolution fix until after the total-cluster-fail fix and associated tests have landed, so that reviewers have reliable tests to confer upon. we need to unblock basic functionality first. | 14:57 |
| thedac | beisner: my branch is updated lp:~thedac/charms/trusty/rabbitmq-server/native-cluster-race-fixes including a hopeful (but untested fix for http://paste.ubuntu.com/12242485/) Before I create an MP we should run some tests | 14:57 |
| beisner | thedac, thanks. i'll merge that into my amulet-test-refactor branch. | 14:57 |
| beisner | thedac, the errors in that paste should have disappeared as rmq/next was reverted. are you still seeing those? | 15:00 |
| thedac | no, but it looked like my change that originally caused it | 15:01 |
| beisner | thedac, nope i believe it was r107, as that removed this: if config('prefer-ipv6') and rdata.get('hostname'): | 15:03 |
| thedac | oh?!!! ok | 15:03 |
| beisner | and just ran ahead without the conditional | 15:03 |
| thedac | agreeed | 15:03 |
| beisner | thedac, but i'd say that your change to rdata.get('private-address') is safer anyway ;-) | 15:05 |
| thedac | yeah, it will do no harm | 15:06 |
| === CyberJacob is now known as zz_CyberJacob | ||
| === cmagina is now known as cmagina_ | ||
| === cmagina_ is now known as cmagina | ||
| thedac | dosaboy: gnuoy tells me you might be working on another rabbitmq issue. Is there a bug report for that? | 15:38 |
| kwmonroe | hey folks, if i have a subordinate relationship between my tomcat war and tomcat (via the implicit juju-info interface), will there be a "juju-info-relation-departed" hook that fires if i destroy my subordinate service? | 15:44 |
| rick_h_ | lazyPower: marcoceppi can a subordinate get juju config data from the charm it's related to/on top of? | 15:45 |
| lazyPower | rick_h_: only if its sent over the wire via relation-set | 15:46 |
| rick_h_ | lazyPower: ok, and if that config changes? | 15:46 |
| rick_h_ | lazyPower: they'd need to know to trigger something on the subordinate? | 15:46 |
| lazyPower | it should be relation-set -r #relid foo=bar | 15:46 |
| marcoceppi | rick_h_: nope | 15:46 |
| rick_h_ | ugh | 15:46 |
| lazyPower | which will cause the relationship-changed sequence to cycle | 15:46 |
| * marcoceppi ninja'd by lazyPower | 15:47 | |
| marcoceppi | rick_h_: well | 15:47 |
| lazyPower | rick_h_: i have an example of this in docker / flannel-docker charms, where we send information out of context (or out of band) | 15:47 |
| rick_h_ | lazyPower: marcoceppi the idea being the subordinate wants to konw if the main charm is configured to pull from deb, source, etc | 15:47 |
| marcoceppi | actually, envermind, juju-run doesn't work in a hook context | 15:47 |
| rick_h_ | lazyPower: marcoceppi and if that changes, the subordinate also needs to do some updating to keep in sync | 15:47 |
| marcoceppi | rick_h_: gotta use the relationship wire | 15:47 |
| lazyPower | rick_h_: thats possible to do | 15:47 |
| lazyPower | you just have to proxy that information through the relationships | 15:47 |
| rick_h_ | lazyPower: ok cool ty | 15:48 |
| lazyPower | rick_h_: lmk if you want examples, i have at least one that i'm aware of | 15:50 |
| rick_h_ | lazyPower: cool, jcsackett and bac will be interested | 15:50 |
| bac | lazyPower: i may have missed it but if you have examples of the technique you explained to rick_h_, please let me know. | 16:00 |
| lazyPower | bac: https://github.com/chuckbutler/flannel-docker-charm/blob/master/playbooks/network-relation-changed.yaml#L21 | 16:01 |
| bac | lazyPower: ty! | 16:01 |
| === gaughen_ is now known as gaughen | ||
| lazyPower | bac: context being, the network relation is responsible for sending networking config data from flannel to the Docker daemon to configure SDN. flannel-docker should not modify the DOCKEROPTS file, hense we send it (potentially out of band) to docker to manage that files configuration | 16:02 |
| lazyPower | bac: the receiving end is a little convoluted with ansible handlers, so if you need a full stack breakdown of how thats getting used feel free to ping and i'll try to explain it as best i can :) | 16:03 |
| ntpttr | Hey everyone, I'm trying to configure my Juju and Maas to work behind a corporate proxy, and for some reason I'm getting "could not access file 'provider-state': get http://10.14.4.1/MAAS/api/1.0/file/provider-state/: http: error connecting to proxy http://http://myproxy:port: dial tcp: lookup http: no such host". The problem seems to be that http:// is showing up twice in the proxy URL, but in .juju/environments.yaml I configured http-proxy, https-proxy, ftp | 16:29 |
| ntpttr | I rebooted my machine and my problem is different now, running 'juju --debug --show-log status' gives me "unable to connect to environment "maas"", and says it got service unavailable back from the server with the operation timing out | 17:05 |
| ntpttr | I do have the IP of the MAAS server listed in my 'no-proxy' section of the environments.yaml file | 17:06 |
| lazyPower | ntpttr: can you reach the state server normally? | 17:07 |
| lazyPower | i mean via ping/ssh et-al | 17:07 |
| ntpttr | yes I can ping it | 17:07 |
| ntpttr | lazyPower: And it was working without any proxy when I was testing it at home | 17:08 |
| lazyPower | Can you ssh to the node, and check if the jujud process is running, no UFW | 17:08 |
| ntpttr | Right now there's no node to ssh to, I'm trying to bootstrap an environment to a node | 17:09 |
| ntpttr | lazyPower: I'm using an Ubuntu Orange Box and am logged in to the controller node | 17:09 |
| lazyPower | hmm, i haven't had much hands on with the orange box but i know it takes some special considerations | 17:10 |
| lazyPower | so you're trying to check status on an environment that hasn't successfully bootstrapped? | 17:10 |
| ntpttr | lazyPower: Yes, well in the default examples that come with the orangebox it has a bootstrap script that first checks if there's an existing deployment by calling status, and then runs bootstrap on a VM | 17:12 |
| lazyPower | ntpttr: this leads me to believe there's a stale .jenv in your control node that thinks the environment is bootstrapped, but th eenvironment is inconsistent with that. | 17:12 |
| ntpttr | lazyPower: if there is, it isnt in ~/.juju/environments, is there another place it could be? | 17:13 |
| lazyPower | is th eorange box using encrypted home? | 17:13 |
| lazyPower | if so i think it maps to /var/juju or something similar. | 17:14 |
| ntpttr | lazyPower: I don't think so, that directory isn't there | 17:14 |
| lazyPower | ntpttr: this is the error you'll see when there's no .jenv and you attempt to run status on an environment | 17:16 |
| lazyPower | http://paste.ubuntu.com/12246973/ | 17:16 |
| lazyPower | if you're seein gsomething else, there's a stale environment configuration somewhere on disk, and i'm sorry i'm not as familiar with the orangebox setup as I should be to help here. | 17:17 |
| ntpttr | lazyPower: All right, running that command now - have to wait a bit for it to time out | 17:17 |
| lazyPower | kwmonroe: do you have a second to lend your orange box experience here? | 17:17 |
| lazyPower | kwmonroe: specifically, do you know what ntpttr is referring to about the setup scripts/examples on the OB? | 17:18 |
| ntpttr | lazyPower: So I did get the first part of that error, saying it's unable to connect and that I should check credentials or use juju bootstrap, but in the erro rdetails I still get 'could not access file provider-state: gomaasapi: got error back from server: 503 Service Unavailable', and then a lot of HTML that says a communication error occurred: operation timed out | 17:20 |
| lazyPower | can you pastebin that error output? | 17:20 |
| ntpttr | looking at the code from the orangebox setup script, all it's doing is running juju status and then juju bootstrap | 17:20 |
| lazyPower | it sounds like it attepted to contact maas and maas 503'd on you. | 17:20 |
| ntpttr | sure one sec | 17:20 |
| ntpttr | pastebin.com/pYzBvHwS | 17:21 |
| ntpttr | http://pastebin.com/pYzBvHwS | 17:21 |
| ntpttr | here's my environments.yaml http://pastebin.com/Xb0ejV4Y | 17:24 |
| lazyPower | ntpttr: can you pull up the MAAS web ui? | 17:26 |
| ntpttr | Yes, I've set the proxy and upstream DNS from there | 17:27 |
| kwmonroe | lazyPower: i believe ntpttr is talking about this bootsrap script | 17:28 |
| kwmonroe | http://bazaar.launchpad.net/~orange-box-examples/orange-box-examples/trunk/view/head:/bin/orange-box-bootstrap-juju | 17:28 |
| kwmonroe | ntpttr: can you verify that node0vm0 is running (it's easy to see if you launch Virtual Manage Manager) | 17:29 |
| lazyPower | ntpttr: i'm not sure why its giving you a 503 then. I'm out of my depth aside from checking basics like ensuring your api key hasn't changed (which should be a 401 response) and running destroy-environment --force -y before attempting a re-bootstrap | 17:29 |
| kwmonroe | we've seen a problem where maas isn't correctly booting KVM instances (like node0vm0) | 17:29 |
| lazyPower | kwmonroe: thanks for lending a hand :) | 17:29 |
| kwmonroe | so you may just have to manually power that sucker on (if it's off) | 17:29 |
| kwmonroe | alternatively ntpttr, you can alter line 31 of that o-b-bootstrap-juju script to "juju bootstrap --to nodeX.maas". that'll try and bootstrap a physical machine instead of a VM. | 17:31 |
| kwmonroe | (replace X with whatever number you want to be your bootstrap node) | 17:32 |
| ntpttr | kwmonrow: node0mv0 is currently "ready", should I power it on before trying to bootstrap? | 17:32 |
| kwmonroe | yeah ntpttr, give that a try | 17:32 |
| kwmonroe | maas may report that it's sent the "power on" command, but you should fire up Virt Machine Manager and make sure it actually starts. | 17:34 |
| ntpttr | kwmonroe: okay, thanks for the help! I'll let you know if that works and try bootstrapping to a physical node if not | 17:34 |
| kwmonroe | np ntpttr, i'll keep fingers crossed for ya :) | 17:34 |
| ntpttr | kwmonroe: Should I also comment out the 'juju status' check in orange-box-bootstrap-juju? It's timing out running that command and exiting | 17:35 |
| ntpttr | kwmonroe: hmm it looks like it's still timing out on bootstrapping both to the powered on vm and a physical machine | 17:40 |
| kwmonroe | ntpttr: you could.. or you could run "juju destroy-environment maas --force" (that'll just remove any maas.jenv file that may or may not exist -- dont' worry, it won't change your maas definition in environment.yaml) | 17:40 |
| kwmonroe | hmm.. that's no good ntpttr. | 17:41 |
| ntpttr | kwmonroe: do you think it could be my network settings in maas somehow? Maybe my upstream DNS? | 17:41 |
| ntpttr | kwmonroe: It's even timing out on the juju destroy-environment command | 17:42 |
| kwmonroe | ntpttr: i'm scratching my head now.. your proxy settings in the env.yaml look fine to me. and i'm not sure what "destroy-environment --force" would be trying to do over the network at that point. | 17:46 |
| sparkiegeek | drop the http:// from the value of the http_proxy | 17:47 |
| kwmonroe | yeah sparkiegeek? the docs show the prefix (https://jujucharms.com/docs/stable/howto-proxies), but i guess removing that is worth a shot. | 17:49 |
| kwmonroe | ntpttr: will you pastebin the output of 'sudo virsh list --all' | 17:50 |
| ntpttr | sparkiegeek kwmonroe: Yeah I have tried that, since the error I was getting before rebooting my machine was that it was trying to connect to http://http://myproxy:port, but after rebooting for some reason it's all changed and it's timing out instead. I just tried it again with the same result | 17:50 |
| ntpttr | kwmonroe: Sure one second | 17:51 |
| ntpttr | http://pastebin.com/mVWePqwM | 17:53 |
| ntpttr | kwmonroe: Hold up I just restarted my computer again and it connected to maas when I ran bootstrap | 17:54 |
| ntpttr | kwmonroe: I rebooted after removing my proxy settings from /etc/environment and ~/.bashrc, it seems like maybe that was causing an issue | 17:54 |
| ntpttr | kwmonroe: I'll let you know if the bootstrap completes without any errors | 17:55 |
| kwmonroe | interesting ntpttr -- so environments.yaml is now the only place the proxy info is being set? | 17:55 |
| ntpttr | kwmonroe: There and in the network configuration portion of the Maas settings page. It's looking like the bootstrap is timing out connecting to streams.canonical.com now though | 17:56 |
| kwmonroe | grrr | 17:57 |
| ntpttr | Oh it's back to saying error connecting to 'http://http://myproxy:port', although I did remove http:// from my environments.yaml | 17:59 |
| ntpttr | kwmonroe: The funny thing is it gives that error even if I comment the proxy settings out entirely | 18:00 |
| kwmonroe | ntpttr: i think that means there's a .jenv that's overriding your environments.yaml.. see what you get with this: "grep -r proxy ~/.juju/environments*" | 18:03 |
| ntpttr | kwmonroe: That command comes up blank, that folder is empty | 18:04 |
| kwmonroe | hm.. lazyPower, is there some way to tell where juju would keep .jenvs if not in ~/.juju/environments? i feel like ntpttr has a mysterious .jenv with proxy settings that are messing with us. | 18:06 |
| lazyPower | kwmonroe: aside from mlocate's updatedb && locate maas.jenv | 18:06 |
| lazyPower | i know encrypted home moves it somewhere, like /var/juju/$user | 18:06 |
| lazyPower | but thats the only instance i can think of where it would do that | 18:07 |
| kwmonroe | ntpttr: can you give that try (sudo updatedb && sudo locate maas.jenv)? | 18:08 |
| ntpttr | kwmonroe: Sure | 18:08 |
| ntpttr | kwmonroe: it came back blank | 18:09 |
| lazyPower | kwmonroe: there are ENV vars it can read too right? | 18:09 |
| lazyPower | kwmonroe: juju set-env (which i' mpretty sure requires a bootstrapped env), but i think there are also some environment vars juju reads that it may be colliding with | 18:10 |
| kwmonroe | ntpttr: i know you removed the proxy bits from /etc/environment and .bashrc, but let's see if they're still around: env | grep -i proxy | 18:14 |
| ntpttr | kwmonroe: heey it looks like they are still around somewhere | 18:15 |
| lazyPower | ntpttr: try unsetting the vars and re-bootstrapping | 18:15 |
| lazyPower | then give it a go | 18:15 |
| kwmonroe | ntpttr: unset http_proxy https_proxy no_proxy | 18:16 |
| ntpttr | kwmonroe: Okay bootstrapping has started fingers crossed | 18:17 |
| kwmonroe | alternatively ntpttr, you may be able to re-set them to the right value (http://myproxy:port vs http://http://myproxy:port) in /etc/environment or ~/.bashrc.. though i'm not sure where they're coming from at the moment.. | 18:20 |
| ntpttr | kwmonroe: I think they were still there because I had edited and env_keep line to /etc/sudoers | 18:21 |
| ntpttr | kwmonroe: hmm it looks like bootstrapping is still timing out on connecting to streams.canonical.com :/ | 18:22 |
| ntpttr | kwmonroe: It seems like the env variables need to be set to connect there but they cant be set because it'll cause the double http:// problem | 18:22 |
| kwmonroe | ntpttr: when you saw them set in the environment with "env | grep -i proxy", where they set to http://http://? | 18:23 |
| ntpttr | kwmonroe: Oh wow it looks like they are... I have no idea how that happened I feel dumb right now hah | 18:25 |
| kwmonroe | :) | 18:25 |
| kwmonroe | so rather than unsetting them ntpttr, let's try setting them to the right values (single http prefix), then re-bootstrapping | 18:26 |
| ntpttr | kwmonroe: Do you know if it's okay for them to have cidr prefixes in the env variables? | 18:26 |
| ntpttr | kwmonroe: for no_proxy specifically | 18:27 |
| kwmonroe | not sure ntpttr, everything i know about proxies is being learned by feverishly typing into google at this very moment. | 18:27 |
| ntpttr | kwmonroe: uh oh now with the proxies set back the correct way I'm getting the timeout problem connecting to maas again | 18:27 |
| ntpttr | kwmonroe: that's why I asked about CIDR, it seems like the no_proxy setting for the MAAS host isn't working maybe | 18:28 |
| kwmonroe | ntpttr: looks like no cidr for no_proxy: http://unix.stackexchange.com/questions/23452/set-a-network-range-in-the-no-proxy-environment-variable | 18:29 |
| kwmonroe | ntpttr: let's try something crazy. on the orange box, there's a squid3 proxy (indepent of your external proxy) that serves content to the maas nodes. let's see if that has crashed: ps -ef | grep squid | 18:34 |
| ntpttr | kwmonroe: I've been trying to add to the no proxy list the way it said in your link, and now running that command says that the argument list is too long, I think because there's so many values in no_proxy | 18:35 |
| ntpttr | kwmonroe: i unset it, and running that command shows squid running | 18:36 |
| kwmonroe | ntpttr: let's kick it anyway: sudo service squid-deb-proxy stop; sudo service squid3 restart | 18:37 |
| ntpttr | squid-deb-proxy is an unrecognized service, but squid3 restarted | 18:39 |
| kwmonroe | hm, ok ntpttr. retry the bootstrap | 18:40 |
| ntpttr | kwmonroe: it got to the deploying stage, so that's good! | 18:43 |
| kwmonroe | has it gotten that far before ntpttr? | 18:44 |
| ntpttr | kwmonroe: No, it timed out on connecting to external hosts before | 18:44 |
| kwmonroe | cool ntpttr! | 18:45 |
| ntpttr | kwmonroe: Thank you a bunch for your time, you've been really helpful. You too lazyPower. | 18:49 |
| lazyPower | np ntpttr | 18:52 |
| lazyPower | sorry I didn't have better info for you :) but happy to help. If you run into any further problems, I suggest mailing the list - juju@lists.ubuntu.com with the problem scenario and someone in the EU timezones with more experience should be able to lend a hand. | 18:53 |
| ntpttr | lazyPower: All right, thank you for the advice! | 18:54 |
| kwmonroe | so ntpttr, while you're bootstrapping, here's what i think may have gone wrong.. the squid3 service that serves the maas nodes came up and read the invalid environment proxy vars (http://http://), and even though you unset/reset them later, that process was already using the bad values and couldn't serve the content needed to bootstrap to node0vm0. restarting squid3 after the envars were corrected meant the maas proxy | 18:55 |
| kwmonroe | could then correctly talk to your proxy. | 18:55 |
| kwmonroe | and least that's what i'm hoping. i'll feel better when your bootstrap succeeds! | 18:55 |
| ntpttr | kwmonroe: okay, thank you! The bootstrap did succeed. Now going to node0vm0.maas is saying that DNS can't resolve the hostname, but it does say that juju gui is running there | 18:58 |
| kwmonroe | ntpttr: do a "juju status juju-gui | grep address" and try to go to the IP directly | 18:59 |
| kwmonroe | you might need a ",.mass" to your no_proxy var | 19:01 |
| kwmonroe | sorry, ".maas", not ".mass" | 19:01 |
| kwmonroe | ntpttr: the orange box knows how to resolve the .maas domain, so we probably just need to make sure requests for those URLs doesn't go through your proxy. | 19:02 |
| ntpttr | kwmonroe: Okay, I added .maas to no_proxy but juju status now is giving me "error dialing wss://node0vm0.maas:17070: no route to host" | 19:03 |
| ntpttr | kwmonroe: oh, it looks like after rebooting my machine again node0vm0 is saying no bootable device when it's trying to start up | 19:05 |
| kwmonroe | eeegads ntpttr, we can't catch a break :/ | 19:12 |
| kwmonroe | unfortuantely ntpttr, i'm not sure how you'd get a "no bootable device" post bootstrap.. we can either try and debug that, or "juju destroy-environment maas" and try to re-bootstrap on a non-VM node (like nodeX.maas instead of node0vm0.maas) | 19:14 |
| ntpttr | kwmonroe: hmm yeah this is a little frustrating haha. I re bootstrapped to the vm again (at home I was able to access the gui at node0vm0.maas), and it is saying that the public-address should be node0vm0.maas | 19:16 |
| kwmonroe | ntpttr: i don't follow your last comment there -- what is saying the public-address should be node0vm0.maas? | 19:19 |
| ntpttr | kwmonroe: That's the result when I run juju status juju-gui | grep address: 'public-address: node0vm0.maas', but it's not resolved by dns | 19:21 |
| kwmonroe | ntpttr: can you 'juju ssh 0'? | 19:22 |
| ntpttr | yep, that brings me into node0vm0 | 19:22 |
| kwmonroe | that might get you into the juju unit.. then 'sudo ifconfig' to find the ip | 19:22 |
| ntpttr | kwmonroe: well the eth0 interface doesn't have an IP, and juju-bro is at 10.14.100.1 | 19:23 |
| kwmonroe | can you get to http://10.14.100.1? | 19:23 |
| ntpttr | kwmonroe: no it times out | 19:24 |
| kwmonroe | ntpttr: try 'export no_proxy=$no_proxy,10.14.100.1' and then hit the url again | 19:26 |
| ntpttr | kwmonroe: I'm afraid I already had that in my no_proxy list | 19:27 |
| kwmonroe | rats | 19:27 |
| ntpttr | kwmonroe: It's odd, without the proxy stuff at home going to node0vm0.maas worked | 19:27 |
| ntpttr | in juju status, under machines "0", it lists dns-name: node0vm0.maas, is that right? | 19:29 |
| kwmonroe | yeah ntpttr, that looks right. the OB should know how to get to .maas hosts. | 19:33 |
| jose | marcoceppi: ping | 19:38 |
| ntpttr | kwmonroe: So I'm not sure, but I think this could be the issue. In order to get the vm to resolve my proxy host and download packages, I configure the Maas upstream DNS to be a server in the same network as the proxy host is, and disable DNSSEC. That way the VM can boot up. But then my controller node in the orange box doesn't resolve node0vm0 because it has a different dns nameserver, 10.14.4.1, which is the gateway to the Orange Box's wireless. Is that poss | 19:39 |
| kwmonroe | ntpttr: i don't think that would cause a problem.. even if node0vm0 is using a different dns server, 10.14.4.1 should know how to resolve node0vm0.maas. | 19:51 |
| kwmonroe | and 10.14.4.1 is the right dns server for the main orange box node -- it's the only one that knows what a .maas host is supposed to resolve to. | 19:52 |
| ntpttr | kwmonroe: Hmm well for whatever reason I'm able to curl node0vm0.maas from inside the VM but not from the controller node | 19:52 |
| kwmonroe | ntpttr: can you ping 10.14.100.1? | 19:52 |
| kwmonroe | (from the controller node) | 19:52 |
| ntpttr | kwmonroe: Yes I can | 19:52 |
| kwmonroe | hmph | 19:53 |
| kwmonroe | but you can't ping node0vm0, right? | 19:53 |
| ntpttr | I can ping node0vm0.maas from the controller, yeah | 19:53 |
| ntpttr | just can't curl or visit it with a browser | 19:53 |
| kwmonroe | ntpttr: let's see what ports are listening on your bootstrap node: juju run --machine=0 'sudo netstat -ntlp' | 19:55 |
| kwmonroe | ntpttr: are 80 and/or 443 listed? | 19:55 |
| ntpttr | Yes, both are listed | 19:56 |
| ntpttr | Should I add 'search maas' to /etc/resolv.conf? | 19:58 |
| ntpttr | kwmonroe: right now it just lists my company's domain next to search | 19:58 |
| kwmonroe | not sure about that ntpttr | 19:59 |
| kwmonroe | can you 'curl --noproxy * http://10.14.100.1'? | 19:59 |
| ntpttr | kwmonroe: no It says there's an error contacting the dns server | 20:01 |
| ntpttr | kwmonroe: to me it looks like the big difference between the controller where it doesn't work and the VM were it does is the 'search maas' line in /etc/resolv.conf, but I'm not sure the correct way to put it there | 20:04 |
| jose | tvansteenburgh: ping | 20:04 |
| tvansteenburgh | jose: hey | 20:04 |
| jose | tvansteenburgh: o/, I'm seeing a stuck test over here, mind taking a look? | 20:05 |
| jose | http://juju-ci.vapour.ws/view/Juju%20Ecosystem/job/charm-bundle-test-lxc/482/ | 20:05 |
| tvansteenburgh | jose: i can kill it if you want | 20:06 |
| jose | welp, just saying, it's completely stuck | 20:06 |
| kwmonroe | ntpttr: you can try adding maas to the resolv.conf search line (add it before your company domain with a space separating the two) | 20:07 |
| tvansteenburgh | jose: killed | 20:09 |
| kwmonroe | ntpttr: but i think 'search' only comes into play when resolving short names (so 'ping node0vm0' would find 'node0vm0.maas'). i don't think that'll help us since we can't even curl the ip address, which should totally not care about dns. | 20:09 |
| jose | tvansteenburgh: cool, thanks! | 20:09 |
| ntpttr | kwmonroe: You're right, it didn't help | 20:09 |
| kwmonroe | ntpttr: you can probably 'ping node0vm0' now from the controller, right? | 20:10 |
| ntpttr | kwmonroe: yep | 20:11 |
| kwmonroe | ntpttr: this is indeed frustrating.. let's make sure the juju-gui service is exposed (even though you shouldn't need to do this on an OB): juju expose juju-gui | 20:13 |
| kwmonroe | and then try to curl with the shortname since we know you can ping it: curl --noproxy * http://node0vm0 | 20:13 |
| kwmonroe | and if that doesn't work, try 'curl http://node0vm0' without the --noproxy option | 20:14 |
| ntpttr | Yes it is exposed, here is my full juju status output http://pastebin.com/XK5NW4V9 | 20:14 |
| ntpttr | kwmonroe: still no luck with curl, it was exposed before as well as the last step of the orange box bootstrap | 20:15 |
| marcoceppi | rick_h_: hey | 20:17 |
| ntpttr | kwmonroe: Could it be that DNSSEC is disabled? | 20:17 |
| rick_h_ | marcoceppi: party? | 20:18 |
| kwmonroe | i dunno ntpttr, but i feel like DNSSEC being disable should make our lives easier, not harder | 20:19 |
| ntpttr | kwmonroe: Yeah, without it disabled the machine couldnt even get through the bootstrap | 20:19 |
| ntpttr | kwmonroe: I just thought it might be related to the problem because it wasn't disabled when I did the same steps at home and it worked | 20:21 |
| kwmonroe | ntpttr: it's worth a shot, but i'm not crossing my fingers for that one ;) | 20:21 |
| kwmonroe | ntpttr: let's also try resetting the maas proxy: sudo orange-box-resetproxy | 20:21 |
| kwmonroe | and then 'curl --noproxy * http://node0vm0 || curl http://node0vm0' | 20:22 |
| marcoceppi | rick_h_: IM ALWAYS PARTYING! | 20:22 |
| marcoceppi | rick_h_: I want to package python-jujubundlelib into the juju stable ppa, is that cool with you? (I' | 20:22 |
| marcoceppi | (I'll do it in a way that won't conflict with future pacakges of the same version in archive if that every happens) | 20:22 |
| rick_h_ | marcoceppi: sure thing | 20:23 |
| rick_h_ | marcoceppi: Makyo and company can help test or anything you want there. | 20:23 |
| marcoceppi | rick_h_: yes, thank you - we're using it for bundle proofing in charmtools | 20:23 |
| ntpttr | kwmonroe: weird it will autocomplete orange-box-resetproxy but it says command not found | 20:23 |
| marcoceppi | rick_h_: I'll make sure to reach out if I have any issues, but I don't forsee a problem | 20:23 |
| ntpttr | kwmonroe: I had to become root with sudo -s and then run it for it to work but it did now | 20:24 |
| rick_h_ | marcoceppi: sounds good | 20:24 |
| ntpttr | kwmonroe: unfortunately didn't fix the DNS problem though | 20:24 |
| ntpttr | kwmonroe: Maybe the files in /etc/bind/maas? When I had it working before I was using maas 1.7.6 and now I'm on maas 1.8.0, which changed some of the files in that firectory | 20:28 |
| kwmonroe | ntpttr: those files define how to resolve the .maas domain, which sounds like it's working -- you being able to ping node0vm0.maas and have it return 10.14.100.1 means the local dns server did it's job. | 20:34 |
| ntpttr | kwmonroe: Then I'm sort of stumped here, I'm not sure which DNS it is that can't resolve node0vm0.maas | 20:35 |
| kwmonroe | ntpttr: what happens when you just try to 'ssh ubuntu@node0vm0'? | 20:36 |
| ntpttr | kwmonroe: It works | 20:37 |
| kwmonroe | ok, so DNS is working.. it's just the http/https traffic that isn't coming over. | 20:37 |
| kwmonroe | which has to mean one of our proxies isn't working | 20:37 |
| kwmonroe | (i think) | 20:37 |
| kwmonroe | or rather, the proxy is working too well.. it's like the request is being sent off to your myproxy:port even though we've told it no_proxy on 10.14.100.1 | 20:38 |
| ntpttr | kwmonroe: Hmm interesting, the proxies were resolved and were working during bootstrapping - update was able to resolve the ubuntu archives | 20:38 |
| ntpttr | kwmonroe: Yeah the no_proxy might be it somehow | 20:39 |
| ntpttr | kwmonroe: And there's still the thing where if I reboot the machine the VM cant boot back up because there's "no bootable device" >.< | 20:40 |
| kwmonroe | right ntpttr, the proxies did their job to get external bits onto the node, but now it seems like the request to http://node0vm0 is being sent to the proxy (which of course can't resolve it because the internet doesn't know about node0vm0) | 20:40 |
| kwmonroe | try this ntpttr, 'unset http_proxy https_proxy; curl http://node0vm0' | 20:42 |
| ntpttr | kwmonroe: Okay one second while I bootstrap again because I restarted my computer | 20:42 |
| kwmonroe | lol | 20:42 |
| kwmonroe | i was hoping the "no bootable device" thing just kinda fixed itself ;) | 20:43 |
| kwmonroe | i'm really not sure about that one | 20:43 |
| kwmonroe | other than trying to bootstrap to nodeX.maas vs node0vm0.maas to see if it's a KVM thing or a bootstrap thing. | 20:43 |
| kwmonroe | so the current plan ntpttr is to bootsrap with the proxies in place, then unset them and try to curl. that'll tell us if the proxy is gobbling up the http request. | 20:44 |
| ntpttr | kwmonroe: sounds good | 20:45 |
| ntpttr | kwmonroe: So a possible lead on the 'no bootable device', it seems like if the VM is powered on the no bootable device thing happens, when I tried to bootstrap again while it was on just now I got the error, but then I released the node with Mass which shut it down, and now bootstrapping is working again | 20:50 |
| kwmonroe | hey kirkland, have you seen an error on the OB where node0vm0 errors with 'no bootable device'? ^^ | 20:52 |
| ntpttr | kwmonroe: In the maas images tab it shows the image that I previously had added to it when setting up the orange box without a proxy, but it also says "Error: No boot sources provide Ubuntu images" above that | 20:55 |
| kwmonroe | ntpttr: you're out of my maas knowledge with that one ^^ i'd recommend asking that in #maas | 21:00 |
| ntpttr | kwmonroe: Well unsetting the proxy did in fact make curl work | 21:00 |
| kwmonroe | awesome.. cya later! | 21:01 |
| kwmonroe | (kidding, of course) | 21:01 |
| ntpttr | kwmonroe: haha the juju gui still seems to be hanging connecting to the juju environment though >.< | 21:01 |
| ntpttr | kwmonroe: wait there it goes it just took a good two minutes | 21:02 |
| kwmonroe | that's a pretty long wait, but i'll take it if it finally worked :) | 21:03 |
| ntpttr | kwmonroe: whew me too, hey it's a start | 21:03 |
| ntpttr | kwmonroe: thanks again | 21:03 |
| kwmonroe | so ntpttr, i think this points to no_proxy as the key here. | 21:03 |
| kwmonroe | we've gotta make sure http and https traffic to 10.14.100.x doesn't get sent off to the proxy | 21:04 |
| kwmonroe | ntpttr: i would have thought simply adding 10.14.100.1 to no_proxy would have been enough, so i'm stumped as to why unsetting both http_proxy and https_proxy was required. | 21:05 |
| ntpttr | kwmonroe: Yeah it is odd... | 21:06 |
| ntpttr | kwmonroe: well I have to get going for now, thank you for spending so much time with me on this I appreciate it | 21:09 |
| kwmonroe | sure thing ntpttr - just ping here (or on juju@lists.ubuntu.com) if you want to poke on this proxy stuff some more | 21:09 |
| ntpttr | kwmonroe: Sounds good, I will :) | 21:12 |
| marcoceppi | rick_h_ Makyo python-jujubundlelib and python3-jujubundlelib are now in the stable ppa, I'll throw the packaging branch up somewhere in a few mins. Happy to roll releases as they come out, we just added it to charm-tools for bundle v4 support/proofing | 21:13 |
| marcoceppi | oy, nvm | 21:14 |
| marcoceppi | I should talk with frankban tomorrow, his package may need some updates | 21:15 |
| rick_h_ | marcoceppi: which package? | 21:21 |
| rick_h_ | marcoceppi: these are new right? Or you mean the python package? | 21:21 |
| marcoceppi | rick_h_: I just now noticed there's a jujubundlelib package in stable that frankban uploaded to the stable ppa | 21:21 |
| marcoceppi | but it only provides jujubundlelib and not python-jujubundlelib python3-jujubundlelib as I expected it to | 21:21 |
| rick_h_ | marcoceppi: oh, ok | 21:22 |
| marcoceppi | I have a new packaging branch that would fix that, but it conflicted when I went to upload it, so I'll have to chat wtih frankban tomorrow on it (unless he's on vacation?) | 21:23 |
| rick_h_ | marcoceppi: well he's here in the room so he'll join irc | 21:23 |
| rick_h_ | marcoceppi: and you guys can hash it out | 21:23 |
| marcoceppi | oh, you guys sprinting? | 21:23 |
| rick_h_ | marcoceppi: I think the package is compatible with both | 21:23 |
| rick_h_ | marcoceppi: yea, in chicago this week | 21:23 |
| marcoceppi | rick_h_: it's not, it doesn't install the python3 path properly | 21:23 |
| rick_h_ | marcoceppi: oh, I'm surprised. | 21:23 |
| marcoceppi | not a big deal since charm-tools is only py2, but amulet is py2 + py3 and we'll see an update to that soon | 21:23 |
| rick_h_ | marcoceppi: ok, well asked frankban to join in and catch up | 21:24 |
| marcoceppi | rick_h_: not frankban's fault, it's a problem with py2dsc which doesn't support py3 ootb | 21:24 |
| rick_h_ | marcoceppi: ah, ok we're all caught up now | 21:26 |
| rick_h_ | marcoceppi: so go ahead and we'll update quickstart to use the python2 package you've got | 21:26 |
| rick_h_ | marcoceppi: and deprecate/clean up the older bundlelib package that's not python ver. specific | 21:26 |
| rick_h_ | (well, not named specific) | 21:26 |
| marcoceppi | rick_h_: okay, I'm happy to create a jujubundlib package target taht just aliases python-jujubundlelib so that it transitions cleanly | 21:27 |
| rick_h_ | marcoceppi: ok, will let you all coordinate it so that we make sure quickstart and the gui are kept happy | 21:27 |
| === cory_fu is now known as cory_out | ||
| firl | anyone able to help me understand how to use juju-deployer config constraint for a maas tag? | 22:35 |
| firl | trying constraints: "tag=25net" | 22:35 |
| wolsen | firl, try tags instead of tag | 22:50 |
| firl | wolsen no dice “could not find expected ‘:’” | 22:52 |
| wolsen | firl, are you using 'constraints: tags=25net'? | 22:53 |
| wolsen | or? | 22:53 |
| firl | http://pastebin.com/205weCSn | 22:54 |
| firl | constraints: "tag=25net" | 22:54 |
| wolsen | firl, I think you should change the constraints from "tag=25net" to "tags=25net" | 22:57 |
| firl | I tried that | 22:57 |
| firl | http://pastebin.com/pPiNPzyR | 22:58 |
| firl | ( same error ) | 22:58 |
| wolsen | firl, oh - you need a space between num_units: and 7 | 23:01 |
| wolsen | on the next line | 23:01 |
| firl | le sigh | 23:01 |
| firl | ty | 23:01 |
| wolsen | np | 23:01 |
| firl | it’s running now, maybe I should step away from the computer haha | 23:01 |
| pmatulis | what is the difference between 'juju add-machine' and 'juju deploy ubuntu' ? | 23:44 |
Generated by irclog2html.py 2.7 by Marius Gedminas - find it at mg.pov.lt!