hloeungis there a way to switch back to using a local charm?05:24
hloeungI switched to use haproxy from the charmstore but can't seem to switch back05:24
hloeung$ juju upgrade-charm --switch local:xenial/haproxy haproxy05:25
hloeungERROR unknown schema for charm URL "local:xenial/haproxy"05:25
hloeung$ juju upgrade-charm --switch xenial/haproxy haproxy05:25
hloeungERROR already running latest charm "cs:haproxy-38"05:25
hloeunghmm, code says local: is meant to be supported05:54
hloeungok worked it out, had to specify full path, so /srv/mojo/...06:00
kjackalgood morning juju world!07:01
khyr0nnites over here :)07:14
=== markusfluer_ is now known as markusfluer
=== frankban|afk is now known as frankban
mattywAnkammarao, morning10:59
Ankammaraomattyw,stiil we are not able to create terms11:00
mattywAnkammarao, what's the error you get?11:00
Ankammaraoerror: cannot get discharge from "https://api.jujucharms.com/identity/v1/discharger": third party refused discharge: cannot discharge: user is not a member of required groups11:00
mattywAnkammarao, what's the name of the term you're uploading?11:01
Ankammaraomattyw,we have tried with different users which they are already members of that group11:01
mattywAnkammarao, and the group you're uploading to?11:02
Ankammaraomattyw,/snap/bin/charm push-term /root/ibm-platform-ac.txt ibmcharmers/ibm-platform-ac is the command using to push terms11:04
mattywAnkammarao, and you're a member of the ibmcharmers group?11:04
urulamaAnkammarao: can you see the groups returned with "charm whoami"?11:04
Ankammaraono i am just seeing the user-name logged in11:05
Ankammaraosry, i am getting like this "root@islrpbeixv685:~# charm whoami User: rajith-pv Group membership: ibmcharmers"11:05
urulamamattyw: ^ this means that the group is ok ... please check access on terms side11:06
Ankammaraomattyw,i am not a member of ibmcharmers group,but my team meber tried with his userid(already a member to that group)11:07
ashipikamattyw: ping?11:08
mattywAnkammarao, so rajith should be able to push the terms then from his machine?11:08
mattywAnkammarao, and to get access you will need to contact one of the admins for that team11:08
Ankammaraomattyw,no he unable to push terms11:08
mattywAnkammarao, what error does he get?11:09
mattywAnkammarao, the same one?11:09
Ankammaraomattyw,the same error11:09
mattywashipika, can you help Ankammarao work out what's going on? they're trying to push terms but keep getting "user is not a member of required group"11:10
ashipikamattyw, Ankammarao: just trying to verify that it works for me.. one minute, please11:10
mattywAnkammarao, how long as rajith been a member of that group?11:10
mattywashipika, is it possible they'd need to login again?11:11
ashipikamattyw: yes, perhaps11:11
Ankammaraomattyw,he is a member since from 6months or more11:12
Ankammaraomattyw,root@islrpbeixv685:~# /snap/bin/charm version charm 2.2.0 charm-tools 2.1.811:12
Ankammaraomattyw,is that fine with version11:13
mattywAnkammarao, looks fine to me - there's some kind of issue with login, ashipika is the expert there so he'll be able to help out11:13
ashipikaAnkammarao: interesting.. so which term are you trying to push? what is the term id?11:13
mattywashipika, ibmcharmers/ibm-platform-ac11:13
mattywashipika, and charm whoami shows ibmcharmers membership11:14
ashipikamattyw: that is unusual11:14
mattywAnkammarao, when you ran charm whoami was that running /snap/bin/charm whoami?11:14
mattywAnkammarao, would be interesting to see if they return the same information11:14
ashipikamattyw: for me "charm whoami" returns "not logged into https://api.ujucharms.com/charmstore" :)11:15
Ankammaraomattyw,showing diffrent user root@islrpbeixv685:~# /snap/bin/charm whoami User: achittet11:15
ashipikaAnkammarao:  could you please try removing your .go-cookies file?11:16
mattywashipika, be careful...11:16
ashipikamattyw: ?11:16
mattywashipika, there's some confusion between the go-cookies being used by the charm command and the one being used by the snap charm command11:16
mattywashipika, and only the snap charm command contains the commands for pushing terms11:16
mattywashipika, so we need to remove go-cookies for that command11:16
mattywashipika, which might not be ~/.go-cookies11:17
ashipikamattyw: ah, interesting.. i wonder why charm whoami does not work for me11:17
ashipikaAnkammarao: launchpad says you are not member of any group..11:18
ashipikamattyw: ^11:18
mattywashipika, they're using login for rajith-pv11:19
Ankammaraoashipika : yes i am not member to that group, but my team member rajith is a member11:19
ashipikaAnkammarao: could you please go to https://jujucharms.com and log in as rajith-pv? and then try again?11:20
Ankammaraoashipika: ok11:20
ashipikaAnkammarao: thank you11:20
Ankammaraomattyw,ashipika: its working now11:26
ashipikaAnkammarao: \o/11:26
ashipikaAnkammarao: it's because of the way our identity manager works.. it cannot get user information until you log in with jujucharms.com at least once..11:26
ashipikaAnkammarao: sorry for the inconvenience11:27
Ankammaraoashika: there is a conflict b/w charm login and /snap/bin/charm login11:28
Ankammaraoashipika : both are different users11:28
Ankammaraomattyw,ashika: root@islrpbeixv685:~# sudo /snap/bin/charm push-term /root/ibm-platform-ac.txt ibmcharmers/ibm-platform-ac ibmcharmers/ibm-platform-ac/111:29
Ankammaraoashika : getting output like "ibmcharmers/ibm-platform-ac/1"11:29
ashipikaAnkammarao: did you do snap login first?11:29
ashipikaAnkammarao: now you can use ibmcharmers/ibm-platform-ac/1 term in your charms..11:30
ashipikamattyw: any idea why the two users should be different?11:30
Ankammaraoashika : but we have mentioned term name like "ibm-platform-ac/1" in metada.yaml11:31
Ankammaraoashipika : no, both users should be same as ealrier i got error because of both users are diffrenet11:31
Spauldingfolks, one simple question... update-status... what is should return11:32
Spauldingit's like a health check?11:32
Spauldingso if i'll try to run curl to see that website is working, and return error code ... that should be fine?11:32
ashipikaAnkammarao: you need to use ibmcharmers/ibm-platform-ac/1 in metadaya.yaml.. because therm id consists of the namespace (ibmcharmers) and the term name (ibm-platform-ac).11:32
ashipikaAnkammarao: if you want your charm to require agreement to a specific term, you need to include the term revision number.. otherwise your charm will always require agreement to the latest revision of the term document11:33
Ankammaraoashika: ok,thank you11:36
ashipikaAnkammarao: you're welcome.. if you have any further issues pushing or releasing terms, please ping me11:37
Ankammaraoashipika : sure i'll ping you11:41
dbuligahey guys. What is the status of this PR? https://code.launchpad.net/~dbuliga/charms/trusty/nagios/nagios/+merge/288614 Nothing changed on it since 2016-10-07. Is it possible to get it reviewed? Thx. Denis15:25
geethaHi, I have installed VNC server and firefox on ubuntu 16.0 s390x machine. When I tried to start firefox through vnc client, it's giving me error: sementation fault (core dumped).15:47
geetha*Segmentation fault15:48
=== freyes__ is now known as freyes
=== Guest36_ is now known as uglyboxer
bdxhey whats up guys, I'm having issues connecting to my controller, see -> https://bugs.launchpad.net/juju/+bug/164463417:13
mupBug #1644634: cannot access controller <juju:New> <https://launchpad.net/bugs/1644634>17:13
bdxthis has basically rendered all of the models I have provisioned on that controller inaccessable17:15
bdxany ideas on what to do here?17:16
marcoceppibdx: so you're not able to restart the mongodb process?17:19
bdxmarcoceppi: I'm not able to get any correspondence from the controller17:20
bdxmarcoceppi: my initial inclination was that the controller is just mad iowaiting17:20
bdxbut my crud monitoring through aws console suggests its not under much load17:21
marcoceppitoo many file descriptors open?17:21
marcoceppibdx: you're not able to SSh or anything?17:21
bdxmy ssh just connected17:22
bdxafter like 5+ minutes17:22
bdxI was right17:22
bdxcontroller is slammed17:22
bdxI couldn't connect all weekend17:23
bdxit would just time out17:23
bdxI'm actually surprised it is still running17:23
bdxmarcoceppi: https://s22.postimg.org/gfo71ug4h/Screen_Shot_2016_11_28_at_9_26_25_AM.png17:23
marcoceppiokya, that's a lot of mongod processes17:24
bdxcloudwatch metrics are poop17:24
marcoceppibdx: can you clean shutdown those processes?17:24
bdxI guess ...17:24
marcoceppiI want to grab logs, it seems like systemd is just going crazy spawning monogd procs17:25
marcoceppirick_h: ^ ?17:25
bdxlike `sudo kill <proc#>`17:25
marcoceppi`killall -15 mongod`17:25
marcoceppimight be quicker17:25
bdxright lol17:25
marcoceppibdx: service juju-db stop might be good too17:27
marcoceppibdx: then getting logs and restarting17:27
bdxrestart the controller?17:27
bdxI must of hit a memory increase, my ssh session froze17:28
bdxeeehh, waiting for ssh again omp17:28
rick_hbdx: marcoceppi sorry was otp, looking17:29
bdxmarcoceppi: ok, back on the controller17:29
marcoceppibdx: restarting the services, though at this point, restarting the controller VM might not be a bad idea, though grabbing logs will be super helpful17:30
rick_hbdx: k, yea worst case reset the VM and it should come up if things are working.17:30
bdxgrabbing logs now17:30
rick_hbdx: ouch, so load of 30 but cpu/memory is ok...disk thrashing to no end?17:30
marcoceppibdx: are you running LXD workloads on here?17:31
bdxmarcoceppi: no, but my controller was initially throwing a bunch of errors due to not having lxd it seemed, so I installed lxd simply to negate the noise17:34
bdxrick_h, marcoceppi: my tar of the controller logs is ~100MB, how can I get this to you guys?17:35
marcoceppibdx: is that also gzipped?17:36
marcoceppidang. I don't know if LP will let you upload that much, but the bug you linked might be a good place17:36
rick_hbdx: if not, try slicing into 10 chunks perhaps? or dropbox link or whatever.17:37
rick_hbdx: get me a url and I'll get it and help get it to folks if needed17:37
bdxrick_h, marcoceppi: https://bugs.launchpad.net/juju/+bug/1644634/comments/117:40
mupBug #1644634: cannot access controller <juju:New> <https://launchpad.net/bugs/1644634>17:40
rick_hbdx: <3 ty17:41
rick_hbdx: are things somewhat sane post-restart?17:41
rick_hbdx: or still nuts?17:41
bdxrick_h, marcoceppi: no, the controller is iowaiting again, fully maxed17:41
marcoceppibdx: is iotop installed on the machine?17:41
bdxrick_h, marcoceppi: I have a redis cluster deployed supporting a staging and QA env, other than that I could ditch this controller, unfortunately I need to keep that QA env up at all costs bc its getting beat on pretty heavily right now17:42
rick_hbdx: can you restart the instance?17:43
rick_hbdx: or did you do that? /me glanced through the backlog but might have missed it17:43
bdxiotop -> https://s21.postimg.org/58l6v7787/Screen_Shot_2016_11_28_at_9_47_38_AM.png17:44
bdxrick_h: no, i didn't restart .... do you think that will send my redis cluster into a state of disarray?17:44
rick_hbdx: shouldn't effect it at all.17:45
rick_hbdx: I mean the controller is not the same machiens as the redis cluster right?17:45
rick_hbdx: restarting the controller should just have agents on the redis machines time out for a bit while it comes back up17:45
rick_hbdx: no damage to the running redis processes at all17:46
bdxrick_h: ok, restarting controller now17:46
rick_hbdx: poking at the logs but will take a sec17:47
rick_hlol 1.7G of logs17:47
bdxrick_h: I've previously had a whole openstack go tits up after losing the controller ... only reason I ask17:47
bdxrick_h: right ...17:47
rick_hbdx: really? I'd be curious about that story. We've had folks deploy OS with juju and then go in and shutdown all jujud on each machine/etc17:48
rick_hbdx: openstack still ran and they made it a manually run openstack at that point17:48
bdxrick_h: yeah, after I shutdown all of the juju agents everywhere I was able to move forward and save it all ... but if I remember correctly my db ended up borked17:49
rick_hbdx: the controller db? or the OS mysql db?17:50
bdxrick_h: openstack mysql db17:50
rick_hbdx: can you add info to the bug about the cloud/instances running the controller/etc?17:50
bdxyea, omp17:51
bdxahhhh shoot, wrong issue17:54
bdxrick_h: comment #317:55
rick_hbdx: k, ty17:55
rick_hbdx: can you note the instance/disk and such of the controller nodes. So there's a storm in the logs, but based on that top output it seems like it's not ram/cpu but disk and so wondering if we can draw some lessons on disk peroformance and the mount of logs/etc going on17:56
rick_hbdx: there's an openstack in here as well as the redis?17:57
rick_hhow many models with what running?17:57
bdxrick_h: no the openstack was yesteryear17:57
bdxrick_h: http://paste.ubuntu.com/23549554/17:58
rick_hbdx: hmm, seeing this in the log17:58
rick_h2016-11-28 17:32:16 DEBUG juju.apiserver request_notifier.go:140 -> [55] unit-openstack-dashboard-0 795.567548ms {"request-id":44,"error":"watcher has been stopped","error-code":"stopped","response":"'body redacted'"} NotifyWatcher["10"].Next17:58
bdxrick_h: yeah, I've been using the dashboard to help troubleshoot keystone v3 ops I'm trying to solidify17:59
rick_hbdx: oh ok, so not all of openstack but that is there ok17:59
bdxrick_h: I've deployed keystone + barbican as a standalone secrets provisioning17:59
rick_hbdx: gotcha, just trying to understand the logs as I go through sorry18:00
rick_hbdx: is the restart back up?18:02
bdxrick_h: yesa18:05
rick_hbdx: sane or slammed?18:05
bdxrick_h: looking much better now -> https://s22.postimg.org/gkfbj32ld/Screen_Shot_2016_11_28_at_10_09_21_AM.png18:06
rick_hbdx: ok, I've got to run to a meeting. I've got hte logs and alexisb is getting someone to look at the bug/details when they come online in a bit.18:06
bdxrick_h: thanks man18:06
rick_hbdx: thanks for the added details, we're currently working on a lot of this so this is good extra data.18:06
bdxrick_h: appreciate it18:06
bdxdo I get a guinea pig award?18:07
alexisbbdx you get the juju ecosystem rockstar award :)18:09
alexisbthanks for all the info in the bug18:09
alexisbwe will get updates in it tonight18:09
=== sg is now known as Guest34504
Guest34504Is it possible to use Juju resource with a bundle? https://jujucharms.com/docs/stable/developer-resources18:12
Guest34504If so, pls let me know the way to do it18:12
bdxalexisb: awesome, thx18:14
bdxafter adding and removing a machine multiple times from the manual provider, I am now not able to add the machine anymore, I'm getting this -> http://paste.ubuntu.com/23549809/19:04
bdxeven though the machine doesn't exist on my controller in any models19:04
bdxI've tried ssh'ing in and 'rm -rf /var/{log,lib}/juju`, and `sudo deluser --remove-all-files ubuntu`19:05
bdxbut I still get the same result19:05
bdxcould the machine be hanging around in the controller somewhere, even though its not shown via `juju status`?19:05
bdxrick_h, marcoceppi: ^19:06
rick_hbdx: looking, want to check what juju looks for to make that determination19:07
=== frankban is now known as frankban|afk
marcoceppirick_h: can you use resources in a bundle?19:08
rick_hmarcoceppi: not a local resource atm, it's on the 17.04 roadmap list19:09
marcoceppirick_h: ack, ta19:09
bdxrick_h: https://bugs.launchpad.net/juju/+bug/164544619:09
mupBug #1645446: juju incorrectly reports machine already provisioned <juju:New> <https://launchpad.net/bugs/1645446>19:09
bdxI'm going to hit every corner case today, fyi19:11
rick_hbdx: wheeee, replied with request for as much detail on the before as we can get please19:15
rick_hbdx: only work around I could think of atm is to manually remove the doc from the db, or change the ip address/hostname of the machine.19:15
rick_hbdx: but looks like some bug in the removal that you hit with the add/remove several times.19:16
bdxrick_h: ok, I don't have dynamic provisioning of spinning these up and down ... I could kill-controller and reprovision it all if that would be easier ?19:17
rick_hbdx: yea, unfortunately that's the non-hackiest path forward.19:17
rick_hbdx: since it's in the mongodb that the doc is hanging around that is triggering that error19:17
bdxrick_h: alright, updated, thanks19:22
rick_hty bdx19:22
bdxrick_h: ehh, even after killing my controller, and re-bootstrapping, I'm getting the same thing19:27
rick_hbdx: ?! /me goes back to the code he thought he understood19:28
rick_hbdx: when you kill-controller in manual...I wonder if it clears the db? I mean that would be nuts...but.19:29
bdxrick_h: yea, I'm wondering if need to manualy tear all that ishk outta there?19:30
rick_hbdx: sec, can you grab the logs please?19:30
rick_hbdx: looking for lines like: "Checking if %s is already provisioned"19:31
rick_hbdx: looks like there's another place that can hit this if the juju service exists on the system19:32
rick_hbdx: so that's in upstart or systemd depending on series19:32
rick_hbdx: so it's probably not the juju directory, but the services directory that's the issue.19:33
bdxrick_h: ok, yeah, its a trusty controller19:34
rick_hbdx: but the machine is xenial?19:34
bdxrick_h: no, the machine is trusty too19:34
bdxooh, the services dir on the machine ... my b19:34
rick_hbdx: right, the systemd script for jujud on that machine that's "already provisioned"19:35
bdxyea, I was looking around in there and didn't see anything that said juju .... oh, where is that?19:35
* rick_h is looking19:35
rick_h /etc/systemd/system ?19:35
rick_hanything juju in there?19:36
bdxrick_h: http://paste.ubuntu.com/23549930/19:36
rick_hbdx: yea, there we go: https://github.com/juju/juju/blob/6cf1bc9d917a4c56d0034fd6e0d6f394d6eddb6e/provider/manual/environ_test.go#L10619:36
rick_hbdx: any others from that code link hit?19:37
jrwrenwhen did trusty get systemd?19:37
rick_hbdx: /etc/init/juju ?19:37
bdxI checked there too .. omp19:37
rick_hjrwren: no, but the controller is trusty, but the machine is xenial (the add-machine one that is)19:37
jrwrenoh, ok.19:38
rick_hbdx: ok, what's omp? /me is ignorant19:38
bdxrick_h: no its trusty19:39
rick_hok, so the controller was killed/restart, the machine in question has no reference to the juju service...19:39
bdxrick_h: here we go -> http://paste.ubuntu.com/23549954/19:40
bdxrick_h: exactly19:40
bdxrick_h: should I kill the controller, then ssh in and rm everything juju and mongo from the controller ?19:40
rick_hbdx: not yet, /me is chasing the code from that log output19:41
rick_hok, that is definitely from https://github.com/juju/juju/blob/6cf1bc9d917a4c56d0034fd6e0d6f394d6eddb6e/environs/manual/init.go#L34 then19:41
rick_hat least the log lines match up19:42
rick_hso time to figure out wtf we run on systemd to get the list of services19:42
rick_hbdx: can you run this on the machine? -- /bin/systemctl list-unit-files --no-legend --no-page -t service | grep -o -P '^\w[\S]*(?=\.service)'19:44
bdxrick_h: bash: /bin/systemctl: No such file or directory19:47
rick_hbdx: this is on the xenial host?19:47
bdxrick_h: there is no xenial host19:47
rick_hbdx: on that pastebin it said xenial?19:47
rick_hbdx: ok sorry, I thought you had a trusty controller with a xenial machine you were trying to add19:48
rick_hbdx: I see now that your other paste said it was --trusty19:48
rick_hbdx: sorry, my confusion. Ok, looking in the wrong place then19:48
rick_hbdx: so new command for trusty: sudo initctl list | awk '{print $1}' | sort | uniq19:50
bdxrick_h: http://paste.ubuntu.com/23549991/19:50
bdxits there19:50
rick_hbdx: there you go, so Juju thinks the machine's provisioned because of the juju entries in there19:51
bdxawesome, I'm gonig to rm the jujud-machine-1719:51
rick_hbdx: rgr19:51
rick_hbdx: the unit one as well if that's done/gone19:52
rick_hbdx: if any line has "juju" in it it's considered provisioned so need both gone: provisioned := strings.Contains(output, "juju")19:52
bdxyeah, just the machine-17 didn't do the trick19:52
bdxthere we go, removing both did it19:53
rick_hbdx: <3 ok...updating the bug with these notes19:53
bdxthanks rick19:54
rick_hbdx: np, see you in an hour or so for round 3? :P19:54
bdxrick_h: I just got elasticsearch (patched with the addition of the python-yaml dep) to successfully deploy to that machine19:57
bdxrick_h: that was my last standing issue in the bunch, for the time being :-)19:58
rick_hbdx: ok, I'm going to take nap for now then :P19:58
lazyPowermbruzek  https://github.com/juju-solutions/jujubox/pull/20#pullrequestreview-10408603  - i had one minor comment. fi you dont want to take any action on that i'm +1 to this as it is.20:35
mbruzeklazyPower: I still think the juju-1 packaging is not final, and I don't want to create a sym-link at this time20:40
mbruzekIf you want to create a bug we can track it that way.20:40
lazyPowerack, not that serious20:40
lazyPoweri'm +1 as it is, merging now20:40
lazyPowermbruzek - both jujubox pr's landed. You made mention of a charmbox pr but i dont see one on the repo. Did it land already or is it MIA?20:45
lazyPowermbruzek - i assume it was this one? https://github.com/juju-solutions/charmbox/commit/3df12bf82ce6bd16d519bb812c710831e799e6fb20:46
mbruzekyes that is it20:46
lazyPowerawesome. landed that one over the weekend20:46
lazyPowerso i think we're good on the box images now, just need to kick the builders and get the flavors setup20:47
lazyPowerplus get this back in CI20:47
lazyPowerwant to batcave it and get that done real quick? it'll only take a few minutes20:47
lazyPowermbruzek ^20:49
mbruzekyes I am there?20:50
kwmonroelazyPower: i see a juju-1 and latest tag now in jujubox.  is the 'devel' branch now deprecated?22:52
lazyPowerkwmonroe - i beleive mbruzek sent a mail to the list, but yes, devel is now :latest22:53
lazyPowerand the devel tag is in the process of being deprecated22:53
lazyPowerjuju-1 will be the best-effort tag to keep a juju-1 compliant charmbox around22:53
kwmonroeword lazyPower -- i see mbruzek's note now (spelling errors and all).  thanks!22:55
lazyPowerkwmonroe np happy to help22:55
hallyn'juju bootstrap' always seems to hang (on both xenial and yakkety hosts), the console claims it's doing "apt-get upgrade", but the process list shows a23:25
hallyn100000   15822 15821  0 22:58 ?        00:00:00 sudo /bin/bash -c /bin/bash -c  set -e tmpfile=$(mktemp) trap "rm -f $tmpfile" EXIT cat > $tmpfile /bin/bash $tmpfile23:25
hallynwhich has a bash child and bash grandchild with no arguments23:25
hallyntych0: using juju 2.0 and lxd, how can i specify that it should use an image i create?  (i need to drop open-iscsi so juju bootstrap won't hang)23:45
tych0hallyn: sec,23:45
tych0hallyn: make an image called ubuntu-$series23:46
tych0or, one that has a tag called that23:46
tych0it'll use that one23:46
hallynoh, ok.  that'll suffice for now,23:46
hallynbut so there's no way to say "use image xenial-base" tha tyou knwo of?23:46
hallyntych0: (trying that - thanks)23:47
tych0hallyn: no, i don't think so23:47
tych0the ubuntu-$series alias is always used for exactly htis reason23:47
tych0i didn't think about making the name custom at the time :)23:47
hallynkewl - easy enough, re-bootstrapping, let's see23:48
hallyntych0: success, thx :)23:50
tych0hallyn: cool, glad it worked!23:51

Generated by irclog2html.py 2.7 by Marius Gedminas - find it at mg.pov.lt!