[08:43] <Chipaca> mo'in
[08:43] <bthomas> Morning
[08:44] <bthomas> I am on a do or die mission to refactor the prometheus charm (except tests) this week. I hope I will have a decent burial and epitaph if I fail.
[09:24]  * bthomas needs more coffeeeeeeeee
[09:36] <Chipaca> bthomas: what do you want on the wreath?
[09:36] <Chipaca> bthomas: I'd go with YOLO
[09:43] <bthomas> Chipaca: I always fancied "Shit Happens" from "Forrest Gump" (love that movie).
[10:10]  * Chipaca bbiab
[11:57] <gnuoy> Chipaca, sorry for the late notice but I don't have much to add from our last meeting. I'd suggest we skip it unless you have something you'd like to discuss ?
[11:58] <Chipaca> gnuoy: skip it is :)
[11:59] <gnuoy> ack, thanks
[11:59] <Chipaca> gnuoy: you miss out on watching me do unthinkable things to a rather boring wrap
[13:02] <gnuoy> I'm sorry to have missed that :)
[13:14] <Chipaca> you're really not :)
[14:39] <narindergupta> Chipaca, I am facing weird issue where I set the PodSpec and run the k8s status in loop since workload status is unknown is fine. But it stops executing the pods nd pod never gets created?
[14:40] <Chipaca> narindergupta: juju's pod spec setting is transactional
[14:40] <Chipaca> narindergupta: i.e. it's applied after your hook finishes successfully
[14:40] <Chipaca> narindergupta: if you set it, and then wait for it to change in a loop in the same hook, you're going to have a bad time
[14:40] <narindergupta> Chipaca, oh ok do we have update_status event?
[14:41] <Chipaca> narindergupta: if you mean an event in juju about k8s status, not currently
[14:41] <narindergupta> Thinking of changing in update_status in that case?
[14:42] <narindergupta> Oh ok so what's the solution as if now as my juju unit status is unknown while my pod status is active
[14:43] <Chipaca> narindergupta: there is update-status, which runs every 5 minutes on a clock
[14:44] <narindergupta> I think I can use that atleast so that status gets updated after 5 minutes.
[14:44] <narindergupta> And once we have solution then I will switch in charm
[14:45] <Chipaca> narindergupta: you can make that shorter, see https://discourse.juju.is/t/whats-the-update-status-interval/2571/5?u=chipaca
[14:47] <narindergupta> Chipaca, Hun it is a model level change.
[14:48] <Chipaca> yes
[15:04] <Chipaca> jldev: you around?
[15:06] <jldev> Chipaca: yes
[15:06] <Chipaca> jldev: i'm wanting to leave a little early to get to physio on time, meaning i wouldn't be there at the start of the standup
[15:07] <Chipaca> jldev: were you planning on going to the standup?
[15:07] <jldev> Chipaca: yes, I'll be there, no worries if you need to leave early
[15:07] <Chipaca> jldev: ok :)
[15:07] <Chipaca> jldev: (i would've stayed if you couldn't make it, and just made it super-fast)
[16:58] <narindergupta> Chipaca, looks like pod status is fetching me unknown everytime from k8s which is weired. While kubectl pod status returns active
[17:22] <Chipaca> narindergupta: what's "pod status" in the above?
[17:22] <Chipaca> as opposed to "kubectl pod status" i mean
[17:22] <narindergupta> Chipaca, I am testing k8s api using the operator library
[17:23] <narindergupta> Chipaca,     k8s_pod_status = k8s.get_pod_status(juju_model=juju_model,
[17:23] <narindergupta>                                         juju_app=juju_app,
[17:23] <narindergupta>                                         juju_unit=juju_unit)
[17:24] <narindergupta> Do I need to encode the parameters?
[17:27] <Chipaca> narindergupta: and what does the kubectl pod status output? (with -o yaml please)
[17:28] <Chipaca> narindergupta: or, better, what is in the debug log?
[17:28] <Chipaca> as that'll include the output i'm looking for
[17:29] <Chipaca> narindergupta: i'm looking for the log line that starts 'Received k8s pod status: '
[17:30] <narindergupta> Chipaca, unit.charm-k8s-cassandra/0.juju-log Received k8s pod status: <k8s.PodStatus object at 0x7f54cbb83df0>
[17:30] <Chipaca> hah, lolfail
[17:30] <Chipaca> i'll file a bug about that
[17:30] <narindergupta> :)
[17:30] <Chipaca> narindergupta: sorry, i'm going to need the output of the kubectl command
[17:30] <narindergupta> Chipaca, yes just a second
[17:31] <narindergupta> microk8s.kubectl get pods -n cassandra
[17:31] <narindergupta> NAME                             READY   STATUS    RESTARTS   AGE
[17:31] <narindergupta> charm-k8s-cassandra-0            1/1     Running   0          25m
[17:31] <narindergupta> charm-k8s-cassandra-operator-0   1/1     Running   0          25m
[17:31] <narindergupta> modeloperator-7d75bd7454-tfzgn   1/1     Running   0          25m
[17:31] <narindergupta> It says running
[17:34] <Chipaca> narindergupta: can you do: microk8s.kubectl get pods -n cassandra -o yaml | pastebinit -f yaml
[17:34] <Chipaca> narindergupta: (you might need to apt install pastebinit first)
[17:34] <narindergupta> Chipaca, somehow pastebinit is not working on this node
[17:34] <Chipaca> ah
[17:34] <narindergupta> I tried to upgrade this to 20.4 and pastbinit messed up now
[17:35] <Chipaca> ok, so copy-paste into a pastebin, please
[17:35] <narindergupta> Yeah will do that give me a sec
[17:35] <Chipaca> narindergupta: and then show me more context of your code above that called get_pod_status, so i know the values of juju_model, _app, and _unit :)
[17:36] <narindergupta> Sure
[17:36] <narindergupta> Those value I am getting from model but let me give you that first
[17:36] <narindergupta>     juju_model = self.model.name
[17:36] <narindergupta>     juju_app = self.model.app.name
[17:36] <narindergupta>     juju_unit = self.model.unit
[17:50] <narindergupta> Chipaca, https://paste.ubuntu.com/p/XqTrYrtBZR/
[17:51] <narindergupta> Chipaca, I have pasted only the Cassandra-pod status rather than operators as well.
[17:52] <Chipaca> narindergupta: charm-k8s-cassandra is the juju app name?
[17:52] <narindergupta> currect
[17:52] <narindergupta> correct
[17:53] <narindergupta> And I can see the same in pod labels as well
[17:53] <narindergupta> Model is cassandra
[17:53] <Chipaca> narindergupta: and charm-k8s-cassandra/0 is the unit?
[17:53] <narindergupta> Yes correct
[17:53] <narindergupta> juju status
[17:53] <narindergupta> Model      Controller  Cloud/Region        Version  SLA          Timestamp
[17:53] <narindergupta> cassandra  k8s-cloud   microk8s/localhost  2.8.1    unsupported  17:53:30Z
[17:53] <narindergupta> App                  Version                         Status  Scale  Charm                Store  Rev  OS          Address        Notes
[17:53] <narindergupta> charm-k8s-cassandra  gcr.io/google-samples/cassa...  active      1  charm-k8s-cassandra  local    0  kubernetes  10.152.183.66
[17:53] <narindergupta> Unit                    Workload     Agent  Address      Ports                       Message
[17:53] <narindergupta> charm-k8s-cassandra/0*  maintenance  idle   10.1.16.172  7000/TCP,7001/TCP,9042/TCP  Waiting for pod to appear
[17:54] <Chipaca> narindergupta: from here it looks like it should work
[17:55] <narindergupta> Chipaca, let me send you my charm code as well in case that helps
[17:55] <Chipaca> narindergupta: can you send me an email, with the charm code and how to deploy it on microk8s?
[17:55] <narindergupta> Chipaca, ok
[17:55] <Chipaca> I'll review it as soon as I can (which is in no less than ~2 hours)
[17:55] <Chipaca> thanks!
[17:55] <narindergupta> Chipaca, https://paste.ubuntu.com/p/s7FYfVqKKZ/
[17:56] <narindergupta> My charm code and I will send you in email as well
[17:57] <narindergupta> Let me checkin into GitHub so ti would be easy for you to grab the whole charm
[18:13] <narindergupta> Chipaca, sent you an email and attached the charm code
[19:55] <Chipaca> narindergupta: reproduced the issue, debugging it now
[19:55] <narindergupta> Ok cool
[20:33] <Chipaca> narindergupta: i think i'm seeing something different though
[20:34] <Chipaca> narindergupta: my kubectl shows the cassandra pod in CrashLoopBackOff
[20:35]  * Chipaca reinstalls just in case
[20:41] <Chipaca> narindergupta: yeah, it's always stuck in CrashLoopBackOff here
[20:42] <narindergupta> Chipaca, in meeting with manager
[20:42] <narindergupta> Chipaca, I can have a look after that but run juju debug-log and see
[20:46] <Chipaca> narindergupta: i think it might be that i was running an older juju
[20:46] <Chipaca> forgot i'd downgraded to test something else :)
[20:46] <Chipaca> will fix
[20:46] <narindergupta> Chipaca, :)
[20:46] <narindergupta> I am sing 2.8.1
[21:00] <Chipaca> narindergupta: and now i got it to status:active
[21:00] <Chipaca> ah, there we go (spoke to soon? was impatient?)
[21:05] <narindergupta> Chipaca, still having crash
[21:06] <Chipaca> no, now the kube side works
[21:06] <narindergupta> Chipaca, ok cool
[21:06] <Chipaca> so now i can look at the data and see what the k8s library is getting wrong :)
[21:06] <narindergupta> But k8 api library show unknown
[21:06] <narindergupta> Ok cool
[21:06] <narindergupta>  juju status
[21:06] <narindergupta> Model      Controller  Cloud/Region        Version  SLA          Timestamp
[21:06] <narindergupta> cassandra  k8s-cloud   microk8s/localhost  2.8.1    unsupported  21:05:31Z
[21:06] <narindergupta> App                  Version                         Status  Scale  Charm                Store  Rev  OS          Address         Notes
[21:06] <narindergupta> charm-k8s-cassandra  gcr.io/google-samples/cassa...  active      1  charm-k8s-cassandra  local    0  kubernetes  10.152.183.223
[21:06] <narindergupta> Unit                    Workload     Agent  Address      Ports                       Message
[21:06] <narindergupta> charm-k8s-cassandra/0*  maintenance  idle   10.1.16.177  7000/TCP,7001/TCP,9042/TCP  Waiting for pod to appear
[21:07] <narindergupta> Hopefully you have the same status
[21:20] <narindergupta> Chipaca, I am stepping out please send me email if I do not respond on this channel
[21:22] <Chipaca> narindergupta: found the problem
[21:24] <Chipaca> or at least _a_ problem :-) seeing if it fixed it now
[21:25] <Chipaca> …