=== ulidtko|kk is now known as ulidtko
[11:50] <icey> hey, I'm seeing a new error: ERROR cannot add relation "prometheus:target prometheus-snmp-exporter:snmp-exporter": establishing a new relation for prometheus:target would exceed its maximum relation limit of 1 (quota limit exceeded)
[11:50] <icey> any suggestions for how to resolve it?
[11:53] <achilleasa> icey: passing --force will allow the relation to be established. Can you point me to the charm versions that you are using?
[11:54] <icey> achilleasa: prometheus (prometheus2): 18
[11:54] <icey> prometheus-snmp-exporter: 1
[11:54] <icey> I've removed the snmp-exporter and re-added it
[11:54] <icey> (testing something out)
[11:55] <achilleasa> FYI: juju 2.8 enforces relation limits as specified in charm metadata; there has been a related bug (https://bugs.launchpad.net/juju/+bug/1887095) but we have not been able to reproduce
[11:55] <mup> Bug #1887095: Default relation limit of 1 prevents adding relations <juju:Incomplete by achilleasa> <https://launchpad.net/bugs/1887095>
[11:55] <icey> $ juju add-relation --force prometheus:target prometheus-snmp-exporter:snmp-exporter
[11:55] <icey> ERROR option provided but not defined: --force
[11:55] <icey> weird that, apparently, removing an application doesn't reduce that count back to zero :)
[11:56] <icey> achilleasa: also, doing a `juju deploy $bundle --force` shows the same erorr about not being able to add the relation
[11:58] <achilleasa> icey: interesting. can you please add a comment to the above bug? the relation count should certainly not be 1 if you remove the application
[12:00] <achilleasa> icey: just curious, did you happen to deploy prometheus to a 2.7.x model and then upgraded to 2.8.x?
[12:00] <icey> I did
[12:01] <icey> (this lab of mine is a lovely source or finding weird bugs :-P )
[12:02] <achilleasa> icey: ok... I see what happened here... the charm parser in 2.7 juju used to set a default relation limit of 1 (even if not present in the metadata). That got persisted to the controller and triggers the error after the upgrade
[12:02] <icey> haha wow
[12:02] <icey> even if the application is removed...
[12:03] <icey> I can remove the application and try again, if you think it's worth it
[12:03] <achilleasa> please do
[12:03] <icey> (alternately, db surgery to update it?)
[12:03] <achilleasa> this explains why we couldn't reproduce the problem
[12:05] <icey> https://pastebin.ubuntu.com/p/TDPzsjghJr/
[12:06] <achilleasa> manadart_: this will probably cause issues when attempting to add relations after a 2.7->2.8 upgrade. any thoughts on how we can fix it? We could play it safe and add an upgrade step to reset the limit in the stored charm metadata
[12:07] <icey> achilleasa: let me redact those ipv6 addresses and I'll post that on the bug as well
[12:07] <achilleasa> icey: thanks
[12:08] <icey> achilleasa: in the mean time, any suggestions for db surgery that I can do to get my snmp-exporter working again?
[12:09] <achilleasa> icey: sure; give me a min to write a fix query
[12:10] <icey> achilleasa: and, I suspect that the issue I'm hitting isn't related to the app reggint removed, it's that there are other consumers of the prometheus-exporter relation (telegraf)
[12:10] <icey> achilleasa: thanks :)
[12:14] <achilleasa> icey: in the meantime, is that the latest version of the charm? If not, doing an upgrade-charm --force should reset the limit
[12:15] <icey> achilleasa: it seems to be - what if I do an upgrade-charm --force on it anyways?
[12:15] <icey> achilleasa: my bundle just has: charm: cs:prometheus2
[12:15] <icey> well that's cool: $ juju upgrade-charm --force prometheus
[12:15] <icey> ERROR already running latest charm "cs:prometheus2-18"
[12:16] <icey> interestingly, my first attempt to redeploy this bundle on the model should have upgraded the _other_ consumer of the exporter relation :-P
[12:23] <manadart_> achilleasa: Need to discuss it with you. Later this afternoon?
[12:27] <achilleasa> manadart_: sure
[12:27] <achilleasa> icey: can you try this? https://pastebin.canonical.com/p/fb9W78NYRV/
[12:28] <achilleasa> I am assuming that this is a disposable model, right?
[12:28] <icey> achilleasa: heh well
[12:28] <icey> it's my controller model :)
[12:29] <icey> as well as where I've stuck my monitoring
[12:29] <icey> but it is in my lab
[12:29] <icey> so, while I'd rather keep it around, it is rebuildable :-D
[12:29] <achilleasa> it just resets the limit in the charm metadata so it should be a pretty harmless change
[12:30] <icey> yeah, I did read the query before looking at hoe to actually connect to juju's mongo :)
[12:30] <icey> bah, not on the master
[12:30]  * icey retries
[12:31] <icey> well that's odd
[12:31] <icey> https://pastebin.ubuntu.com/p/vKB8yQMYZd/
[12:32] <icey> same thing worked on the first unit I tried to connect
[12:32] <icey> weird, has a different passsword saved, other one works
[12:32] <achilleasa> errr not sure about that one
[12:33] <icey> achilleasa: tried https://paste.ubuntu.com/21232100/
[12:33] <icey> the three units (who doesn't run juju in HA) had different passwords stored, apparently
[12:33] <icey> anyways, query seems to have worked, and I can still talk to juju
[12:33] <achilleasa> does add-relation work now?
[12:35] <icey> achilleasa: I did successfully redeploy that bundle :)
[12:35] <icey> so, yes!
[12:38] <achilleasa> I will add this as a quick fix solution to the bug and chat with manadart_ about the best way to fix this