| === ulidtko|kk is now known as ulidtko | ||
| icey | hey, I'm seeing a new error: ERROR cannot add relation "prometheus:target prometheus-snmp-exporter:snmp-exporter": establishing a new relation for prometheus:target would exceed its maximum relation limit of 1 (quota limit exceeded) | 11:50 |
|---|---|---|
| icey | any suggestions for how to resolve it? | 11:50 |
| achilleasa | icey: passing --force will allow the relation to be established. Can you point me to the charm versions that you are using? | 11:53 |
| icey | achilleasa: prometheus (prometheus2): 18 | 11:54 |
| icey | prometheus-snmp-exporter: 1 | 11:54 |
| icey | I've removed the snmp-exporter and re-added it | 11:54 |
| icey | (testing something out) | 11:54 |
| achilleasa | FYI: juju 2.8 enforces relation limits as specified in charm metadata; there has been a related bug (https://bugs.launchpad.net/juju/+bug/1887095) but we have not been able to reproduce | 11:55 |
| mup | Bug #1887095: Default relation limit of 1 prevents adding relations <juju:Incomplete by achilleasa> <https://launchpad.net/bugs/1887095> | 11:55 |
| icey | $ juju add-relation --force prometheus:target prometheus-snmp-exporter:snmp-exporter | 11:55 |
| icey | ERROR option provided but not defined: --force | 11:55 |
| icey | weird that, apparently, removing an application doesn't reduce that count back to zero :) | 11:55 |
| icey | achilleasa: also, doing a `juju deploy $bundle --force` shows the same erorr about not being able to add the relation | 11:56 |
| achilleasa | icey: interesting. can you please add a comment to the above bug? the relation count should certainly not be 1 if you remove the application | 11:58 |
| achilleasa | icey: just curious, did you happen to deploy prometheus to a 2.7.x model and then upgraded to 2.8.x? | 12:00 |
| icey | I did | 12:00 |
| icey | (this lab of mine is a lovely source or finding weird bugs :-P ) | 12:01 |
| achilleasa | icey: ok... I see what happened here... the charm parser in 2.7 juju used to set a default relation limit of 1 (even if not present in the metadata). That got persisted to the controller and triggers the error after the upgrade | 12:02 |
| icey | haha wow | 12:02 |
| icey | even if the application is removed... | 12:02 |
| icey | I can remove the application and try again, if you think it's worth it | 12:03 |
| achilleasa | please do | 12:03 |
| icey | (alternately, db surgery to update it?) | 12:03 |
| achilleasa | this explains why we couldn't reproduce the problem | 12:03 |
| icey | https://pastebin.ubuntu.com/p/TDPzsjghJr/ | 12:05 |
| achilleasa | manadart_: this will probably cause issues when attempting to add relations after a 2.7->2.8 upgrade. any thoughts on how we can fix it? We could play it safe and add an upgrade step to reset the limit in the stored charm metadata | 12:06 |
| icey | achilleasa: let me redact those ipv6 addresses and I'll post that on the bug as well | 12:07 |
| achilleasa | icey: thanks | 12:07 |
| icey | achilleasa: in the mean time, any suggestions for db surgery that I can do to get my snmp-exporter working again? | 12:08 |
| achilleasa | icey: sure; give me a min to write a fix query | 12:09 |
| icey | achilleasa: and, I suspect that the issue I'm hitting isn't related to the app reggint removed, it's that there are other consumers of the prometheus-exporter relation (telegraf) | 12:10 |
| icey | achilleasa: thanks :) | 12:10 |
| achilleasa | icey: in the meantime, is that the latest version of the charm? If not, doing an upgrade-charm --force should reset the limit | 12:14 |
| icey | achilleasa: it seems to be - what if I do an upgrade-charm --force on it anyways? | 12:15 |
| icey | achilleasa: my bundle just has: charm: cs:prometheus2 | 12:15 |
| icey | well that's cool: $ juju upgrade-charm --force prometheus | 12:15 |
| icey | ERROR already running latest charm "cs:prometheus2-18" | 12:15 |
| icey | interestingly, my first attempt to redeploy this bundle on the model should have upgraded the _other_ consumer of the exporter relation :-P | 12:16 |
| manadart_ | achilleasa: Need to discuss it with you. Later this afternoon? | 12:23 |
| achilleasa | manadart_: sure | 12:27 |
| achilleasa | icey: can you try this? https://pastebin.canonical.com/p/fb9W78NYRV/ | 12:27 |
| achilleasa | I am assuming that this is a disposable model, right? | 12:28 |
| icey | achilleasa: heh well | 12:28 |
| icey | it's my controller model :) | 12:28 |
| icey | as well as where I've stuck my monitoring | 12:29 |
| icey | but it is in my lab | 12:29 |
| icey | so, while I'd rather keep it around, it is rebuildable :-D | 12:29 |
| achilleasa | it just resets the limit in the charm metadata so it should be a pretty harmless change | 12:29 |
| icey | yeah, I did read the query before looking at hoe to actually connect to juju's mongo :) | 12:30 |
| icey | bah, not on the master | 12:30 |
| * icey retries | 12:30 | |
| icey | well that's odd | 12:31 |
| icey | https://pastebin.ubuntu.com/p/vKB8yQMYZd/ | 12:31 |
| icey | same thing worked on the first unit I tried to connect | 12:32 |
| icey | weird, has a different passsword saved, other one works | 12:32 |
| achilleasa | errr not sure about that one | 12:32 |
| icey | achilleasa: tried https://paste.ubuntu.com/21232100/ | 12:33 |
| icey | the three units (who doesn't run juju in HA) had different passwords stored, apparently | 12:33 |
| icey | anyways, query seems to have worked, and I can still talk to juju | 12:33 |
| achilleasa | does add-relation work now? | 12:33 |
| icey | achilleasa: I did successfully redeploy that bundle :) | 12:35 |
| icey | so, yes! | 12:35 |
| achilleasa | I will add this as a quick fix solution to the bug and chat with manadart_ about the best way to fix this | 12:38 |
Generated by irclog2html.py 2.7 by Marius Gedminas - find it at mg.pov.lt!