mupBug #1647897 opened: Juju bootstrap proxy support <juju-core:New> <https://launchpad.net/bugs/1647897>01:40
=== iatrou_ is now known as iatrou
=== Tribaal_ is now known as Tribaal
=== frankban|afk is now known as frankban
gsamfirahello folks. We have about 120 nodes constantly being deployed and redeployed by juju in a CI09:56
gsamfirathis environment has been online for a while now09:56
gsamfiraas a result, some collections have grown quite a bit09:56
gsamfiraone of them is: presence.beings09:56
gsamfirajuju:PRIMARY> db.presence.beings.count()09:56
gsamfirathe other was logs.logs (which had about 14.000.000 entries)09:57
perrito666gsamfira: hey09:57
gsamfirahey perrito66609:57
perrito666gsamfira: logs should be rotated, something is wrong there09:57
gsamfiraI don;t think juju got a chance to rotate09:57
gsamfirathe db was under huge load09:58
perrito666gsamfira: but the rotation is done by juju09:58
perrito666and by rotation I mean just deletion09:58
gsamfiramostly because of a bunch of queries against precence.being09:58
gsamfirawhich had no index on model-uuid09:58
gsamfiraand did a COLLSCAN for every query09:58
perrito666gsamfira: this is 2.x?09:59
gsamfiraI can imagine. But juju kind of stopped working, erroring out with i/o error while talking to mongo09:59
gsamfirai/o error and i/o timeout09:59
gsamfiraI had to drop all connections to the state machine port, go into the database and create an index in presence.beings10:00
gsamfiradb.presence.beings.createIndex({"model-uuid": 1})10:00
gsamfiraalso on txns10:00
gsamfiradb.txns.createIndex({"s": 1})10:00
gsamfirajust to get rid of a lot of spam in syslog about COLLSCANs10:00
perrito666thumper: babbageclunk any of you might be interested in this?10:01
gsamfiraalso dropped all logs and saved 1.3 GB of disk space10:01
perrito666gsamfira:  lol10:01
perrito666so, for how long has this been running?10:01
gsamfirathe state machine is a 16 CPU core, 32 GB of RAM VM hosted on RAID10 10kRPM SAS disks10:01
thumpergsamfira: logs is a capped collection so size bound10:01
thumperpresence is bollocks10:02
mupBug #1647897 changed: Juju bootstrap proxy support <juju-core:Invalid> <https://launchpad.net/bugs/1647897>10:02
thumperand grows forever10:02
gsamfiraalso might be worth mentioning we have enabled HA on this particular setup10:02
gsamfirathumper: ouch10:02
gsamfira2 low hanging fruit we can implement is simply creating indexes on stuff we query10:02
gsamfiraespecially if we query those frequently10:03
gsamfirathe environment has been up for a couple of months I think10:03
gsamfiralemme check10:03
gsamfirasince october. Rougly 2 months10:05
gsamfiraperrito666: ^10:05
perrito666gsamfira: tx that is a nice point of data10:06
gsamfirahttp://paste.ubuntu.com/23592735/ <-- this might also be of interest10:06
gsamfirathis is a CI environment, a lot of units get torn down and spun up again10:06
gsamfiraso there is a lot of traffic10:06
gsamfiraperrito666:  juju:PRIMARY> db.statuseshistory.count()10:08
gsamfiraso you get an idea10:08
perrito666gsamfira: that is quite a reasonable size for status :) because its also capped but that one seems to be working10:09
gsamfirayup. The environment seems stable again after creating the indexes and cleaning the logs10:10
perrito666gsamfira: that is good to know :) a quick workaround and low hanging fruit all together10:11
gsamfiraI'll create a patch for the indexes later this week10:12
mgzvoidspace: http://streams.canonical.com/juju/tools/agent/2.0.2/10:17
mupBug #1357760 changed: ensure-availability (aka HA) should work with manual provider <cloud-installer> <ha> <landscape> <manual-provider> <manual-story> <juju:Triaged> <juju-core:Won't Fix> <https://launchpad.net/bugs/1357760>10:20
mupBug #1493058 changed: ensure-availability fails on GCE <docteam> <enable-ha> <gce-provider> <ha> <jujuqa> <juju:Invalid by thumper> <juju-core:Won't Fix> <juju-core 1.24:Won't Fix> <juju-core 1.25:Won't Fix> <https://launchpad.net/bugs/1493058>10:20
mupBug #1512569 changed: UniterSuite.TestRebootNowKillsHook fails with: uniter still alive <ci> <test-failure> <unit-tests> <juju:Triaged> <juju-core:Won't Fix> <https://launchpad.net/bugs/1512569>10:20
mgzalexisb: can you backport your fix for bug 1631369 for 2.0 branch as well?10:20
mupBug #1631369: ExpireSuite.TestClaim_ExpiryInFuture_TimePasses took way too long <ci> <intermittent-failure> <regression> <unit-tests> <juju:In Progress by alexis-bruemmer> <https://launchpad.net/bugs/1631369>10:20
alexisbmgz, sure10:21
axw_perrito666: I've created https://github.com/juju/juju/tree/feature-persistent-storage10:44
perrito666axw_: ack10:44
perrito666axw_: we should propose against that I presume10:44
axw_perrito666: yup, so we don't mess up 2.110:45
perrito666we dont mess up things, we awesome-ize them10:45
mupBug #1557726 changed: Restore fails on some openstacks like prodstack <backup-restore> <jujuqa> <openstack-provider> <juju:Fix Released by hduran-8> <juju 2.0:Fix Released by reedobrien> <juju-core:Won't Fix> <https://launchpad.net/bugs/1557726>10:47
=== akhavr1 is now known as akhavr
natefinchthumper: https://bugs.launchpad.net/juju/+bug/164806312:46
mupBug #1648063: kill-controller removes machines from migrated model <model-migration> <juju:Triaged> <https://launchpad.net/bugs/1648063>12:47
=== freyes__ is now known as freyes
voidspacejam: from within a hook context (debug-hooks) is there a way to get a list of all the bindings defined for the charm?12:56
voidspacefrobware: ^^^ do you know?12:57
jamvoidspace: generally it will be the things listed under "provides" or "requires" in charm-metadata.yaml13:00
frobwarevoidspace: https://github.com/frobware/testcharms13:00
voidspacejam: referring to the charm is what I was hoping to avoid13:01
voidspacejam: but thanks13:01
rick_hvoidspace: :/ had hoped but nothing here of use https://jujucharms.com/docs/2.0/authors-hook-environment13:15
voidspacerick_h: it would be a nice tool to have13:16
voidspacerick_h: however...13:16
voidspacerick_h: I have now torn down that environment and frobware has a test charm that I can deploy locally13:16
voidspacerick_h: that defines useful stuff explicitly for playing with bindings13:16
rick_hvoidspace: yea, all good. Just noting that I can't find anything useful for what you were asking13:17
voidspacerick_h: thanks for lookinh13:17
voidspacerick_h: *looking13:17
voidspacerick_h: but frobware has useful tools for playing with network-get13:17
natefinchReview me? 2 line change: https://github.com/juju/juju/pull/667013:25
thumperbabbageclunk: https://github.com/juju/juju/pull/666913:56
=== jamespag` is now known as jamespage
natefinchperrito666: http://reports.vapour.ws/releases/4631/job/functional-backup-restore/attempt/490015:59
mupBug #1625624 changed: juju 2 doesn't remove openstack security groups <ci> <landscape> <openstack-provider> <sts> <juju:Fix Committed by gz> <juju 2.0:Fix Committed by gnuoy> <juju-core:Fix Released by gnuoy> <https://launchpad.net/bugs/1625624>16:50
=== frankban is now known as frankban|afk

Generated by irclog2html.py 2.7 by Marius Gedminas - find it at mg.pov.lt!