[00:31] <axw> burton-aus: standup?
[02:34]  * babbageclunk goes for a run
[22:18] <babbageclunk> bdx: hey, if you need I can help clean up those failing cleanups that are preventing a migration?
[22:43] <bdx> babbageclunk: that would be amazing
[22:44] <babbageclunk> bdx: are you familiar with doing stuff in the mongo shell?
[22:44] <bdx> yeah
[22:44] <bdx> what do I need to do
[22:44] <babbageclunk> bdx: cool
[22:44] <babbageclunk> bdx: find the records in the cleanups collection and delete them
[22:45] <bdx> awesome
[22:45] <babbageclunk> bdx: (they should be the only records in the collection, so that shouldn't be hard)
[22:45] <bdx> ok
[22:46] <babbageclunk> bdx: You should be able to see them with db.cleanups.find().pretty()
[22:49] <bdx> babbageclunk: should I need to pass any args to get into the mongo shell?
[22:49] <bdx> ahh port
[22:49] <bdx> should be all eh?
[22:50] <babbageclunk> bdx: hang on , pulling up the wiki page - I've got a script so I've forgotten all of the details.
[22:51] <babbageclunk> bdx: you'll need the password from the agent config as well: https://github.com/juju/juju/wiki/Login-into-MongoDB
[22:52] <bdx> thats a goodie!
[22:52] <bdx> ok im in
[22:53] <babbageclunk> bdx: can you see the problem cleanups?
[22:53] <bdx> http://paste.ubuntu.com/25135798/
[22:53] <bdx> they are resources
[22:53] <bdx> strange
[22:54] <babbageclunk> bdx: great, just like we'd expect from the log.
[22:54] <babbageclunk> Oh, have you seen my comment on the bug?
[22:54] <bdx> do you think there might be a bug where juju isn't cleaning up resources adequately on application removal?
[22:54] <bdx> no, checking
[22:55] <babbageclunk> Well, the error in the log is that the cleanup is failing because the resource is already removed.
[22:56] <babbageclunk> So I'm fixing the cleanup to call that success (although I don't know why the resource has already gone).
[22:56] <bdx> wow
[22:56] <bdx> that makes sense
[22:56] <bdx> ok
[22:56] <babbageclunk> Potentially the cleanup was queued but some other operation removed the resource before it ran?
[22:57] <bdx> I cant really be sure ... I think this is happening on another controller too
[22:57] <bdx> checking
[22:59] <bdx> babbageclunk: looks like I do have another one of these plaguing another controller http://paste.ubuntu.com/25135820/
[23:00] <bdx> so I can just remove this entry from the db and all will be good?
[23:00] <bdx> to some extent
[23:01] <babbageclunk> bdx: assuming you have the same error in the logs for that controller, then yes.
[23:01] <bdx> as far as getting migrations to work goes
[23:01] <babbageclunk> Yup
[23:06] <bdx> babbageclunk: I can just run db.cleanups.remove() ?
[23:06] <bdx> to clear thee out
[23:08] <babbageclunk> bdx: Might be better to do them by id just to make sure you don't inadvertently get any others.
[23:09] <bdx> entirely
[23:09] <babbageclunk> bdx: So db.cleanups.remove({_id: ""129a3d11-4d70-4504-86b6-c8442c95ae12:ObjectIdHex(\"590a50bbfd51631474ae0182\")"})
[23:09] <babbageclunk> etc
[23:09] <bdx> I got the same resource cleanup error in the logs on the other controller http://paste.ubuntu.com/25135845/
[23:10] <bdx> awesome, I was just getting there
[23:10] <bdx> thx
[23:10] <babbageclunk> bdx: yeah, looks the same
[23:11] <babbageclunk> bdx: ok, after that the migration should work (hopefully)
[23:13] <bdx> ok they are cleaned up
[23:13] <bdx> trying the migration now
[23:17] <bdx> babbageclunk: I'm migrating!
[23:17] <bdx> babbageclunk: many thanks
[23:17] <babbageclunk> bdx: no worries!
[23:46] <axw> anastasiamac: standup?
[23:46] <axw> veebers: ^
[23:47] <veebers> axw: ah oops, need a couple secs to get headset, omw