[00:31] burton-aus: standup? [02:34] * babbageclunk goes for a run === cmars` is now known as cmars === rogpeppe1 is now known as rogpeppe [22:18] bdx: hey, if you need I can help clean up those failing cleanups that are preventing a migration? [22:43] babbageclunk: that would be amazing [22:44] bdx: are you familiar with doing stuff in the mongo shell? [22:44] yeah [22:44] what do I need to do [22:44] bdx: cool [22:44] bdx: find the records in the cleanups collection and delete them [22:45] awesome [22:45] bdx: (they should be the only records in the collection, so that shouldn't be hard) [22:45] ok [22:46] bdx: You should be able to see them with db.cleanups.find().pretty() [22:49] babbageclunk: should I need to pass any args to get into the mongo shell? [22:49] ahh port [22:49] should be all eh? [22:50] bdx: hang on , pulling up the wiki page - I've got a script so I've forgotten all of the details. [22:51] bdx: you'll need the password from the agent config as well: https://github.com/juju/juju/wiki/Login-into-MongoDB [22:52] thats a goodie! [22:52] ok im in [22:53] bdx: can you see the problem cleanups? [22:53] http://paste.ubuntu.com/25135798/ [22:53] they are resources [22:53] strange [22:54] bdx: great, just like we'd expect from the log. [22:54] Oh, have you seen my comment on the bug? [22:54] do you think there might be a bug where juju isn't cleaning up resources adequately on application removal? [22:54] no, checking [22:55] Well, the error in the log is that the cleanup is failing because the resource is already removed. [22:56] So I'm fixing the cleanup to call that success (although I don't know why the resource has already gone). [22:56] wow [22:56] that makes sense [22:56] ok [22:56] Potentially the cleanup was queued but some other operation removed the resource before it ran? [22:57] I cant really be sure ... I think this is happening on another controller too [22:57] checking [22:59] babbageclunk: looks like I do have another one of these plaguing another controller http://paste.ubuntu.com/25135820/ [23:00] so I can just remove this entry from the db and all will be good? [23:00] to some extent [23:01] bdx: assuming you have the same error in the logs for that controller, then yes. [23:01] as far as getting migrations to work goes [23:01] Yup [23:06] babbageclunk: I can just run db.cleanups.remove() ? [23:06] to clear thee out [23:08] bdx: Might be better to do them by id just to make sure you don't inadvertently get any others. [23:09] entirely [23:09] bdx: So db.cleanups.remove({_id: ""129a3d11-4d70-4504-86b6-c8442c95ae12:ObjectIdHex(\"590a50bbfd51631474ae0182\")"}) [23:09] etc [23:09] I got the same resource cleanup error in the logs on the other controller http://paste.ubuntu.com/25135845/ [23:10] awesome, I was just getting there [23:10] thx [23:10] bdx: yeah, looks the same [23:11] bdx: ok, after that the migration should work (hopefully) [23:13] ok they are cleaned up [23:13] trying the migration now [23:17] babbageclunk: I'm migrating! [23:17] babbageclunk: many thanks [23:17] bdx: no worries! [23:46] anastasiamac: standup? [23:46] veebers: ^ [23:47] axw: ah oops, need a couple secs to get headset, omw