[02:02] axw: can I grab you in about 30 min for a chat? [02:03] thumper: yes but I need to go out for an appointment in 50 mins [02:03] ack [02:17] axw: now? [02:17] thumper: sure, 1:1? [02:17] ack [02:18] axw: I had to go hunting for an old hangout link [02:18] heh me too [04:28] * thumper EODs [07:24] ▄▄▄▄▄▄▄▄▄▄▄▄ SPECIAL ANNOUNCEMENT: IRC.SUPERNETS.ORG #SUPERBOWL IS HAVING A SUPERBOWL WATCH PARTY ON FEB. 4TH. MESSAGE CHRONO FOR DETAILS!! onwbnzhmn: tinwood rogpeppe cmars ▄▄▄▄▄▄▄▄▄▄ [07:24] ▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄ SPECIAL ANNOUNCEMENT: IRC.SUPERNETS.ORG #SUPERBOWL IS HAVING A SUPERBOWL WATCH PARTY ON FEB. 4TH. MESSAGE CHRONO FOR DETAILS!! gcyboohq: alexlist coreycb jog_ ▄▄▄▄▄▄▄▄▄▄▄▄ [07:24] ▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄ SPECIAL ANNOUNCEMENT: IRC.SUPERNETS.ORG #SUPERBOWL IS HAVING A SUPERBOWL WATCH PARTY ON FEB. 4TH. MESSAGE CHRONO FOR DETAILS!! zebvh: rogpeppe tasdomas joedborg ▄▄▄▄▄▄▄▄▄▄▄▄ [07:24] ▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄ SPECIAL ANNOUNCEMENT: IRC.SUPERNETS.ORG #SUPERBOWL IS HAVING A SUPERBOWL WATCH PARTY ON FEB. 4TH. MESSAGE CHRONO FOR DETAILS!! onqha: blahdeblah Mmike meetingology ▄▄▄▄▄▄▄▄▄▄▄ [07:24] â–„â–„â–„â–„â–„â–„â–„â–„â–„â–„â–„â–„â–„â–„â–„â–„â–„ SPECIAL ANNOUNCEMENT: IRC.SUPERNETS.ORG #SUPERBOWL IS HAVING A SUPERBOWL WATCH PARTY ON FEB. 4TH. MESSAGE CHRONO FOR DETAILS!! dvhznvpupg: meetingology jillr zeestrat â–„â–„â–„â–„â–„â–„â–„â–„â–„â–„â–„â–„â–„â–„â [07:24] ▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄ SPECIAL ANNOUNCEMENT: IRC.SUPERNETS.ORG #SUPERBOWL IS HAVING A SUPERBOWL WATCH PARTY ON FEB. 4TH. MESSAGE CHRONO FOR DETAILS!! gkizhbv: rick_h elmo jose ▄▄▄▄▄▄▄▄▄▄▄▄▄ [07:24] ▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄ SPECIAL ANNOUNCEMENT: IRC.SUPERNETS.ORG #SUPERBOWL IS HAVING A SUPERBOWL WATCH PARTY ON FEB. 4TH. MESSAGE CHRONO FOR DETAILS!! fhhzaeg: ejat rick_h redir ▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄ [07:24] ▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄ SPECIAL ANNOUNCEMENT: IRC.SUPERNETS.ORG #SUPERBOWL IS HAVING A SUPERBOWL WATCH PARTY ON FEB. 4TH. MESSAGE CHRONO FOR DETAILS!! ksqzlqs: niemeyer chaology blahdeblah ▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄ [07:24] â–„â–„â–„â–„â–„â–„â–„â–„â–„â–„â–„â–„â–„â–„â–„â–„â–„â–„â–„ SPECIAL ANNOUNCEMENT: IRC.SUPERNETS.ORG #SUPERBOWL IS HAVING A SUPERBOWL WATCH PARTY ON FEB. 4TH. MESSAGE CHRONO FOR DETAILS!! zjwrxodat: mpontillo jw4 bogdanteleaga â–„â–„â–„â–„â–„â–„â–„â–„â–„â–„â–„â–„â– [07:25] â–„â–„â–„â–„â–„â–„â–„â–„â–„â–„â–„â–„â–„â–„â–„â–„ SPECIAL ANNOUNCEMENT: IRC.SUPERNETS.ORG #SUPERBOWL IS HAVING A SUPERBOWL WATCH PARTY ON FEB. 4TH. MESSAGE CHRONO FOR DETAILS!! lyndkhssz: niedbalski anastasiamac tvansteenburgh â–„â–„â–„â–„â–„â–„â–„â–„â–„â–„â–„â–„â–„â–„â === frankban|afk is now known as frankban === mup_ is now known as mup [14:34] rogpeppe1: I might have targetted the wrong branch for the mgo change. I probably wanted to target the v1 branch, looks like I targeted master by mistake [14:35] jam: no, looks like the one that landed was targeting v1 [14:35] jam: https://github.com/go-macaroon-bakery/macaroon-bakery/pull/196 [14:46] rogpeppe1: thx, I guess I just missed seeing it from gopkg, but it seems to be there === frankban is now known as frankban|afk [20:00] morning [20:26] morning thumper [20:26] alexisb: hey, still miss you [20:26] :) I miss you guys as well [20:35] thumper: morning. I've got about 30-45 min before EOD if you'd like me to poke at #1746265 [20:35] Bug #1746265: juju-upgrade from 2.2.9 to 2.3.2 fails with state changing too quickly [20:36] zeestrat: ack, otp just now [20:38] Roger. If I'm out, just leave some steps in the comments and I can mess about and get some results for monday. [20:41] zeestrat: can you do google hangouts? [20:44] thumper: sorry, remoting from home atm without a good setup [20:44] hmm... [20:44] ok [20:44] I suppose I can talk you through it :) [20:45] zeestrat: so, am I right in thinking that you have a reproduction of the upgrade problem right now? [20:45] where it is mid-upgrade [20:45] and one agent hasn't restarted ? [20:47] Yes. Though the agents in controller 0,1,2 all seem to be spewing different error messages so I'm not sure which one's are stuck [20:47] :) [20:47] kk [20:48] the first thing to do is to work out which one is stuck [20:48] so... open terminals into each of the three machines [20:49] * thumper quickly bootstraps a lxd to make sure he gets the commands right [20:49] :) [20:49] you want to look for 'running jujud' in the machine log files [20:49] grep machine*.log 'running jujud' | tail [20:49] something like that [20:49] look for the timestamp of when each of the agents most recently started [20:51] in decending order, machine-0 11:12:51, machine-2 11:15:25, machine-1 11:15:27, so machine-1 was the last one to the party [20:52] does the 11:15 time match the upgrade-juju? [20:52] roughly? [20:53] can you pastebin a bunch of the logs from machine-1 to pastebin.ubuntu.com ? [20:53] They're all in the bug so you could take a look there :) [20:54] Time sounds right, though I don't have the exact timestamps [20:57] zeestrat: how is your mongo foo ? [20:58] mongo fu? [20:58] fubar [20:59] But if you have some docs I'll probably get through them. [20:59] let me pastebin you a cool command [21:00] These are still usable? https://github.com/juju/juju/wiki/Login-into-MongoDB [21:00] https://pastebin.ubuntu.com/26502452/ [21:00] zeestrat: this is better [21:00] put that in your path somewhere as executable called 'juju-db' [21:01] then you can just type 'juju db' to get a mongo shell on the controller' [21:02] Very handy indeed. What's next? [21:04] zeestrat: I need to see the content of the leases collection [21:05] which you can get with db.leases.find().pretty() [21:05] it appears that the leases are a bit fubared [21:05] Would you like that for all controlelrs? [21:05] no, just one [21:05] since the db is replicated, it should be the same info [21:05] no matter which we look at [21:05] Roger [21:05] although by default you can only query the master [21:06] so if the command doesn't show "juju:PRIMARY" [21:06] it is probably quicker to try another machine rather than me digging the command to query from replicas [21:07] Yeah, sorry, was just finding a nice way to dump mongo into pastebinit [21:08] https://paste.ubuntu.com/26502475/ [21:09] Ah, didn't spot the type "it" for more. Here's the complete dump: http://paste.ubuntu.com/26502478/ [21:11] cool [21:11] now I want to get the transaction that is failing... [21:11] this will be more interesting [21:15] db.txns.find({"o.c": "leases", "s": 5}).sort({"$natural":-1}).limit(1).pretty() [21:15] zeestrat: which says [21:15] find the last transaction that was aborted that tried to modify the leases collection [21:16] http://paste.ubuntu.com/26502504/ [21:17] ok, thanks [21:17] now I have what I need to work out why it is failing [21:17] if you want to wait a few minutes I might be able to get you past it... [21:19] I've got about 10 minutes, but no rush to get it fixed right now. A comment in the bug will do fine. [21:20] Do let me know if you want to run some other permutations or fixes over the weekend [21:20] zeestrat: ok, you best go then [21:22] thumper: Thanks for taking the time and effort. We owe you and the team a round of alcoholic beverages if any of you ever stop by Norway :) [21:22] :) [21:22] unlikely, but not impossible [21:23] beeroverip.org just doesn't hit the spot [21:23] Thanks again and have nice weekend. [21:24] cheers, you too [21:47] hey thumper, is it true that only 1 hook will ever run on a unit? so like if a million apps are smashed onto a single unit, the agent will coordinate such that only 1 hook is running at any given time? [21:48] the key part there being "at 1 time", which i didn't make clear in that first part [21:51] kwmonroe: yes [21:52] thx [21:52] kwmonroe: although I think you mean a million units on a single machine [21:52] but the answer is the same [21:52] thumper: let's be real here, there are a lot of things i would have changed about that initial ping if i could. [21:53] :)