[02:02] <thumper> axw: can I grab you in about 30 min for a chat?
[02:03] <axw> thumper: yes but I need to go out for an appointment in 50 mins
[02:03] <thumper> ack
[02:17] <thumper> axw: now?
[02:17] <axw> thumper: sure, 1:1?
[02:17] <thumper> ack
[02:18] <thumper> axw: I had to go hunting for an old hangout link
[02:18] <axw> heh me too
[04:28]  * thumper EODs
[07:24] <codenameba> ▄▄▄▄▄▄▄▄▄▄▄▄ SPECIAL ANNOUNCEMENT: IRC.SUPERNETS.ORG #SUPERBOWL IS HAVING A SUPERBOWL WATCH PARTY ON FEB. 4TH. MESSAGE CHRONO FOR DETAILS!!  onwbnzhmn: tinwood rogpeppe cmars ▄▄▄▄▄▄▄▄▄▄
[07:24] <codenameba> ▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄ SPECIAL ANNOUNCEMENT: IRC.SUPERNETS.ORG #SUPERBOWL IS HAVING A SUPERBOWL WATCH PARTY ON FEB. 4TH. MESSAGE CHRONO FOR DETAILS!!  gcyboohq: alexlist coreycb jog_ ▄▄▄▄▄▄▄▄▄▄▄▄
[07:24] <codenameba> ▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄ SPECIAL ANNOUNCEMENT: IRC.SUPERNETS.ORG #SUPERBOWL IS HAVING A SUPERBOWL WATCH PARTY ON FEB. 4TH. MESSAGE CHRONO FOR DETAILS!!  zebvh: rogpeppe tasdomas joedborg ▄▄▄▄▄▄▄▄▄▄▄▄
[07:24] <codenameba> ▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄ SPECIAL ANNOUNCEMENT: IRC.SUPERNETS.ORG #SUPERBOWL IS HAVING A SUPERBOWL WATCH PARTY ON FEB. 4TH. MESSAGE CHRONO FOR DETAILS!!  onqha: blahdeblah Mmike meetingology ▄▄▄▄▄▄▄▄▄▄▄
[07:24] <codenameba> â–„â–„â–„â–„â–„â–„â–„â–„â–„â–„â–„â–„â–„â–„â–„â–„â–„ SPECIAL ANNOUNCEMENT: IRC.SUPERNETS.ORG #SUPERBOWL IS HAVING A SUPERBOWL WATCH PARTY ON FEB. 4TH. MESSAGE CHRONO FOR DETAILS!!  dvhznvpupg: meetingology jillr zeestrat â–„â–„â–„â–„â–„â–„â–„â–„â–„â–„â–„â–„â–„â–„â
[07:24] <codenameba> ▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄ SPECIAL ANNOUNCEMENT: IRC.SUPERNETS.ORG #SUPERBOWL IS HAVING A SUPERBOWL WATCH PARTY ON FEB. 4TH. MESSAGE CHRONO FOR DETAILS!!  gkizhbv: rick_h elmo jose ▄▄▄▄▄▄▄▄▄▄▄▄▄
[07:24] <codenameba> ▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄ SPECIAL ANNOUNCEMENT: IRC.SUPERNETS.ORG #SUPERBOWL IS HAVING A SUPERBOWL WATCH PARTY ON FEB. 4TH. MESSAGE CHRONO FOR DETAILS!!  fhhzaeg: ejat rick_h redir ▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄
[07:24] <codenameba> ▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄ SPECIAL ANNOUNCEMENT: IRC.SUPERNETS.ORG #SUPERBOWL IS HAVING A SUPERBOWL WATCH PARTY ON FEB. 4TH. MESSAGE CHRONO FOR DETAILS!!  ksqzlqs: niemeyer chaology blahdeblah ▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄
[07:24] <codenameba> â–„â–„â–„â–„â–„â–„â–„â–„â–„â–„â–„â–„â–„â–„â–„â–„â–„â–„â–„ SPECIAL ANNOUNCEMENT: IRC.SUPERNETS.ORG #SUPERBOWL IS HAVING A SUPERBOWL WATCH PARTY ON FEB. 4TH. MESSAGE CHRONO FOR DETAILS!!  zjwrxodat: mpontillo jw4 bogdanteleaga â–„â–„â–„â–„â–„â–„â–„â–„â–„â–„â–„â–„â–
[07:25] <codenameba> â–„â–„â–„â–„â–„â–„â–„â–„â–„â–„â–„â–„â–„â–„â–„â–„ SPECIAL ANNOUNCEMENT: IRC.SUPERNETS.ORG #SUPERBOWL IS HAVING A SUPERBOWL WATCH PARTY ON FEB. 4TH. MESSAGE CHRONO FOR DETAILS!!  lyndkhssz: niedbalski anastasiamac tvansteenburgh â–„â–„â–„â–„â–„â–„â–„â–„â–„â–„â–„â–„â–„â–„â
[14:34] <jam> rogpeppe1: I might have targetted the wrong branch for the mgo change. I probably wanted to target the v1 branch, looks like I targeted master by mistake
[14:35] <rogpeppe1> jam: no, looks like the one that landed was targeting v1
[14:35] <rogpeppe1> jam: https://github.com/go-macaroon-bakery/macaroon-bakery/pull/196
[14:46] <jam> rogpeppe1: thx, I guess I just missed seeing it from gopkg, but it seems to be there
[20:00] <thumper> morning
[20:26] <alexisb> morning thumper
[20:26] <thumper> alexisb: hey, still miss you
[20:26] <alexisb> :) I miss you guys as well
[20:35] <zeestrat> thumper: morning. I've got about 30-45 min before EOD if you'd like me to poke at #1746265
[20:35] <mup> Bug #1746265: juju-upgrade from 2.2.9 to 2.3.2 fails with state changing too quickly <upgrade-juju> <juju:Triaged> <juju 2.2:Won't Fix> <juju 2.3:Triaged> <https://launchpad.net/bugs/1746265>
[20:36] <thumper> zeestrat: ack, otp just now
[20:38] <zeestrat> Roger. If I'm out, just leave some steps in the comments and I can mess about and get some results for monday.
[20:41] <thumper> zeestrat: can you do google hangouts?
[20:44] <zeestrat> thumper: sorry, remoting from home atm without a good setup
[20:44] <thumper> hmm...
[20:44] <thumper> ok
[20:44] <thumper> I suppose I can talk you through it :)
[20:45] <thumper> zeestrat: so, am I right in thinking that you have a reproduction of the upgrade problem right now?
[20:45] <thumper> where it is mid-upgrade
[20:45] <thumper> and one agent hasn't restarted ?
[20:47] <zeestrat> Yes. Though the agents in controller 0,1,2 all seem to be spewing different error messages so I'm not sure which one's are stuck
[20:47] <thumper> :)
[20:47] <thumper> kk
[20:48] <thumper> the first thing to do is to work out which one is stuck
[20:48] <thumper> so... open terminals into each of the three machines
[20:49]  * thumper quickly bootstraps a lxd to make sure he gets the commands right
[20:49] <zeestrat> :)
[20:49] <thumper> you want to look for 'running jujud' in the machine log files
[20:49] <thumper> grep machine*.log 'running jujud' | tail
[20:49] <thumper> something like that
[20:49] <thumper> look for the timestamp of when each of the agents most recently started
[20:51] <zeestrat> in decending order, machine-0 11:12:51, machine-2 11:15:25, machine-1 11:15:27, so machine-1 was the last one to the party
[20:52] <thumper> does the 11:15 time match the upgrade-juju?
[20:52] <thumper> roughly?
[20:53] <thumper> can you pastebin a bunch of the logs from machine-1 to pastebin.ubuntu.com ?
[20:53] <zeestrat> They're all in the bug so you could take a look there :)
[20:54] <zeestrat> Time sounds right, though I don't have the exact timestamps
[20:57] <thumper> zeestrat: how is your mongo foo ?
[20:58] <thumper> mongo fu?
[20:58] <zeestrat> fubar
[20:59] <zeestrat> But if you have some docs I'll probably get through them.
[20:59] <thumper> let me pastebin you a cool command
[21:00] <zeestrat> These are still usable? https://github.com/juju/juju/wiki/Login-into-MongoDB
[21:00] <thumper> https://pastebin.ubuntu.com/26502452/
[21:00] <thumper> zeestrat: this is better
[21:00] <thumper> put that in your path somewhere as executable called 'juju-db'
[21:01] <thumper> then you can just type 'juju db' to get a mongo shell on the controller'
[21:02] <zeestrat> Very handy indeed. What's next?
[21:04] <thumper> zeestrat: I need to see the content of the leases collection
[21:05] <thumper> which you can get with db.leases.find().pretty()
[21:05] <thumper> it appears that the leases are a bit fubared
[21:05] <zeestrat> Would you like that for all controlelrs?
[21:05] <thumper> no, just one
[21:05] <thumper> since the db is replicated, it should be the same info
[21:05] <thumper> no matter which we look at
[21:05] <zeestrat> Roger
[21:05] <thumper> although by default you can only query the master
[21:06] <thumper> so if the command doesn't show "juju:PRIMARY"
[21:06] <thumper> it is probably quicker to try another machine rather than me digging the command to query from replicas
[21:07] <zeestrat> Yeah, sorry, was just finding a nice way to dump mongo into pastebinit
[21:08] <zeestrat> https://paste.ubuntu.com/26502475/
[21:09] <zeestrat> Ah, didn't spot the type "it" for more. Here's the complete dump: http://paste.ubuntu.com/26502478/
[21:11] <thumper> cool
[21:11] <thumper> now I want to get the transaction that is failing...
[21:11] <thumper> this will be more interesting
[21:15] <thumper> db.txns.find({"o.c": "leases", "s": 5}).sort({"$natural":-1}).limit(1).pretty()
[21:15] <thumper> zeestrat: which says
[21:15] <thumper> find the last transaction that was aborted that tried to modify the leases collection
[21:16] <zeestrat> http://paste.ubuntu.com/26502504/
[21:17] <thumper> ok, thanks
[21:17] <thumper> now I have what I need to work out why it is failing
[21:17] <thumper> if you want to wait a few minutes I might be able to get you past it...
[21:19] <zeestrat> I've got about 10 minutes, but no rush to get it fixed right now. A comment in the bug will do fine.
[21:20] <zeestrat> Do let me know if you want to run some other permutations or fixes over the weekend
[21:20] <thumper> zeestrat: ok, you best go then
[21:22] <zeestrat> thumper: Thanks for taking the time and effort. We owe you and the team a round of alcoholic beverages if any of you ever stop by Norway :)
[21:22] <thumper> :)
[21:22] <thumper> unlikely, but not impossible
[21:23] <zeestrat> beeroverip.org just doesn't hit the spot
[21:23] <zeestrat> Thanks again and have nice weekend.
[21:24] <thumper> cheers, you too
[21:47] <kwmonroe> hey thumper, is it true that only 1 hook will ever run on a unit?  so like if a million apps are smashed onto a single unit, the agent will coordinate such that only 1 hook is running at any given time?
[21:48] <kwmonroe> the key part there being "at 1 time", which i didn't make clear in that first part
[21:51] <thumper> kwmonroe: yes
[21:52] <kwmonroe> thx
[21:52] <thumper> kwmonroe: although I think you mean a million units on a single machine
[21:52] <thumper> but the answer is the same
[21:52] <kwmonroe> thumper: let's be real here, there are a lot of things i would have changed about that initial ping if i could.
[21:53] <thumper> :)