| thumper | axw: can I grab you in about 30 min for a chat? | 02:02 |
|---|---|---|
| axw | thumper: yes but I need to go out for an appointment in 50 mins | 02:03 |
| thumper | ack | 02:03 |
| thumper | axw: now? | 02:17 |
| axw | thumper: sure, 1:1? | 02:17 |
| thumper | ack | 02:17 |
| thumper | axw: I had to go hunting for an old hangout link | 02:18 |
| axw | heh me too | 02:18 |
| * thumper EODs | 04:28 | |
| codenameba | ▄▄▄▄▄▄▄▄▄▄▄▄ SPECIAL ANNOUNCEMENT: IRC.SUPERNETS.ORG #SUPERBOWL IS HAVING A SUPERBOWL WATCH PARTY ON FEB. 4TH. MESSAGE CHRONO FOR DETAILS!! onwbnzhmn: tinwood rogpeppe cmars ▄▄▄▄▄▄▄▄▄▄ | 07:24 |
| codenameba | ▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄ SPECIAL ANNOUNCEMENT: IRC.SUPERNETS.ORG #SUPERBOWL IS HAVING A SUPERBOWL WATCH PARTY ON FEB. 4TH. MESSAGE CHRONO FOR DETAILS!! gcyboohq: alexlist coreycb jog_ ▄▄▄▄▄▄▄▄▄▄▄▄ | 07:24 |
| codenameba | ▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄ SPECIAL ANNOUNCEMENT: IRC.SUPERNETS.ORG #SUPERBOWL IS HAVING A SUPERBOWL WATCH PARTY ON FEB. 4TH. MESSAGE CHRONO FOR DETAILS!! zebvh: rogpeppe tasdomas joedborg ▄▄▄▄▄▄▄▄▄▄▄▄ | 07:24 |
| codenameba | ▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄ SPECIAL ANNOUNCEMENT: IRC.SUPERNETS.ORG #SUPERBOWL IS HAVING A SUPERBOWL WATCH PARTY ON FEB. 4TH. MESSAGE CHRONO FOR DETAILS!! onqha: blahdeblah Mmike meetingology ▄▄▄▄▄▄▄▄▄▄▄ | 07:24 |
| codenameba | â–„â–„â–„â–„â–„â–„â–„â–„â–„â–„â–„â–„â–„â–„â–„â–„â–„ SPECIAL ANNOUNCEMENT: IRC.SUPERNETS.ORG #SUPERBOWL IS HAVING A SUPERBOWL WATCH PARTY ON FEB. 4TH. MESSAGE CHRONO FOR DETAILS!! dvhznvpupg: meetingology jillr zeestrat â–„â–„â–„â–„â–„â–„â–„â–„â–„â–„â–„â–„â–„â–„â | 07:24 |
| codenameba | ▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄ SPECIAL ANNOUNCEMENT: IRC.SUPERNETS.ORG #SUPERBOWL IS HAVING A SUPERBOWL WATCH PARTY ON FEB. 4TH. MESSAGE CHRONO FOR DETAILS!! gkizhbv: rick_h elmo jose ▄▄▄▄▄▄▄▄▄▄▄▄▄ | 07:24 |
| codenameba | ▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄ SPECIAL ANNOUNCEMENT: IRC.SUPERNETS.ORG #SUPERBOWL IS HAVING A SUPERBOWL WATCH PARTY ON FEB. 4TH. MESSAGE CHRONO FOR DETAILS!! fhhzaeg: ejat rick_h redir ▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄ | 07:24 |
| codenameba | ▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄ SPECIAL ANNOUNCEMENT: IRC.SUPERNETS.ORG #SUPERBOWL IS HAVING A SUPERBOWL WATCH PARTY ON FEB. 4TH. MESSAGE CHRONO FOR DETAILS!! ksqzlqs: niemeyer chaology blahdeblah ▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄ | 07:24 |
| codenameba | â–„â–„â–„â–„â–„â–„â–„â–„â–„â–„â–„â–„â–„â–„â–„â–„â–„â–„â–„ SPECIAL ANNOUNCEMENT: IRC.SUPERNETS.ORG #SUPERBOWL IS HAVING A SUPERBOWL WATCH PARTY ON FEB. 4TH. MESSAGE CHRONO FOR DETAILS!! zjwrxodat: mpontillo jw4 bogdanteleaga â–„â–„â–„â–„â–„â–„â–„â–„â–„â–„â–„â–„â– | 07:24 |
| codenameba | â–„â–„â–„â–„â–„â–„â–„â–„â–„â–„â–„â–„â–„â–„â–„â–„ SPECIAL ANNOUNCEMENT: IRC.SUPERNETS.ORG #SUPERBOWL IS HAVING A SUPERBOWL WATCH PARTY ON FEB. 4TH. MESSAGE CHRONO FOR DETAILS!! lyndkhssz: niedbalski anastasiamac tvansteenburgh â–„â–„â–„â–„â–„â–„â–„â–„â–„â–„â–„â–„â–„â–„â | 07:25 |
| === frankban|afk is now known as frankban | ||
| === mup_ is now known as mup | ||
| jam | rogpeppe1: I might have targetted the wrong branch for the mgo change. I probably wanted to target the v1 branch, looks like I targeted master by mistake | 14:34 |
| rogpeppe1 | jam: no, looks like the one that landed was targeting v1 | 14:35 |
| rogpeppe1 | jam: https://github.com/go-macaroon-bakery/macaroon-bakery/pull/196 | 14:35 |
| jam | rogpeppe1: thx, I guess I just missed seeing it from gopkg, but it seems to be there | 14:46 |
| === frankban is now known as frankban|afk | ||
| thumper | morning | 20:00 |
| alexisb | morning thumper | 20:26 |
| thumper | alexisb: hey, still miss you | 20:26 |
| alexisb | :) I miss you guys as well | 20:26 |
| zeestrat | thumper: morning. I've got about 30-45 min before EOD if you'd like me to poke at #1746265 | 20:35 |
| mup | Bug #1746265: juju-upgrade from 2.2.9 to 2.3.2 fails with state changing too quickly <upgrade-juju> <juju:Triaged> <juju 2.2:Won't Fix> <juju 2.3:Triaged> <https://launchpad.net/bugs/1746265> | 20:35 |
| thumper | zeestrat: ack, otp just now | 20:36 |
| zeestrat | Roger. If I'm out, just leave some steps in the comments and I can mess about and get some results for monday. | 20:38 |
| thumper | zeestrat: can you do google hangouts? | 20:41 |
| zeestrat | thumper: sorry, remoting from home atm without a good setup | 20:44 |
| thumper | hmm... | 20:44 |
| thumper | ok | 20:44 |
| thumper | I suppose I can talk you through it :) | 20:44 |
| thumper | zeestrat: so, am I right in thinking that you have a reproduction of the upgrade problem right now? | 20:45 |
| thumper | where it is mid-upgrade | 20:45 |
| thumper | and one agent hasn't restarted ? | 20:45 |
| zeestrat | Yes. Though the agents in controller 0,1,2 all seem to be spewing different error messages so I'm not sure which one's are stuck | 20:47 |
| thumper | :) | 20:47 |
| thumper | kk | 20:47 |
| thumper | the first thing to do is to work out which one is stuck | 20:48 |
| thumper | so... open terminals into each of the three machines | 20:48 |
| * thumper quickly bootstraps a lxd to make sure he gets the commands right | 20:49 | |
| zeestrat | :) | 20:49 |
| thumper | you want to look for 'running jujud' in the machine log files | 20:49 |
| thumper | grep machine*.log 'running jujud' | tail | 20:49 |
| thumper | something like that | 20:49 |
| thumper | look for the timestamp of when each of the agents most recently started | 20:49 |
| zeestrat | in decending order, machine-0 11:12:51, machine-2 11:15:25, machine-1 11:15:27, so machine-1 was the last one to the party | 20:51 |
| thumper | does the 11:15 time match the upgrade-juju? | 20:52 |
| thumper | roughly? | 20:52 |
| thumper | can you pastebin a bunch of the logs from machine-1 to pastebin.ubuntu.com ? | 20:53 |
| zeestrat | They're all in the bug so you could take a look there :) | 20:53 |
| zeestrat | Time sounds right, though I don't have the exact timestamps | 20:54 |
| thumper | zeestrat: how is your mongo foo ? | 20:57 |
| thumper | mongo fu? | 20:58 |
| zeestrat | fubar | 20:58 |
| zeestrat | But if you have some docs I'll probably get through them. | 20:59 |
| thumper | let me pastebin you a cool command | 20:59 |
| zeestrat | These are still usable? https://github.com/juju/juju/wiki/Login-into-MongoDB | 21:00 |
| thumper | https://pastebin.ubuntu.com/26502452/ | 21:00 |
| thumper | zeestrat: this is better | 21:00 |
| thumper | put that in your path somewhere as executable called 'juju-db' | 21:00 |
| thumper | then you can just type 'juju db' to get a mongo shell on the controller' | 21:01 |
| zeestrat | Very handy indeed. What's next? | 21:02 |
| thumper | zeestrat: I need to see the content of the leases collection | 21:04 |
| thumper | which you can get with db.leases.find().pretty() | 21:05 |
| thumper | it appears that the leases are a bit fubared | 21:05 |
| zeestrat | Would you like that for all controlelrs? | 21:05 |
| thumper | no, just one | 21:05 |
| thumper | since the db is replicated, it should be the same info | 21:05 |
| thumper | no matter which we look at | 21:05 |
| zeestrat | Roger | 21:05 |
| thumper | although by default you can only query the master | 21:05 |
| thumper | so if the command doesn't show "juju:PRIMARY" | 21:06 |
| thumper | it is probably quicker to try another machine rather than me digging the command to query from replicas | 21:06 |
| zeestrat | Yeah, sorry, was just finding a nice way to dump mongo into pastebinit | 21:07 |
| zeestrat | https://paste.ubuntu.com/26502475/ | 21:08 |
| zeestrat | Ah, didn't spot the type "it" for more. Here's the complete dump: http://paste.ubuntu.com/26502478/ | 21:09 |
| thumper | cool | 21:11 |
| thumper | now I want to get the transaction that is failing... | 21:11 |
| thumper | this will be more interesting | 21:11 |
| thumper | db.txns.find({"o.c": "leases", "s": 5}).sort({"$natural":-1}).limit(1).pretty() | 21:15 |
| thumper | zeestrat: which says | 21:15 |
| thumper | find the last transaction that was aborted that tried to modify the leases collection | 21:15 |
| zeestrat | http://paste.ubuntu.com/26502504/ | 21:16 |
| thumper | ok, thanks | 21:17 |
| thumper | now I have what I need to work out why it is failing | 21:17 |
| thumper | if you want to wait a few minutes I might be able to get you past it... | 21:17 |
| zeestrat | I've got about 10 minutes, but no rush to get it fixed right now. A comment in the bug will do fine. | 21:19 |
| zeestrat | Do let me know if you want to run some other permutations or fixes over the weekend | 21:20 |
| thumper | zeestrat: ok, you best go then | 21:20 |
| zeestrat | thumper: Thanks for taking the time and effort. We owe you and the team a round of alcoholic beverages if any of you ever stop by Norway :) | 21:22 |
| thumper | :) | 21:22 |
| thumper | unlikely, but not impossible | 21:22 |
| zeestrat | beeroverip.org just doesn't hit the spot | 21:23 |
| zeestrat | Thanks again and have nice weekend. | 21:23 |
| thumper | cheers, you too | 21:24 |
| kwmonroe | hey thumper, is it true that only 1 hook will ever run on a unit? so like if a million apps are smashed onto a single unit, the agent will coordinate such that only 1 hook is running at any given time? | 21:47 |
| kwmonroe | the key part there being "at 1 time", which i didn't make clear in that first part | 21:48 |
| thumper | kwmonroe: yes | 21:51 |
| kwmonroe | thx | 21:52 |
| thumper | kwmonroe: although I think you mean a million units on a single machine | 21:52 |
| thumper | but the answer is the same | 21:52 |
| kwmonroe | thumper: let's be real here, there are a lot of things i would have changed about that initial ping if i could. | 21:52 |
| thumper | :) | 21:53 |
Generated by irclog2html.py 2.7 by Marius Gedminas - find it at mg.pov.lt!