[12:20] <benji> gary_poster: a church friend is taking Katie to the one doctor appointment so I just added a half vacation day for Thursday afternoon to take her to the other
[12:27] <gary_poster> benji, somehow missed that IRC msg, but I see it now, thanks.  I'll go approve
[12:27] <benji> thanks
[12:28] <gary_poster> approved
[12:28] <gary_poster> bac benji danilos gmb, call in 2
[12:28] <bac> ko
[12:28] <gmb> ok
[12:30] <danilos> gary_poster, actually I am on, headset doesn't work
[12:31] <danilos> gary_poster, yeah, you are on the speakers :)
[12:31] <danilos> hanging up so I fix that :)
[12:32] <gary_poster> danilos keeps rejecting my Skype overtures
[12:33] <danilos> sorry
[12:33] <danilos> my microphone still doesn't work
[12:34] <danilos> gary_poster, haha :)
[12:34] <danilos> gary_poster, incidentlog (linked from it)?
[12:34] <danilos> gary_poster, I'll check it up, ok
[12:37] <danilos> danilos, I have plenty of things to say! :)
[12:37] <danilos> gary_poster, ^
[12:38] <danilos> gary_poster, nah, just kidding, thanks :)
[12:50] <danilos> ok, microphone fixed, I had to install libpulse0:i386 now that skype is installed in an entirely different way
[13:09] <benji> oops, I logged into the Canonical server twice
[13:13]  * gmb -> late lunch / errands
[14:59] <gary_poster> early early lunch (== taking care of baby while wife is out)
[19:06] <bac> gary_poster: got a sec for a skype?
[19:06] <gary_poster> sure bac
[19:06] <gary_poster> logged on now bac
[19:26] <benji> How does ec2 land communicate with PQM?  I had an ec2 run that ended without me getting any notification.  It at least started and ran for the expected amount of time.
[19:34] <bac> gary_poster: https://pastebin.canonical.com/53482/
[19:34] <gary_poster> benji, it sends an email
[19:34] <bac> benji: like all PQM submissions it is via email
[19:34] <gary_poster> bac, looking
[19:35] <gary_poster> bac, try
[19:35] <gary_poster> !print rec
[19:35] <benji> hmm, I recently changed my bzr whoami setting (because ec2 land stopped working because my GPG key has my benji@benjiyork.com address not my @canonical.com address); I wonder if that is the problem.
[19:35] <gary_poster> oh, but that gives you more than you want bac
[19:35] <gary_poster> benji, that sounds suspucious, certainly
[19:35] <benji> ("p rec" is easier to type)
[19:36] <benji> (and "pp rec" is pretty-printed)
[19:36] <gary_poster> the pretty print won't help here; it is a string
[19:37] <gary_poster> so in fact !print is nicer
[19:37] <gary_poster> (p will give you \n s in it and stuff)
[19:37] <benji> sounds like you need ppp ;)
[19:37] <gary_poster> :-)
[19:37] <benji> oh you're right, I always forget that "p" is really the repr, that's stupid
[19:37] <gary_poster> except when it's not :-)
[19:38] <bac> gary_poster:  hey, cool, the __repr__ is much nicer than the parts
[19:38] <gary_poster> yeah
[19:38] <bac> let me paste that
[19:38] <bac> and you don't need the !  :)
[19:38] <gary_poster> heh, I am paranoid in pdb
[19:38] <benji> well, if "p" did print, then if you wanted the repr you could just do "!rec"
[19:39] <gary_poster> true, benji
[19:39] <bac> https://pastebin.canonical.com/53484/
[19:41] <benji> I could just make an alias to override "p".  I guess an economist would say that I therefore don't really care.
[19:41] <gary_poster> OK, so that's as we expected, to some degree, bac: the first thing that happens is the insert, then something else happens.  I would personally just be stupid and put a pdb right before the flush now.  Another approach would be to instrument the storm stuff and then put a conditional exactly when what we care about happens.  I'll describe that, since it might be more practical
[19:43] <gary_poster> so the easiest thing to do to add instrumentation in these tracebacks is to make a variable that has the name __traceback_info__, with the values that you care about, bac
[19:43] <bac> gary_poster: i've stepped into the flush at the point where we're concerned but didn't see anything suspicious, just the one newly created bugtask in the _dirty list not the one for generic_task
[19:43] <gary_poster> so in flush I might stash obj_info
[19:44] <bac> ok
[19:44] <gary_poster> bac, right, what we want to know is when the _dirty is mutated, and then by whom
[19:44] <bac> ok
[19:45] <bac> so, just create a local var called __traceback_info__?
[19:45] <gary_poster> bac, then in _flush_one we could scribble expr in there I guess...thinking...
[19:46] <gary_poster> yeah, bac, it will be str'ed
[19:47] <gary_poster> bac, here's my crazy plan, I think:
[19:48] <gary_poster> you do that, then we will have the expression that is sent initially through _flush_one
[19:49] <gary_poster> then in _flush_one, right before _connection.execute, you set a pdb if that expression is there.  That way we don't have to press a mind-numbing collection of "continue"s in pdb
[19:49] <gary_poster> then we look at _dirty.
[19:49] <gary_poster> then after each line we look at dirty, being particularly suspicious of _enable_change_notification and _add_to_alive
[19:50] <gary_poster> until we see when _dirty has been mutated
[19:50] <gary_poster> then we know what to step into
[19:50] <gary_poster> then we re-run, and we actually step into it.  Or similar
[19:50] <gary_poster> bac, end of crazy plan.
[19:50] <bac> ok, so in flush_one i've put in obj_info
[19:50] <bac> into the traceback
[19:50] <gary_poster> also expr, bac
[19:51] <gary_poster> of
[19:51] <gary_poster> oh
[19:51] <gary_poster> that won't work
[19:52] <bac> trying...see what happens
[19:52] <gary_poster> because it is an instance
[19:52] <gary_poster> bac, so maybe "changes"? ok
[19:52] <bac> gary_poster: btw, for those following along at home, the endoscopy didn't work so the vet is doing real surgery.  :(
[19:53] <gary_poster> bac :-(
[19:53] <gary_poster> bac, what is the object?
[19:58] <bac> benji: over in #lp-dev sackett is complaining of pqm woes.  maybe y'all should compare notes.
[19:58] <benji> bac: thanks, I hadn't noticed
[20:07] <gary_poster> bac, about to get on call with flacoste
[20:07] <gary_poster> will check in with you after
[20:47] <gary_poster> bac, I'm off call
[20:49] <bac> gary_poster: got distracted with some CHR stuff from abentley
[20:51] <gary_poster> np bac, I have my own mysteries (Trying to work with the new pgbouncer fixture)