[03:20] <Saj0577> Hey anyone here?
[11:00]  * persia apologises for the delay, but will not be able to start for about 5 more minutes.
[11:00] <dholbach> persia: that's fine - let's wait for the people who show up a bit late :-)
[11:09] <persia> Right.  Pologies again for the (extended) delay.
[11:09] <persia> Welcome to the session on interpreting and using apport retraces.
[11:09] <norsetto> Pologies accepted
[11:10] <persia> Today, we'll be looking to understand, and hopefully discover a solution for Bug #180363 in nicotine
[11:10] <persia> To help follow along, I encourage everyone to collect a text editor, and the hardy nicotine source.
[11:11] <persia> So, apport is the incredibly useful tool that is installed on all our machines to allow us to get better quality bug reports, especially for crashes.
[11:12] <persia> When a program crashes, apport notices, and prepares a crash report.  When so enabled, it prompts the user to submit the crash report to launchpad as a new bug.
[11:13] <persia> There, the apport-retracer system will review the bug, possible determine it to be duplicate, and help with additional information (e.g. retraced stacktrace with symbols for stripped binaries).
[11:14] <persia> Apport bug reports typically explain the type of crash in the title, have some information in the description, along with basic information about the package version, installed Ubuntu version, and user environment.
[11:14] <persia> This is followed by a single comment with a number of attachments, representing further information apport collected from the user system.
[11:15] <persia> For this bug, we have four attachments: Dependencies.txt, ProcMaps.txt, ProcStatus.txt, and Traceback.txt, which seems fairly normal for python crash bugs.
[11:16] <persia> Dependencies.txt shoes the entire tree of recursive dependencies for the crashing package.  This can be useful to determine if someone has upgraded to the latest version, or if the bug appears to also be in a dependency, which version of the code one ought inspect.
[11:17] <persia> ProcMaps.txt shows the local address space for associated objects (libraries, data) used by the application.  I've never found this useful, but it may provide insight in cases of library symbol contention and the like.
[11:18] <persia> ProcStatus.txt provides information about the process itself, including the pid access permissions, memory allocations, and permissions/capabilities.  In general, I usually only check this to make sure the memory allocations (Vm* are not overly large  (100s of GB), which may indicate a runaway memory leak.
[11:19] <persia> Lastly, we have what I consider most interesting, Traceback.txt, which explains the call stack at the time of the crash, and will guide the investigation of the code towards determining the problem exactly.
[11:20] <persia> Are there any questions about the basic format of a python apport-crash bug?
[11:21] <james_w> not from me
[11:23] <norsetto> persia: perhaps few words about ThreadStacktrace.txt and Registers.txt ?
[11:23] <persia> OK.  Now, when triaging an apport crash bug, it's important to make sure it's complete.  If there isn't a trace, or it is unreadable, it may need to be retraced, or if the bug is old, it may be worth asking the submitter to submit an updated crash report.
[11:24] <persia> norsetto: Those happen in other languages, but good point.
[11:24]  * persia grabs a bug with those attachments
[11:25] <siretart> when working on crasher bugs for packages xine, vlc or mplayer, I get a lot of bugs like bug #103756
[11:26] <siretart> how am I supposed to triage them?
[11:26] <persia> OK.  If we look at bug #242674, we can see a few more attachments.
[11:26] <siretart> sorry for disturbing, will ask again later.
[11:26] <persia> siretart: No need.  queued.
[11:27] <persia> CoreDump.gz is the snapshot of the process memory when it crashed.  Most of the time, you won't need to use this.
[11:27] <persia> ProcEnviron.txt provides some of the environment variables that are set
[11:28] <persia> Registers.txt contains the state of the processor registers at the time of the crash (which is typically only meaningful for very low-level crashes).
[11:29] <persia> Stacktrace.txt is similar to Traceback.txt, except that it's a C-style stacktrace, rather than a python-style traceback.
[11:29] <persia> ThreadStacktrace.txt is a stacktrace of all the currently running threads in the application, which can be interesting if there is thread contention.
[11:31] <persia> For triage, I tend to consider that there is often mostly sufficient information provided by apport itself.  If the information is complete (no ?? in the traces, see bug #239842 for an example of this issue), one can likely understand the issue.
[11:32] <persia> When I triage them, I often try to put a short summary of the issue in a comment before setting "Triaged", explaining where or why it crashed, as a result of the code investigation.  Often, the code investigation is sufficient to produce a small patch, which makes a welcome attachment.
[11:33] <persia> Any other questions about the format of a apport bug, or the basic information provided?  (When we're done with this, we'll be investigating the nicotine but more deeply)
[11:35] <persia> OK.  Now, let's take a deeper look at our source code and the Traceback.
[11:36] <persia> The first thing to note is that the version of nicotine has changed between the time the bug was reported, and our current source.
[11:37] <persia> This means that the line numbers will not be reliable.  This is very frequently true when investigating apport crashes.
[11:37] <persia> Often, the best solution is to search the file for the relevant section.
[11:37] <siretart> what was the bug number of the nicotine bug again? (sorry, I arrived late)
[11:37] <persia> So, When reviewing a Traceback, we start at the top, and process downwards.
[11:38] <persia> Each step represents another layer of nested function, and leads to the crash.
[11:39] <persia> The first error is in /usr/bin/nicotine.  Looking at the source, there is a python script at the top level also named nicotine.
[11:40] <persia> Generally, this will be the same file.  In those cases where the file we seek is not immediately available, I usually use a construction like `find . -$(filename) -print` to get the file in the source.
[11:40] <norsetto> siretart: bug 180363
[11:40] <siretart> thanks
[11:40] <persia> norsetto: Thank you.
[11:41] <persia> OK.  The Traceback says that we want line 152, which is supposed to be "result = checkenv()".
[11:41] <persia> In our newer source, line 152 is "import locale", which is clearly not the issue.  If we scroll down from here a bit, we can find "result = checkenv()" at line 191.
[11:43] <persia> Although sometimes source will have multiple lines with roughly the same content, we can be fairly confident this is the correct line because it is still in the root source, rather than being in a specific function (and the Traceback reports "in <module>"
[11:44] <persia> At this point, we'd like to get some idea as to what we expect for result.  Scrolling down a bit first we can see that no result will continue the program, and any result will print the result and exit.
[11:44] <persia> Next, we'll look for the definition of checkenv()
[11:45] <persia> This is up at line 49, and we'll be looking for a gettext call in checkenv()
[11:45] <persia> gettext is exceedingly commonly used, with the shorthand _(), which makes it a little hard to read the message.
[11:47] <persia> We can guess that there are about 40 extra lines added, given the difference between lines 152 and 191 from our previous search, so we want to look for something somewhere between lines 90 and 130 (although this is inexact)
[11:49] <persia> To help us know which is the right line, we can look further down the traceback.  While sometimes the error is in the library (gettext in this example), we ought first search our code.
[11:50] <persia> Reading the Traceback, it calls gettext, which calls dgettext, which calls translation, which calls _init__, which calls _parse, which crashes.
[11:50] <persia> The final error message tells us that the list index is out of range.
[11:51] <persia> This looks like an attempt to translate a plural word that couldn't find the right information in the translation file.
[11:53] <persia> Now, reading 40 lines of code to determine which might have a poor plural translation will be an exercise in frustration.
[11:53] <persia> Luckily, launchpad keeps a copy of every source package ever uploaded, so we can look back to see which line we really wanted.
[11:53] <persia> If we go back to the main bug page, and click "nicotine", we go to the nicotine package page.
[11:54] <persia> On this page, we can scroll down through the versions until we find 1.8.2+dfsg-1ubuntu1.  Clicking on this headline takes us to the summary page for this version, where we can examine it.
[11:55] <persia> In this case, line 90 is close to the string "You do not have Python Vorbis bindings installed.
[11:55] <persia> Others will not be able to see the lengths and the bitrates
[11:55] <persia> of Ogg Vorbis files that you share. You can get the from
[11:55] <persia> http://www.andrewchatham.com/pyogg/.
[11:55] <persia> If you're using Debian, install the python-pyvorbis package."
[11:56] <persia> Returning to our current source, we can see that string starting from line 111
[11:56] <persia> Now, we know the issue is with the hungarian translation, because our bug page says "LANG=hu_HU.UTF-8".
[11:57] <persia> Our source package has a languages/hu directory, containing nicotine.po, which holds the translations.
[11:58] <persia> Here, we can find that there is a translation available, from around line 4469
[11:59] <persia> My Hungarian isn't very strong, so I'm going to trust that the translation is likely correct (although it may not be).
[11:59] <persia> However, I do know that in order to check for the crash, I need to test with a hungarian locale, and try with the python-pyvorbis package uninstalled.
[12:00] <persia> As a result, I can now uninstall python-pyvorbis, and execute `LANG=hu_HU.UTF-8 nicotine` to try to reproduce the crash.
[12:00] <persia> If it breaks, we can describe the problem in detail (or if we know Hungarian, Python, or gettext maybe track it down).
[12:01] <persia> If it works properly, we can document that it worked properly under that set of conditions, ask the submitter to verify it works for them, and suggest installing the python-pyvorbis package as a workaround.
[12:01] <persia> Any questions about the Traceback review, and the process of discovering the problem?
[12:03] <siretart> excellent description of how to track that particular issue down!
[12:03] <persia> siretart: Thank you.
[12:04]  * norsetto think it appropriate that persia should change his nick to holmes
[12:04] <persia> OK.  Anyone willing to volunteer to use this session as a basis, and triage the bug on launchpad?
[12:04] <siretart> or rather dr.watson ;)
[12:05] <persia> Anyone?
[12:06] <persia> It would really benefit from someone adding a comment about the cause of the problem, and testing to see if it can be replicated locally.  If anyone knows Hungarian, Python, or gettext, they may be able to provide more insight.
[12:06] <norsetto> well, it seems that all translations are f*d up, all those msgid are marked as obsolete
[12:07] <persia> norsetto: Thank you :)
[12:08] <james_w> there's also siretart's question
[12:08] <siretart> yes
[12:08] <persia> Which one?  I'm sorry I missed it.
[12:08] <siretart> in media players, I have the problem that there are many failed retraces
[12:08] <siretart> e.g. bug #103756
[12:08] <siretart> I'm unsure how to handle these
[12:09] <siretart> I don't really expect from the casual reporter to actually understand how to do a proper retrace
[12:09] <james_w> in that case the user just used the wrong procedure to file the bug
[12:09] <persia> siretart: For cases like that, it would be good to educate the submitter to use gnome-open (or the equivalent) to open the crash report, which ought trigger apport.
[12:10] <james_w> you should close the bug and ask them to submit it using apport. apport-cli -c "/var/crash/..." or double clicking in nautilus will file it properly so that apport can do it's thing.
[12:10] <siretart> okay, I see
[12:10] <siretart> close with wontfix or invalid?
[12:10] <james_w> Invalid
[12:10] <siretart> ok
[12:10] <persia> Invalid, with a comment requesting them to use apport to open a new bug.
[12:11] <persia> (using the same crash report).
[12:11] <james_w> explain that you are only closing it as the above procedure will open a new bug report, you are not rejecting their bug.
[12:13] <persia> In cases where the retracer cannot find the symbols (?? in the trace), it usually means that no ddeb exists for the version the submitter is using.  In these cases, verify that there are ddebs for all dependencies available for the release the submitter is using, and then ask them to replicate, opening a new bug.
[12:14] <norsetto> what about adding explicit dbg packages? Seems like a good idea to do for packages which have frequent crashes
[12:14] <siretart> I'm providing -dbg packages for ffmpeg and xine packages. does apport get confused by those?
[12:14] <siretart> norsetto: :)
[12:14] <norsetto> siretart: ;-)
[12:16] <persia> I don't actually know if apport gets confused: I'll suggest asking pitti, but I seem to remember something about apport handling those from the changelog at some point: might check the code first.
[12:16] <persia> apport does get confused if the package doesn't call dh_strip somewhere in debian/rules: manual stripping or failure to strip binaries are the most common cause of failed retraces.
[12:17] <persia> While -dbg packages are incredibly helpful in cases where submitters are expected to produce their own stacktraces, they have little benefit for an apport-enabled environment.
[12:19] <persia> Any further questions?  We're a little over time (Sorry that I started late), but I'd like to make sure everyone's questions are answered before we close.
[12:20]  * norsetto hands over a boquet of flowers to persia
[12:21] <pleia2> great session persia :)
[12:21] <james_w> thanks persia
[12:22] <persia> Thanks everyone for attending.  Please feel free to ask me in #ubuntu-motu if there are any future questions about tracking down a specific trace, or in #ubuntu-bugs if you have triage questions.
[12:22] <siretart> indeed, thanks for the session persia.
[12:26] <persia> james_w: pleia2: Do logs of these sessions happen automatically?  Is there a team that edits them for readability and puts them on the wiki?
[12:27] <james_w> persia: I was going to do that.
[12:27] <pleia2> persia: I'll go ahead and post the logs - if you'd like to do a summary that's cool, or I'll see if someone on the team would like to
[12:27] <pleia2> or james_w can :D
[12:27] <pleia2> thanks james_w
[12:27] <persia> james_w: Excellent.  Thank you.  It's nice to have such a proactive Dean :)
[12:28] <james_w> pleia2: perhaps we should have a discussion about MOTU school/classroom interaction.
[12:28]  * pleia2 nods
[12:28] <pleia2> certainly don't want to step on each others toes
[12:28]  * persia hopes the discussion happens here so hecklers can pursue their interests :)
[12:29] <james_w> pleia2: do you have time now?
[12:30] <pleia2> sure
[12:31] <james_w> Unfortunately I'm not on the -classroom list, and I've missed all the discussion around starting it up again. Is there a "mission statement" for the project?
[12:32] <pleia2> essentially it's what is in the Introduction on the wiki: https://wiki.ubuntu.com/Classroom
[12:33] <james_w> so MOTU school seems to fit in with that quite well.
[12:33] <james_w> it's about getting involved, and it's more advanced stuff.
[12:34]  * pleia2 nods
[12:34] <pleia2> reading your wiki page - this is great, you guys keep track of your sessions
[12:34] <pleia2> so we can just link to your resources
[12:34] <james_w> there's not many to track :-)
[12:34] <pleia2> this is a great way to do things
[12:35]  * pleia2 ponders encouraging other teams to do this sort of thing
[12:35] <pleia2> and having classroom be a sort of aggregation
[12:37] <james_w> that might help with finding topics and presenters.
[12:37]  * pleia2 nods
[12:38] <james_w> so are you happy to continue like we did for this session for a while and see how it goes?
[12:38] <pleia2> yep!
[12:38] <james_w> as in I organise sessions and Cc: the list.
[12:38] <pleia2> do you typically send a message to the lists with a link to the logs/notes of the session?
[12:39] <james_w> "typically" :-)
[12:39] <james_w> I think I did once, and then didn't last time. Do you think I should.
[12:39] <james_w> ?
[12:41] <pleia2> yeah, there was one fellow who replied to the lists asking if there would be logs
[12:41] <pleia2> plus it'll help me keep the Classroom wiki updated - like a reminder "this session is done, go link up"
[12:44] <james_w> sure, I'll do that once they are up.
[12:45] <pleia2> thanks :)
[13:07] <james_w> https://wiki.ubuntu.com/MOTU/School/IntrepretingApportRetraces
[13:11] <james_w> spam sent.
[19:30] <maek> so whats it all about ??
[19:35] <RonLu1> jrib: worked only in irc.ubuntu.com.... in irc.freenode.net doesn't work
[19:35] <RonLu1> identify
[19:36] <jrib> RonLu1: ok
[19:36] <jrib> erm
[19:37] <RonLut> jrib: ok... what's now?
[19:37] <maek> so whats it all about ??
[19:37] <jrib> RonLut: you know the path you are copying to correct?
[19:38] <RonLut> right.
[19:38] <jrib> RonLut: and the file is on your desktop
[19:38] <RonLut> jrib: right.
[19:38] <jrib> so you just do: cp '~/Desktop/name of file' /place/you/want/to/copy/to
[19:39] <jrib> then give the error...
[19:39] <RonLut> jrib: without sudo?
[19:39] <jrib> right
[19:42] <RonLut> ok jrib, here is what I wrote in terminal and what I get back:
[19:42] <RonLut> ronlut@ronlut-desktop:~$ cp '/home/ronlut/Desktop/EURO08.exe' '/home/ronlut/.wine/drive_c/Program Files/EA Sports/UEFA EURO 2008'
[19:42] <RonLut> cp: cannot create regular file `/home/ronlut/.wine/drive_c/Program Files/EA Sports/UEFA EURO 2008/EURO08.exe': Permission denied
[19:45] <RonLut> jrib?
[19:45] <jrib> RonLut: ok now give permissions of the location and all of its parents directories
[19:47] <RonLut> http://paste.ubuntu.com/23180/
[19:48] <RonLut> Which parent directories?
[19:48] <jrib> /home/ronlut/.wine/drive_c/Program Files/EA Sports
[19:48] <jrib> /home/ronlut/.wine/drive_c/Program Files/
[19:48] <RonLut> By the way, thank you for helping me... I really appreciate it!
[19:48] <jrib> /home/ronlut/.wine/drive_c/
[19:48] <jrib> etc...
[19:48] <jrib> RonLut: wait, isn't the file there already?
[19:49] <RonLut> it is... I need to replace
[19:49] <jrib> give yourself write permissions on it
[19:49] <RonLut> jrib:  :( how to do that?
[19:50] <jrib> chmod +w /path/to/file     or just right click -> properties
[19:54] <RonLut> thank you
[19:54] <RonLut> It works...