| ecanto | date -u | 01:22 |
|---|---|---|
| cjohnston | ecanto: you need to do that in the command line | 01:25 |
| cjohnston | @now | 01:25 |
| ubottu | Current time in Etc/UTC: January 28 2010, 01:25:16 | 01:25 |
| ecanto | cjohnston thanks. (: | 01:25 |
| === ecanto is now known as dmesg | ||
| === dmesg is now known as ecanto | ||
| === tobias is now known as Guest12445 | ||
| * jcastro taps on the mic | 15:47 | |
| * dholbach looks for the technicians to fix the feedback loop | 15:49 | |
| === dholbach changed the topic of #ubuntu-classroom to: Ubuntu Classroom || Support in #ubuntu || https://wiki.ubuntu.com/Classroom || https://wiki.ubuntu.com/Packaging/Training || Upcoming: Mon 25 Jan - Fri 29 Jan 2010: https://wiki.ubuntu.com/UbuntuDeveloperWeek || Run 'date -u' in a terminal to find out the UTC time | Ask questions in #ubuntu-classroom-chat | ||
| === tobias is now known as Guest11086 | ||
| dholbach | WELCOME EVERYBODY TO DAY 4 OF UBUNTU DEVELOPER WEEK! | 15:59 |
| dholbach | If you don't know why you're here and what's going on, check out https://wiki.ubuntu.com/UbuntuDeveloperWeek :-) | 15:59 |
| dholbach | Questions all go to #ubuntu-classroom-chat and please prefix them with QUESTION so they stick out! | 16:00 |
| dholbach | if you're uncomfortable asking questions in English, please head into one of the channels listed at the end of https://wiki.ubuntu.com/UbuntuDeveloperWeek | 16:00 |
| dholbach | there will be people who will translate questions and answers for you if necessary (consider helping out yourself, please :-)) | 16:00 |
| dholbach | OK, enough organisational mumbo-jumbo! | 16:01 |
| dholbach | I'm here with the amazing Jorge O. Castro, jcastro! | 16:01 |
| jcastro | woo hoo! | 16:01 |
| jcastro | we have been preparing real hard for this session, hope you like it! | 16:01 |
| === godzilla is now known as Guest53616 | ||
| dholbach | And together we'll talk about Adopting an Upstream! Why it's fun, necessary and makes the world a better place. | 16:02 |
| dholbach | jcastro: What's an Upstream? | 16:02 |
| jcastro | that's a good question | 16:02 |
| jcastro | first, when you get an ubuntu CD it comes with a bunch of great software | 16:02 |
| jcastro | things like Firefox, Openoffice, GNOME, KDE, etc. | 16:02 |
| jcastro | what Ubuntu does as a distribution is take all those things, assemble them into one operating system, and then ship | 16:03 |
| jcastro | so, these projects, we call them upstream, like in a river | 16:03 |
| jcastro | and distributions like us are downstram | 16:03 |
| jcastro | and derived distributions like Mint would be downstream from us | 16:03 |
| dholbach | so that's like all the software authors of those projects? | 16:03 |
| jcastro | working with upstreams and downstreams is a critical part of ubuntu | 16:03 |
| jcastro | right! | 16:03 |
| dholbach | (I hope I encourage YOU ALL to ask questions too, I asked the first ones myself... :-)) | 16:03 |
| jcastro | so, for example, it's critical for us to be able to work with upstreams like Firefox to ensure that we're shipping something users will love | 16:04 |
| jcastro | so for example this week in #ubuntu-mozillateam you'll see our Ubuntu teams working with people from Mozilla | 16:04 |
| dholbach | So there's millions of Ubuntu users and there's thousands of Upstream projects... looks like being there for both huge groups is quite an undertaking | 16:04 |
| jcastro | so there's a bunch of cross boundary work going on there that some people don't see | 16:04 |
| jcastro | right, especially when you're trying to fix a problem! But we'll show you how to do that when we get there | 16:05 |
| jcastro | you'll see that sometimes putting a bug in the right place that we can be a service to upstreams when it comes to testing. | 16:05 |
| jcastro | and when we fix problems it's of course our duty to make sure that those fixes get back upstream so that they can improve the overall quality of their project | 16:06 |
| jcastro | and of course, the other half is ensuring that when upstream fixes bugs that we get those out to users, because at the end of the day that's what it's all about | 16:06 |
| jcastro | so | 16:06 |
| jcastro | I have been working on some wiki pages to help people get started | 16:06 |
| jcastro | https://wiki.ubuntu.com/Upstream | 16:07 |
| jcastro | It's unreasonable to expect every upstream author to know in detail how ubuntu works | 16:07 |
| jcastro | some are content just making their software | 16:07 |
| dholbach | and the other way around :) | 16:07 |
| jcastro | so I've put together a little cheat sheet here for them (you can help improve this too if you find mistakes) | 16:08 |
| jcastro | so that they can have one place to find important things, like when we freeze, what our security policy is, etc. | 16:08 |
| jcastro | and also give them information on how things are going on the ubuntu platform | 16:08 |
| jcastro | so if you wanted to find out about notify osd or messaging indicators, I've listed those there | 16:08 |
| jcastro | now | 16:09 |
| jcastro | one thing we've all been working on is a little program to give upstreams an "in" in the distro | 16:09 |
| jcastro | this would be a person that does know how Ubuntu works | 16:09 |
| jcastro | and is passionate about that specific piece of software | 16:09 |
| jcastro | and wants to work on it in Ubuntu but also be there for the upstreams to answer their questions, be an information source, and be there when things go wrong | 16:10 |
| jcastro | we call this Adopt a Package | 16:10 |
| jcastro | dholbach: want to go into that for a little bit? | 16:10 |
| dholbach | sure | 16:10 |
| jcastro | https://wiki.ubuntu.com/BugSquad/AdoptPackage | 16:11 |
| dholbach | and https://wiki.ubuntu.com/Upstream/Adopt | 16:11 |
| dholbach | so to Adopt an Upstream can mean a couple of things | 16:11 |
| dholbach | the most important one being that you act as a tie betwen Ubuntu and that upstream project | 16:11 |
| dholbach | you can answer questions for people on that project, you know what's going on there, you know what's going on in Ubuntu | 16:12 |
| dholbach | an easy example for "knowing what's going on" is: Upstream wants to ship a new feature release 2 weeks before the new Ubuntu release comes out | 16:12 |
| dholbach | in that case you can talk to them and make it clear that it's hard to get that new feature release into Ubuntu because Ubuntu is already almost frozen | 16:13 |
| dholbach | you can help with that kind of misinformation and help with the coordination early and explain what to do | 16:13 |
| dholbach | jcastro and I will talk about the release cycle in more detail later on | 16:13 |
| dholbach | another thing you can work on is bug reports | 16:14 |
| dholbach | as we said before Ubuntu exposes great software to millions of users | 16:14 |
| dholbach | of course these users occasionally run into problems | 16:14 |
| jcastro | and they might not know what an upstream is! | 16:14 |
| dholbach | exactly | 16:14 |
| dholbach | so it's our job to triage those bugs, debug them, get as much information as possible and forward them to the upstream maintainers | 16:14 |
| dholbach | because they are in the best position to work on the bug effectively | 16:15 |
| dholbach | if we can fix it that's great, then we'll forward the patch to upstream | 16:15 |
| jcastro | oh, so we can be good filters you mean? | 16:15 |
| dholbach | exactly | 16:15 |
| jcastro | we can weed out all the junk bugs so upstreams can concentrate on the ones with the best debugging info, descriptions, etc. | 16:15 |
| dholbach | I've heard we have great tools for finding duplicates and dealing with crashes? | 16:15 |
| dholbach | jcastro: maybe we should explain first what "to triage" means | 16:16 |
| jcastro | yes. | 16:16 |
| dholbach | you wanna do it? :) | 16:16 |
| jcastro | https://wiki.ubuntu.com/Bugs/HowToTriage/ | 16:16 |
| jcastro | sure! | 16:16 |
| jcastro | so when users report bugs | 16:16 |
| jcastro | they go into this big pile | 16:17 |
| jcastro | to triage a bug means to subdivide that pile from one huge unmanageable mess to smaller piles | 16:17 |
| jcastro | and as the bug continues on in it's lifecycle it will end up in the right pile | 16:17 |
| jcastro | so, we might start with a big pile | 16:17 |
| jcastro | but then we have smaller piles, kernel, desktop, printing, etc. | 16:17 |
| jcastro | and then those get smaller | 16:18 |
| === tobias is now known as Guest96094 | ||
| jcastro | until you have a package, like say, nautilus. | 16:18 |
| jcastro | bug triagers are people who sort the bugs in the right pile | 16:18 |
| dholbach | then we might be able to get the bug confirmed, make sure we know in which release the bug occurs | 16:18 |
| jcastro | the great thing about this is that you can do this without being too technical | 16:18 |
| dholbach | then find the right steps to reproduce it | 16:18 |
| jcastro | these days we have tools that do a good job of guessing what pile something should go in | 16:18 |
| jcastro | but we also need to try to reproduce the bug so we can confirm it | 16:19 |
| jcastro | this involves following what the reporter did | 16:19 |
| jcastro | so if the report is "clicking on this button breaks printing" | 16:19 |
| dholbach | https://help.ubuntu.com/community/ReportingBugs has a bit more detail about how this "guessing" works | 16:19 |
| jcastro | then usually you try it, and see if you see the problem too | 16:19 |
| jcastro | also, remember that we have millions of users | 16:19 |
| dholbach | the better we isolate the issue, the less work the upstreams have, they can concentrate on fixing the bug :-) | 16:20 |
| jcastro | I find that most times bugs sitting in the big pile have already been reported, so you can do a bunch of good just marking things as duplicates (if they are indeed duplicates) | 16:20 |
| dholbach | right | 16:20 |
| dholbach | also there's bugs with crash reportsf | 16:20 |
| === tobias is now known as Guest76403 | ||
| dholbach | Ubuntu has a fantastic machinery that gets as much debug information about those crashes as possible | 16:21 |
| dholbach | automatically | 16:21 |
| jcastro | https://wiki.ubuntu.com/Apport <--- here are pictures of what it looks like | 16:21 |
| dholbach | so what was a painstaking job before (https://wiki.ubuntu.com/Backtrace) is a lot easier now | 16:21 |
| dholbach | the end result is that we can say in function do_something() in line 45 of file bla/something.c the whole thing crashed | 16:22 |
| dholbach | plus other things that developers might want to know :) | 16:22 |
| dholbach | go and see Emmet Hikory's session on Friday about "Interpreting stacktraces" if you want to know more :) | 16:22 |
| jcastro | are these important to send to upstreams? | 16:22 |
| jcastro | they look important! | 16:23 |
| dholbach | holy cow yes! | 16:23 |
| dholbach | with all this information it should be a bit easier for upstream developers to find the cause for the crash and fix it | 16:23 |
| dholbach | a crash in most cases means: the program is gone, it stopped working, unsaved work, etc you know what it's like | 16:23 |
| dholbach | sometimes these crashes happen in special circumstances that didn't happen for the upstream developers when they tested it | 16:24 |
| dholbach | in almost every discussion I was part of upstreams were happy about that kind of conversations and grateful for detailed bug reports | 16:24 |
| dholbach | if you have questions about how to get started with bugs and everything check out https://wiki.ubuntu.com/Upstream/Adopt and talk to the fine people in #ubuntu-bugs | 16:24 |
| jcastro | ok, so apport finds this and reports this to launchpad, what's the next step? | 16:25 |
| dholbach | the Launchpad machinery will do its magic and add more debug information to the bug report | 16:25 |
| dholbach | the next step would be trying to confirm the issue | 16:25 |
| dholbach | if you can confirm it and maybe add some more details to what happened, you can have a look and see if it was already reports in the upstream bug tracker | 16:26 |
| dholbach | if it was, see if you can add more information to the bug | 16:26 |
| dholbach | if it wasn't report it | 16:26 |
| dholbach | sometimes it's useful to have a look and see if upstreams have some kind of "debugging information" documented somewhere | 16:26 |
| dholbach | for some of them it will be something like "start the program with --debug option" or something | 16:27 |
| dholbach | a lot of that kind of information is listed at https://wiki.ubuntu.com/DebuggingProcedures and you should add info for your upstream project if you have any :) | 16:27 |
| dholbach | also Launchpad has a great feature: you can link the upstream bug to the Ubuntu bug, right? | 16:28 |
| jcastro | yeah | 16:28 |
| jcastro | let me show you about that | 16:28 |
| === godzilla is now known as Guest30844 | ||
| jcastro | actually, give me a minute to find a bug | 16:29 |
| jcastro | dholbach: do a dance! | 16:29 |
| * dholbach puts on some music and does the funky Upstream-Bug-Linking dance | 16:29 | |
| jcastro | https://wiki.ubuntu.com/Bugs/Upstream | 16:29 |
| jcastro | ok so here is some documentation, depending on the bug | 16:30 |
| jcastro | however, in general, you can just click "Also affects project" in launchpad, and paste in the bug url from the other bug tracker | 16:30 |
| jcastro | https://wiki.ubuntu.com/Bugs/Watches | 16:30 |
| jcastro | they looks like this | 16:30 |
| jcastro | so you see there, on the top example, there's the bug in ubuntu, and the bug in the upstream project's bug tracker | 16:30 |
| jcastro | in this case linux kernel bug 10911 | 16:30 |
| jcastro | these are useful because with certain bug trackers | 16:31 |
| jcastro | like mozilla's bugzilla, we can do neat things like syncing comments between the 2 bug trackers | 16:31 |
| jcastro | https://bugs.edge.launchpad.net/ubuntu/+source/notify-osd/+bug/501393 | 16:32 |
| jcastro | here is a good example of both ubuntu and mozilla developers working on a bug for example | 16:32 |
| jcastro | however, just linking the bug can help | 16:32 |
| jcastro | because it gives everyone a place to gather discussion around a bug | 16:32 |
| jcastro | and especially for new users that might not know how things work | 16:33 |
| jcastro | what other information can we help upstreams with dholbach? | 16:33 |
| jcastro | tell me about this schedule | 16:33 |
| dholbach | another fine thing you can do is: plan a Bug Day with your Upstream: invite them to https://wiki.ubuntu.com/UbuntuBugDay - triage bugs together, give people some coaching with that particular software - once you're good friends with your upstream, sky's the limit | 16:33 |
| dholbach | alright... release schedule! | 16:34 |
| jcastro | upstreams ask me all the time about the schedule | 16:34 |
| jcastro | but sometimes I don't know what means what! | 16:34 |
| dholbach | have a look at https://wiki.ubuntu.com/ReleaseSchedule | 16:34 |
| dholbach | it shows our current release cycle | 16:34 |
| dholbach | you can see how it goes from green in the beginning of the 6 months cycle to red at the end | 16:35 |
| dholbach | green doesn't mean "it's all good, everything works" :-) | 16:35 |
| dholbach | it rather means "everything's allowed from a developer point of view" | 16:35 |
| dholbach | so we start with setting up the build tools in the beginning, plan features at UDS and merge with Debian (more on merges, syncs and Debian in a bit :-)) | 16:36 |
| dholbach | at some stage we hit feature freeze, that's where we plan to have most of the feature in the distro | 16:37 |
| jcastro | Xazax | why is the release cycle for upstream projects important? | 16:37 |
| dholbach | doesn't mean they have to be all perfect | 16:37 |
| dholbach | Xazax: will get to that in a sec | 16:37 |
| dholbach | but it means that they have to be there and be test-able :) | 16:37 |
| dholbach | so we move from feature development to fixing | 16:38 |
| dholbach | and stabilisation | 16:38 |
| dholbach | the later it gets in the release cycle the more conservative we get | 16:38 |
| dholbach | to upload a small patch that fixes a crash 3 weeks before release is probably fine | 16:38 |
| dholbach | but to upload a GIANT new release with n+1 new features will probably get you in trouble with the release managers | 16:38 |
| dholbach | Jorge and I worked on a cheatsheet that makes it easier to understand what is allowed at which stage of the release cycle to ease the planning and coordination somewhat: http://people.canonical.com/~dholbach/cheatsheet.pdf | 16:39 |
| dholbach | that's linked from https://wiki.ubuntu.com/Upstream/Adopt too | 16:39 |
| dholbach | I hope that answers Xazax's question somewhat. | 16:39 |
| jcastro | i would like to chime in on his question | 16:40 |
| dholbach | sure | 16:40 |
| jcastro | with things like a Long Term Support release it's critical for us as a downstream distro to communicate our release schedule | 16:40 |
| jcastro | we sync our schedule with GNOME | 16:40 |
| jcastro | so, we're kind of lined up | 16:40 |
| jcastro | some upstreams want to know when they should release to have their software in ubuntu | 16:41 |
| jcastro | so they can make a release, or decided to skip an ubuntu release, etc. | 16:41 |
| jcastro | so, when we're at UDS | 16:41 |
| jcastro | and we're talking to upstreams we make decisions on things dependent on release cycles | 16:41 |
| jcastro | so it might be "well, that feature won't be ready in time, so we'll go with version 1.x instead" or something | 16:42 |
| dholbach | that sounds very reasonable | 16:42 |
| dholbach | whatever we can't fix during the release time, we have to fix after the release... jcastro: do you know how that works? | 16:42 |
| jcastro | yes | 16:42 |
| jcastro | it's a big source of confusion | 16:43 |
| jcastro | so let's talk about Stable Release Updates | 16:43 |
| dholbach | I think it all started with something like this: http://blog.somekool.net/images/img_4232.jpg | 16:43 |
| jcastro | https://wiki.ubuntu.com/StableReleaseUpdates | 16:43 |
| jcastro | doh! | 16:43 |
| dholbach | shall I tell the story quickly before you move on? | 16:43 |
| jcastro | yes! | 16:43 |
| dholbach | ok | 16:43 |
| dholbach | so in the very early days of Ubuntu we uploaded a couple of fixes (they all were upstream already) to the xserver to a stable release | 16:44 |
| dholbach | that means that users who had installed the release (and not the development version) got those fixes | 16:44 |
| dholbach | the developer had tested them all, they were upstream fixes and everything seemed good | 16:44 |
| dholbach | the problem was that a certain percentage of our users (x% of some millions) suddenly had a broken Xserver | 16:45 |
| dholbach | which resulted in people like my dad having to wrestle with: http://blog.somekool.net/images/img_4232.jpg | 16:45 |
| dholbach | so we realised that we need to be more conservative with fixes like that | 16:45 |
| dholbach | that's where SRUs originate from | 16:46 |
| dholbach | jcastro: so how does this all work now? | 16:46 |
| jcastro | https://wiki.ubuntu.com/StableReleaseUpdates | 16:46 |
| jcastro | we have these | 16:46 |
| jcastro | don't worry, it only looks scary! | 16:47 |
| jcastro | the key point to remember here when talking to upstreams | 16:47 |
| jcastro | is that really bad bugs should be communicated to distros | 16:47 |
| jcastro | you see this all the time on like planet gnome | 16:47 |
| jcastro | "wow, we found a bad bug in gstreamer, distros should ship this fix: $url" or something | 16:48 |
| jcastro | as you hang out and talk with upstreams | 16:48 |
| jcastro | at some point they will have a critical bug that will need to be fixed | 16:48 |
| jcastro | so usually you can take ownership of that, talk to the ubuntu maintainer, and communicate the needs that way | 16:48 |
| jcastro | sometimes upstreams fix small bugs | 16:48 |
| kfogel | dhasenan: thanks | 16:48 |
| jcastro | and of course those are good | 16:49 |
| kfogel | whhooops | 16:49 |
| kfogel | dholbach: thanks | 16:49 |
| * kfogel watches out for nick completion | 16:49 | |
| jcastro | but it can be frustrating to an upstream when the fix doesn't get into an SRU | 16:49 |
| jcastro | it's just one of those things where the fix needs to be weighed with the risk | 16:49 |
| dholbach | https://wiki.ubuntu.com/StableReleaseUpdates is pretty good at explaining what kind of fixes we want and which make sense | 16:50 |
| jcastro | In general I think we do a good job here, but if an upstream still wants new crack in the distro, we have things like PPAs and backports | 16:50 |
| jcastro | that's also an area where you can contribute | 16:50 |
| jcastro | many ubuntu contributors set up and run PPAs as a service to upstreams for apps people love | 16:50 |
| dholbach | http://people.canonical.com/~dholbach/cheatsheet.pdf should give you an overview over what to do in which case | 16:50 |
| jcastro | that is highly encouraged, and a good way to learn packaging if you want to start the road to becoming an ubuntu developer | 16:51 |
| jcastro | we only have 5 minutes left, so let's talk about debian for a bit | 16:51 |
| dholbach | one of our most important upstreams is Debian | 16:52 |
| dholbach | and we inherit a lot of packages and lots of great work from Debian developers | 16:52 |
| dholbach | usually it works like this: if a package is unmodified in Ubuntu and we're before DebianImportFreeze (Check the release schedule again), it gets "synced" automatically | 16:52 |
| dholbach | which means that the package in Ubuntu will get overwritten with the source that is in Debian right now (and rebuilt) | 16:53 |
| === zen is now known as Guest90818 | ||
| dholbach | if it's modified in Ubuntu, we need to figure out if we can drop out changes (then we'll ask for a sync) | 16:53 |
| dholbach | and if we can't drop our changes, we need to merge the changes semi-manually | 16:53 |
| dholbach | https://wiki.ubuntu.com/UbuntuDevelopment/Merging explains how the merging works and which tools we use | 16:54 |
| dholbach | common scenarios for differences between Ubuntu and Debian are: | 16:54 |
| dholbach | - we're in different parts of our release cycle and we decide to ship a newer version | 16:54 |
| dholbach | - new improvements we want to try out in Ubuntu (launchpad integration of gnome apps, new indicators, new notifications, etc.) | 16:55 |
| dholbach | and lots of other small things because of different things we support and not support | 16:55 |
| dholbach | but generally the idea is: the more we can have in sync with Debian, the better | 16:55 |
| dholbach | it means less merging, less maintenance hassle and a common code base to look at | 16:55 |
| dholbach | so whatever is suitable in terms of patches to forward to Debian, we do forward | 16:56 |
| dholbach | https://wiki.ubuntu.com/Debian/Bugs has more details on that | 16:56 |
| jcastro | whew | 16:56 |
| jcastro | that was alot of info! | 16:56 |
| jcastro | and only one question, so either we were really great or really bad! | 16:56 |
| jcastro | \o/ | 16:57 |
| dholbach | that's the great thing about working on a certain piece of software is you get to know: lots of users (and make them happy with fixes), the upstream maintainers, lots of great Ubuntu developers that work with you and maintainers of others distros too | 16:57 |
| dholbach | you not only make the world a better place by working on making software better for everybody, but you also get to know a lot of great people AND learn something new :) | 16:57 |
| dholbach | one page you should definitely bookmark is: https://wiki.ubuntu.com/Upstream/Adopt :) | 16:58 |
| dholbach | any final questions? | 16:58 |
| dholbach | jcastro: any final words? :) | 16:58 |
| jcastro | nope, thanks for coming everyone! | 16:58 |
| dholbach | thanks everybody - you ROCK! | 16:59 |
| * ogasawara fades in | 17:00 | |
| dholbach | next up is Leann Ogasawara who will talk to us about "Kernel patches"! | 17:00 |
| ogasawara | Hi Everyone! Welcome to the Kernel Patches Session. | 17:00 |
| ogasawara | My name is Leann Ogasawara and I'm a member of the Ubuntu Kernel Team. | 17:00 |
| ogasawara | If you have any questions, please post them to #ubuntu-classroom-chat prefixed with QUESTION: | 17:00 |
| ogasawara | One of the biggest issues we face as a team is the large volume of bug reports we receive against the kernel. | 17:01 |
| ogasawara | One possible way to lend a helping hand is submitting patches to fix these bugs. | 17:01 |
| ogasawara | Submitting a kernel patch might seem like a daunting task, but hopefully after today's session it won't seem so intimidating. | 17:01 |
| ogasawara | There are five main items I'd like to cover in today's talk: | 17:01 |
| ogasawara | 1) Choosing bugs to patch | 17:01 |
| ogasawara | 2) Patch submission process | 17:01 |
| ogasawara | 3) Walk through an actual example | 17:01 |
| ogasawara | 4) Reasons for not applying a patch | 17:01 |
| ogasawara | 5) Getting Help | 17:01 |
| ogasawara | I should mention that the Ubuntu kernel team's ideal scenario is to have little to no divergence from the upstream linux kernel. | 17:02 |
| ogasawara | If a kernel patch will benefit Ubuntu users, it'll likely benefit all. | 17:02 |
| ogasawara | This is why it is important to make sure patches get submitted upstream. | 17:02 |
| ogasawara | Also, keeping the delta we have between our kernel and the upstream kernel as small as we can makes it much much easier to maintain the kernel from our end. | 17:02 |
| ogasawara | As such, the Ubuntu kernel team's preferred policy is for all patches to be submitted and accepted into the upstream kernel before pulling them into the Ubuntu kernel. | 17:03 |
| ogasawara | So how does this happen? | 17:03 |
| ogasawara | Let's begin with our first talking point. | 17:03 |
| ogasawara | == Choosing bugs to patch == | 17:03 |
| ogasawara | Ideally we want to isolate good bug candidates that could easily be resolved if a patch were applied. | 17:03 |
| ogasawara | A good approach might be to tackle the low hanging fruit first, ie bugs with patches already attached. | 17:03 |
| ogasawara | Launchpad has the ability to search for kernel bugs with patches attached. | 17:04 |
| ogasawara | https://bugs.edge.launchpad.net/ubuntu/+source/linux/+bugs?field.has_patch=on | 17:04 |
| ogasawara | As I had mentioned previously, due to the large volume of bugs, there are times when bugs are filed with patches attached to fix an issue but they unfortunately get lost in the noise. | 17:04 |
| ogasawara | If we have additional volunteers trying to keep and eye on these types of quick fixes, it would be much appreciated. | 17:04 |
| ogasawara | Even just getting these types of bugs into a triaged state would be a great help. | 17:05 |
| ogasawara | If anyone is interested in getting involved, ping me after the session and I'be be happy to help get you started. | 17:05 |
| ogasawara | Many times we see a reporter will open a bug and mention the issue can be fixed with a patch that exists upstream alread. | 17:05 |
| ogasawara | Since the patch is already upstream, it's a simple process for a kernel developer to cherry pick the fix back into the Ubuntu kernel. | 17:05 |
| ogasawara | That is one example of how patches get pulled from the upstream kernel into the Ubuntu kernel. | 17:06 |
| ogasawara | However, not all bugs with patches can be resolved so easily. | 17:06 |
| ogasawara | Sometimes a bug will have a patch attached but the patch itself has not been submitted/applied upstream nor to the Ubuntu kernel. | 17:06 |
| ogasawara | These are the types of patches we want to focus on getting merged upstream and back into the Ubuntu kernel, assuming they are a valid fix. | 17:06 |
| ogasawara | It's helpful to inquire with the patch author about the nature of the patch. | 17:07 |
| ogasawara | Is the patch just a workaround for the issue but does not address the root cause? | 17:07 |
| ogasawara | Was the patch only meant for debugging purposes to isolate the issue? | 17:07 |
| ogasawara | We have to keep in mind that not all patches will be the proper fix. | 17:07 |
| ogasawara | If the patch is a band aid and not a proper fix, it shouldn't be applied. | 17:07 |
| ogasawara | On the other hand, if the patch author just needs assistance with submitting the patch, hopefully the information I talk about next will help. | 17:08 |
| ogasawara | If anyone ever has a situation where they have a question regarding a patch they've seen or personally written, please don't hesitate to contact the kernel team. | 17:08 |
| ogasawara | We're usually in the #ubuntu-kernel IRC channel on FreeNode and obviously also closely watch the Ubuntu kernel team mailing list. | 17:08 |
| ogasawara | I'd like to take a second here to pause and field any questions before we move onto discussing the patch submission process. | 17:08 |
| ogasawara | If there are no questions, we'll just move right along . . . | 17:09 |
| ogasawara | Navaneeth> QUESTION: Patches applied on ubuntu kernel are sent back to upstream? | 17:09 |
| ogasawara | Navaneeth: indeed we actually strive to push the patch upstream first, before applying to Ubuntu | 17:09 |
| ogasawara | Navaneeth: that might not always be the case so after each release, we review our delta from upstream and work to get the non-ubuntu specific patches pushed up | 17:10 |
| ogasawara | == Patch Submission Process == | 17:11 |
| ogasawara | First we need to confirm the patch does indeed resolve the issue. | 17:11 |
| ogasawara | If you yourself can confirm the issue is resolved with the patch, that's perfect. | 17:11 |
| ogasawara | Otherwise, we need to work with whomever has reported the issue and have them confirm the patch fixes the problem. | 17:11 |
| ogasawara | Once we have confirmation the fix is valid, we're ready to send the patch upstream. | 17:12 |
| ogasawara | If the bug the patch fixes happens to be reported in Launchpad, for completeness sake, the patch can be uploaded as an attachment to the bug report. | 17:12 |
| ogasawara | When uploading the patch, be sure to select the "This attachment is a patch" option. | 17:12 |
| ogasawara | Now, lets go through preparing the patch to submit upstream. | 17:12 |
| ogasawara | Upstream has this process clearly documented at the following link. | 17:12 |
| ogasawara | http://git.kernel.org/?p=linux/kernel/git/torvalds/linux-2.6.git;a=blob_plain;f=Documentation/SubmittingPatches | 17:12 |
| ogasawara | I'll highlight some of the more important bits below . . . | 17:13 |
| ogasawara | 1. Know who/where to submit the patch. | 17:13 |
| ogasawara | This can be done by examining the MAINTAINERS file in the linux kernel tree | 17:13 |
| ogasawara | http://git.kernel.org/?p=linux/kernel/git/torvalds/linux-2.6.git;a=blob;f=MAINTAINERS | 17:13 |
| ogasawara | 2. Be sure to create the patch against the upstream kernel source, not the Ubuntu kernel source. | 17:13 |
| ogasawara | http://git.kernel.org/?p=linux/kernel/git/torvalds/linux-2.6.git | 17:13 |
| ogasawara | Note that some subsystems maintain their own git trees. | 17:14 |
| ogasawara | If this is the case, create the patch against the subsystem's tree. | 17:14 |
| ogasawara | The location of the subsystem's tree can also be found in the MAINTAINERS file | 17:14 |
| ogasawara | 3. Clearly describe what the patch is fixing. | 17:14 |
| ogasawara | Also be sure to break large patches up into small logical pieces. | 17:14 |
| ogasawara | A good rule of thumb is each patch should fix one specific issue. | 17:15 |
| ogasawara | 4. Be sure to note the version of the kernel the patch applies to. | 17:15 |
| ogasawara | When submitting a patch upstream, the patch should apply cleanly to the latest version of the upstream kernel available. | 17:15 |
| ogasawara | 5. Make sure the patch adheres to the kernel's coding style. | 17:15 |
| ogasawara | http://git.kernel.org/?p=linux/kernel/git/torvalds/linux-2.6.git;a=blob_plain;f=Documentation/CodingStyle | 17:15 |
| ogasawara | If you run scripts/checkpatch.pl, this should quickly detect any coding style violations. | 17:15 |
| ogasawara | 6. Make sure to maintain full provenance of the patch by signing off on the patch. | 17:16 |
| ogasawara | Signing off on the patch allows others to see the full history of who's written/reviewed the patch. | 17:16 |
| ogasawara | Signing off is done by simply adding the "Signed-off-by:" line to your patch. | 17:16 |
| ogasawara | 7. Send the patch as plain text in the email. | 17:16 |
| ogasawara | Do not attach the patch. | 17:17 |
| ogasawara | It's quicker to review a patch that's already present in the body of the email rather than having to download and open an attachment. | 17:17 |
| ogasawara | Your email client should provide a way to inline the text directly from the patch file into the body of the email. | 17:17 |
| ogasawara | Additionally, plain text allows others to respond and annotate their feedback directly in the context of the patch. | 17:17 |
| ogasawara | Be careful if you just cut and paste as this might have negative affects like switching tabs to spaces and therefore violating the kernel coding style. | 17:17 |
| ogasawara | Also, be careful that your email client doesn't mangle the patch by line wrapping the patch. | 17:18 |
| ogasawara | 8. Prefix the subject of the email with "[PATCH]" | 17:18 |
| ogasawara | This allows people to immediately know this is a patch being submitted. | 17:18 |
| ogasawara | 9. Don't give up. Be persistent :) | 17:18 |
| ogasawara | If the patch is not immediately applied there are usually reasons. | 17:18 |
| ogasawara | Take suggestions constructively and fix up the patch if necessary. | 17:19 |
| ogasawara | Patches might go through a few rounds of reviews and iterations before being accepted. | 17:19 |
| ogasawara | If there is no response or feedback after submitting a patch, don't be afraid to submit it again. | 17:19 |
| ogasawara | Eventually the patch should hopefully receive the required Ack's from upstream and get applied. | 17:19 |
| ogasawara | Once the patch is applied upstream it will automatically make it's way into the Ubuntu kernel upon the next rebase. | 17:20 |
| ogasawara | If the patch should be considered for a Stable Release Update to a previous kernel release, the patch must come via the upstream stable tree. | 17:20 |
| ogasawara | In order for a patch to be accepted into the upstream stable tree, the patch must meet certain criteria. | 17:20 |
| ogasawara | The patch should be relatively small in size (less than 100 lines of code). | 17:20 |
| ogasawara | The patch must fix a real issue, no hypothetical use case fixes. | 17:20 |
| ogasawara | The patch must already exist in Linus' tree upstream. | 17:21 |
| ogasawara | For a full list of criteria, please refer to the following: | 17:21 |
| ogasawara | http://git.kernel.org/?p=linux/kernel/git/torvalds/linux-2.6.git;a=blob;f=Documentation/stable_kernel_rules.txt | 17:21 |
| ogasawara | Submitting a patch to the upstream stable tree is as easy as sending the patch in an email to stable at kernel.org | 17:21 |
| ogasawara | Alternatively, when originally submitting a patch upstream, one can add a "CC:" line under their "Signed-off-by:" line. | 17:21 |
| ogasawara | For a more detailed explanation of sending a patch to upstream stable, ssee the stable_kernel_rules.txt file referred to above. | 17:22 |
| ogasawara | Once a patch has been submitted upstream, I usually like to add a reference to the upstream thread from the bug report. | 17:22 |
| ogasawara | This allows anyone wanting to track the current status of the patch to read and follow along with the thread upstream. | 17:22 |
| ogasawara | Again, if any patch author needs helps submitting a patch upstream, don't hesitate to ask those on the kernel team for help. | 17:22 |
| ogasawara | I'm going to pause once more to field any questions. | 17:23 |
| ogasawara | mirsal> QUESTION: What about using git to produce patches ? | 17:23 |
| ogasawara | mirsal: of course, that's an excellent way to generate your patch | 17:23 |
| ogasawara | mirsal: for those not familiar with git, the command it 'git format-patch' | 17:23 |
| ogasawara | Navaneeth> QUESTION: Will this session covers debugging kernel source code and fixing a bug? If not, is there any helpful materials available online? | 17:24 |
| ogasawara | Navaneeth: I'm going to walk through and example of a bug that was fixed and the patch pushed upstream | 17:24 |
| ogasawara | Navaneeth: it likely won't go as in depth into the debugging process as you'd like, I'll point you to helpful materials after the session | 17:25 |
| ogasawara | vishalrao_vbox> QUESTION: Do you @ubuntu have a hardware testing lab of sorts? For example can you test SSD support as you can see LP #502219 ? | 17:25 |
| ogasawara | vishalrao_vbox: we do have access to hardware we maintain in a lab | 17:25 |
| ogasawara | vishalrao_vbox: I do not know off the top of my head about the hw in the bug you note, but I'll take a look after the session | 17:25 |
| ogasawara | any other questions? | 17:26 |
| ogasawara | ok, we'll move along | 17:26 |
| ogasawara | == Example of Submitting a Patch Upstream == | 17:27 |
| ogasawara | I thought it would be a good idea to walk through an actual bug whose patch was sent upstream. | 17:27 |
| ogasawara | Lets take a look at https://bugs.launchpad.net/ubuntu/+source/linux/+bug/366682 | 17:27 |
| ogasawara | This bug was about a system hanging on reboot. | 17:27 |
| ogasawara | This particular bug was fixed by creating a patch to add a reboot quirk for this particular system. | 17:27 |
| ogasawara | This patch would be beneficial not only for Ubuntu users but all users who had this hardware. | 17:28 |
| ogasawara | As such, this was a good patch candidate to send upstream. | 17:28 |
| ogasawara | From reading the report, you can see the original bug reporter was able to confirm the patch fixed the issue. | 17:28 |
| ogasawara | I, the submitter, carefully read the upstream SubmittingPatches document and made sure to adhere to the items outlined in that document. | 17:28 |
| ogasawara | Let's look at the actual email I posted: | 17:29 |
| ogasawara | http://lists-archives.org/linux-kernel/27263461-x86-asus-p4s800-reboot-quirk.html | 17:29 |
| ogasawara | 1) I tried to make sure I had a clear and well written description. | 17:29 |
| ogasawara | I made sure to clearly state what the issue was and what it is the patch was fixing. | 17:29 |
| ogasawara | I provided system information to help identify hardware that this patch would impact. | 17:29 |
| ogasawara | I made sure to note that the bug reporter had tested and confirmed the patch resolves the issue. | 17:29 |
| ogasawara | Additionally, I explained how and why it is I crafted the patch the way I did. | 17:30 |
| ogasawara | 2) I made sure to follow the upstream kernel coding style | 17:30 |
| ogasawara | 3) As you can see in the link, I inlined the patch within the body of the email | 17:30 |
| ogasawara | 4) "[PATCH]" was prefixed in the subject line of the email | 17:30 |
| ogasawara | 5) I examined the MAINTAINERS files to see that I should send the patch to the three subsystem maintainers (Thomas Gleixner, Ingo Molnar, H. Peter Anvin) and the x86 mailing list. I also CC'd LKML for good measure. | 17:31 |
| ogasawara | 6) I also saw from the MAINTAINERS file that they had their own linux-2.6-x86.git tree so I made sure my patch cleanly applied to the tip of this tree. | 17:31 |
| ogasawara | 7) As a final task, I made sure I signed off on the patch by adding my "Signed-off-by:" line. | 17:31 |
| ogasawara | The one thing that I failed to do was mention the version of the kernel this should be applied to. | 17:32 |
| ogasawara | :/ | 17:32 |
| ogasawara | Thankfully the patch was accepted and applied even with this slight oversight. | 17:32 |
| ogasawara | http://git.kernel.org/?p=linux/kernel/git/torvalds/linux-2.6.git;a=commit;h=4832ddda2ec4df96ea1eed334ae2dbd65fc1f541 | 17:32 |
| ogasawara | ^^ there's the patch in it's upstream form | 17:32 |
| ogasawara | Additionally the maintainer CC'd upstream stable when applying this patch. | 17:32 |
| ogasawara | As a result we have already had the patch come back to Karmic via the 2.6.31.9 upstream stable kernel. | 17:33 |
| ogasawara | This is the typical scenario of how a patch goes upstream first and then makes it's way back into the Ubuntu kernel. | 17:33 |
| ogasawara | There are obviously exceptions that are sometimes made. | 17:33 |
| ogasawara | If a patch is for example a Critical bug fix, the Ubuntu kernel team may opt to temporarily carry the patch while it's going through the upstream submission process. | 17:33 |
| ogasawara | The patch must still adhere to the usual patch submission policy and procedures. | 17:34 |
| ogasawara | There are also cases where the patch is Ubuntu specific and therefore does not make sense being submitted upstream. | 17:34 |
| ogasawara | For example, https://bugs.launchpad.net/ubuntu/+source/linux/+bug/393517 | 17:34 |
| ogasawara | This was an Ubuntu specific issue where we failed to completely remove a driver. | 17:34 |
| ogasawara | If the patch is an Ubuntu specific patch, just send it to the Ubuntu kernel team mailing list. | 17:34 |
| ogasawara | Additionally opening a bug in Launchpad to track the issue would also be good (and required when we go to fix the issue) | 17:35 |
| ogasawara | Let's quickly pause again here for questions. | 17:35 |
| ogasawara | Otherwise I'll keep going | 17:35 |
| ogasawara | OK, lets keep moving | 17:36 |
| ogasawara | == Reasons for not applying a patch == | 17:36 |
| ogasawara | If it isn't good enough for upstream it likely isn't good enough for us. | 17:36 |
| ogasawara | That's not 100% true, but the basic idea is that if the patch is not in upstream, that generally means the patch is not likely ready for mass consumption. | 17:36 |
| ogasawara | Ubuntu's kernel is not a testing ground for experimental features. We need stability. | 17:37 |
| ogasawara | As I mentioned earlier, maintaining patches in our tree is not an easy task if we have a large delta of divergence between our tree and the upstream tree. | 17:37 |
| ogasawara | If we have a small delta to rebase, it makes maintaining the kernel much simpler. | 17:37 |
| ogasawara | Also, if the code has deviated from what's upstream, it's hard to get any additional help from the upstream developers to debug any issues that may arise. | 17:37 |
| ogasawara | There could also be security repercussions by carrying custom patches that are not upstream. | 17:38 |
| ogasawara | The patch could contain potential security vulnerabilities, which we would likely never hear about since the code isn't part of the stock vanilla kernel and routinely being examined. | 17:38 |
| ogasawara | Patching stock code could also introduce bugs. Adding bugs is bad. | 17:38 |
| ogasawara | It also might be that the patch is not against a new enough kernel. | 17:39 |
| ogasawara | The issue and fix may not be relevant against the actively developed kernel. | 17:39 |
| ogasawara | We also have to be careful about patches that may have a broad reaching impact. | 17:39 |
| ogasawara | Even though a patch may fix an issue for one person, it may regress 10 others. | 17:39 |
| ogasawara | Finally, the simplest explanation for why a patch might not be applied is that the patch might not be a legitimate patch which should be carried. | 17:40 |
| ogasawara | Like I mentioned earlier, band aid's and hacky workarounds will not be applied wholesale. | 17:40 |
| ogasawara | So what happens now if a patch was not accepted??? | 17:40 |
| ogasawara | There's usually a good reason why a patch wasn't accepted and it's usually explained in a response to the patch submission email. | 17:40 |
| ogasawara | If there are suggestions for improvements that need to be completed before a patch will be accepted, make the improvements and resubmit. | 17:41 |
| ogasawara | Most patches go through this type of review process and will need to be modified and re-submitted a few times before they are accepted. | 17:41 |
| ogasawara | As I mentioned before, don't give up. | 17:41 |
| ogasawara | In the end, there will however still be patches that never make it upstream nor into Ubuntu. | 17:42 |
| ogasawara | An alternative if someone so chooses would be to roll their own kernel with the patch applied. | 17:42 |
| ogasawara | Anyone should be able to do this by following the kernel compile guide | 17:42 |
| ogasawara | https://help.ubuntu.com/community/Kernel/Compile | 17:42 |
| ogasawara | Note that building and booting your own custom kernel is done purely at the users own risk | 17:43 |
| ogasawara | Next week we're looking to clean up this Kernel/Compile guide to be more user friendly and easier to follow. | 17:43 |
| ogasawara | I was going to pause once more in case there are any additional questions, otherwise I'll start to wrap it up | 17:44 |
| ogasawara | vishalrao_vbox> QUESTION: after you submit a patch, how do you get notified whether it was accepted or rejected? | 17:44 |
| ogasawara | vishalrao_vbox: typically you've receive an email from the maintainer that the patch has been applied | 17:44 |
| ogasawara | vishalrao_vbox: otherwise they'll likely send feedback on what needs to be fixed up with the patch | 17:45 |
| ogasawara | Navaneeth> QUESTION: Do you add new features to Ubunutu kernel without upstream having it? If yes, can you list out a feature like that? | 17:45 |
| ogasawara | Navaneeth: we typically try to refrain from doing such a thing | 17:45 |
| ogasawara | Navaneeth: again, it's that whole pain point of maintaining something out of tree | 17:46 |
| ogasawara | anything else? | 17:46 |
| ogasawara | vishalrao_vbox> QUESTION: what is this "SAUCE" we see mentioned in some ubuntu kernel changelogs? | 17:46 |
| ogasawara | vishalrao_vbox: those are the Ubuntu specific patches | 17:46 |
| ogasawara | vishalrao_vbox: those ones will likely never go back upstream as they don't make sense to go back upstream | 17:47 |
| ogasawara | vishalrao_vbox> ogasawara: S.A.U.C.E. stands for? (acronym?) | 17:47 |
| ogasawara | heh, I don't think it's actually an acronym in our usage of it, if it is I don't know what it stands for :) | 17:48 |
| ogasawara | my understanding it implies it our "special sauce" patches | 17:48 |
| ogasawara | anything else? | 17:48 |
| ogasawara | vishalrao_vbox> yes, i figured, i was wondering if its all uppercase for a reason :) | 17:49 |
| ogasawara | it's uppercase so it's easy for us to spot and search against in the logs | 17:49 |
| ogasawara | we try to examine all our sauce patches each release and figure out why we have them and if they are still needed | 17:49 |
| ogasawara | ok, lets move on to the last item | 17:50 |
| ogasawara | == Getting Help == | 17:50 |
| ogasawara | I'd like to remind anyone that if you need assistance or have any questions regarding a patch, need a patch to be reviewed, or need help getting the patch submitted upstream, the Ubuntu kernel team would be happy to help. | 17:50 |
| ogasawara | Please contact us through the kernel team mailing list or in the #ubuntu-kernel IRC channel on FreeNode. | 17:50 |
| ogasawara | If you need to know the kernel team mailing list address, refer to our wiki.ubuntu.com/KernelTeam | 17:51 |
| ogasawara | Also, most of what we covered today I'd previously documented at https://wiki.ubuntu.com/KernelTeam/KernelPatches | 17:51 |
| ogasawara | I've also started graphing some trends for bugs with patches attached: | 17:51 |
| ogasawara | http://qa.ubuntu.com/reports/ogasawara/csv-stats/bugs-with-patches/linux/ | 17:52 |
| ogasawara | It would be great to see if we can drive the number of bugs with patches attached down to 0. | 17:52 |
| ogasawara | We've already made this a focus of previous kernel team bug days, but any additional help is appreciated. | 17:52 |
| ogasawara | And one last shameless plea. . . if anyone is interested in getting involved but just doesn't know where to start, let me know! | 17:52 |
| ogasawara | I'll be glad to help get you started. | 17:53 |
| ogasawara | Anyways, thanks again to all for listening. I'll finish to fielding the remaining questions. | 17:53 |
| ogasawara | as time permits :) | 17:53 |
| ogasawara | vocx> QUESTION: what percentage of bug reports filed against the kernel are invalid? I'm under the impression that a lot of new users incorrectly file bug reports against the kernel, which contributes to much "noise". | 17:53 |
| ogasawara | vocx: it is true that the kernel does get a large amount of bugs that are not valid against the kernel, but they are still valid in a sense that it's an issue against a different package | 17:54 |
| ogasawara | vocx: I can't quote an exact number unfortunately | 17:54 |
| ogasawara | vocx: but we do appreciate the community triage help on making sure the bug finds it's way to the proper package | 17:55 |
| ogasawara | vocx> QUESTION: what percentage of patches are rejected? | 17:55 |
| ogasawara | vocx: I'd say a very small percentage of patches are outright rejected | 17:55 |
| ogasawara | vocx: probably less than 5%? | 17:55 |
| ogasawara | vocx: If an issue is legitimate it can stay open | 17:56 |
| ogasawara | vocx: patches are always welcome :) | 17:56 |
| ogasawara | vocx> QUESTION: how much experience do you need to be able to submit a patch to the Linux-freaking-Kernel? | 17:56 |
| ogasawara | vocx: you will need to have a general understanding of the kernel to know that the patch is correct | 17:57 |
| ogasawara | vocx: but it's not rocket science :) | 17:57 |
| ogasawara | <ecm123> QUESTION: I've heard the Nouveau driver was going to be included in in lucid's kernel, however lucid will not use the 2.6.33 kernel in which this driver is included by default, does that mean the ubuntu kernel team will maintain the driver all by itself in lucid? | 17:57 |
| ogasawara | ecm123: yes there are talks/works to have this backported and available | 17:58 |
| ogasawara | anything else? | 17:59 |
| ogasawara | 1min . . . | 17:59 |
| ogasawara | Navaneeth> QUESTION: As promised, can we have the online materials for debugging kernel source? | 17:59 |
| ogasawara | of course, give me a few minutes to gather the links and I'll post it to ubuntu-classroom-chat so not to run over into the next session | 17:59 |
| ogasawara | ok, that's it for me | 18:00 |
| ogasawara | thanks everyone! | 18:00 |
| * ogasawara hands off to bdmurray | 18:00 | |
| * bdmurray takes the mic | 18:00 | |
| bdmurray | Hi, my name is Brian Murray and I'm a member of the Ubuntu QA team. | 18:01 |
| bdmurray | I'm here today to talk about how you can get higher quality bug reports about packages that you care about. | 18:01 |
| bdmurray | One way you can do this is by writing an apport hook for your particular package. | 18:02 |
| bdmurray | Let's start off by talking about what apport is. | 18:02 |
| bdmurray | Apport is a system which intercepts crashes right when they happen, in development releases of Ubuntu, and gathers useful information about the crash and the operating system environment. | 18:02 |
| bdmurray | Additionally, it is used as a mechanism to file non-crash bug reports about software. | 18:02 |
| bdmurray | This allows us to define what software we want to receive bug reports about and what information will be included in those bug reports. | 18:03 |
| bdmurray | Let's look at a sample apport bug report - http://launchpad.net/bugs/416701. | 18:03 |
| bdmurray | The bzr package does not have an apport hook but some useful information is still collected. | 18:03 |
| bdmurray | We have the architecture, the release being used, the package version and the source package name. | 18:04 |
| bdmurray | Additionally, in the Dependencies.txt attachment we have information about the versions of packages upon which bzr depends. | 18:04 |
| bdmurray | Are there any questions so far? | 18:04 |
| bdmurray | Okay, so while all of that can be helpful an apport hook for a package allows us to gather specific information for a package. | 18:06 |
| bdmurray | For example, consider a bug report about usplash. usplash has a dedicated configuration file, located at "/etc/usplash.conf", and this would be something quite helpful in debugging a usplash bug report but not very useful in other package bug reports. | 18:06 |
| bdmurray | Apport looks for package hooks in "/usr/share/apport/package-hooks/" on the local system for ones named after the package for which they will be used. | 18:06 |
| bdmurray | Looking in "/usr/share/apport/package-hooks/" lets take a look at the usplash hook - with the filename usplash.py. I've also pastebin'ed the hook to http://pastebin.ubuntu.com/364661/. | 18:07 |
| bdmurray | The package hooks are written in python. | 18:07 |
| bdmurray | We can see that the usplash.py hook imports the apport.hookutils module - "from apport.hookutils import *". | 18:07 |
| bdmurray | hookutils is a collection of readymade and safe functions for many commonly used things. There are functions for attaching a file's contents, getting a command's output, grabbing hardware information and much more. | 18:08 |
| bdmurray | This package hook is using 'attach_file_if_exists' and 'attach_hardware'. 'attach_file_if_exists' is pretty self explanatory but what does attach_hardware include? | 18:08 |
| bdmurray | Let's look at the hookutils module to find out. You can use "python -c 'import apport.hookutils; help(apport.hookutils)'" or you can view it using codebrowse - http://bazaar.launchpad.net/~apport-hackers/apport/trunk/annotate/head%3A/apport/hookutils.py. | 18:09 |
| bdmurray | The 'attach_hardware' function starts at line 72. | 18:09 |
| bdmurray | As we can see it adds a wide variety of hardware information to a bug report which can be quite useful for some packages like the kernel and usplash! | 18:10 |
| bdmurray | Having the apport package hooks has reduced the amount of bug "ping pong" necessary to get information out of a bug reporter and having these convenience functions reduces the amount of work it takes to write a package hook. | 18:11 |
| bdmurray | In addition to 'attach_hardware' and 'attach_file_if_exists' other functions include: 'attach_conffiles', 'attach_dmesg', 'attach_alsa', 'command_output', 'recent_syslog', 'pci_devices' and 'attach_related_packages'. | 18:12 |
| bdmurray | In the event that you have a group of packages that would benefit from a shared convenience function, please file a bug about apport and include the function you'd like added. | 18:13 |
| bdmurray | Are there any questions so far? | 18:13 |
| bdmurray | So, back to the usplash hook, we can see that the usplash configuartion file is added like so "attach_file_if_exists(report, '/etc/usplash.conf', 'UsplashConf')". | 18:15 |
| bdmurray | This means that a bug reported about usplash using apport should have an attachment named 'UsplashConf' and it will contain the reporter's usplash.conf file. | 18:15 |
| bdmurray | Looking at http://launchpad.net/bugs/393238 we can see this actually isn't the case. | 18:16 |
| bdmurray | Because usplash.conf is only 4 lines it ends up getting put into the bug description. However, most items end up getting added as attachments in Launchpad. | 18:16 |
| bdmurray | For example, lspci-vnvn.log or BootDmesg.txt | 18:17 |
| bdmurray | Let's take a look at another package hook for apport. | 18:17 |
| bdmurray | The totem hook is particularly interesting as it is utilizes interactive questions - which is a new feature in apport. | 18:18 |
| bdmurray | You can see how this works by executing "ubuntu-bug totem", but please don't actually report the bug! ;-) | 18:18 |
| bdmurray | More detailed information regarding how to use the hook user interface can be found in the python help for the available functions: "python -c 'import apport.ui; help(apport.ui.HookUI)'". | 18:19 |
| bdmurray | The totem hook asks questions and runs tests to determine if the bug report is related to alsa, pulseaudio or codecs in gstreamer. | 18:20 |
| bdmurray | It then sets the affected package appropriately. | 18:20 |
| bdmurray | The interactive hook greatly reduces the amount of triaging work required (as questions are asked before the bug is reported instead of after) and helps make the initial bug report much more complete. | 18:21 |
| bdmurray | Oh, here's a link to the totem hook - http://pastebin.ubuntu.com/364672/ | 18:22 |
| bdmurray | Are there any questions regarding the material I've covered so far? | 18:23 |
| bdmurray | Now that we know a lot of what hooks can do, lets talk about how to write one. | 18:25 |
| bdmurray | After the Karmic Ubuntu Developer Summit I compiled a list of packages that had recently received a fair number of bug reports and subsequently might benefit from an apport package hook. | 18:26 |
| bdmurray | That list can be found at https://wiki.ubuntu.com/QATeam/Specs/IncreaseApportCoverage. | 18:26 |
| bdmurray | < yltsrc> QUESTION: where i can find hooks for all default ubuntu | 18:27 |
| bdmurray | applications? | 18:27 |
| bdmurray | yltsrc: they will be installed on your system in "/usr/share/apport/package-hooks". Each hook is distributed with the package it is used for instead of being distributed with apport. | 18:28 |
| bdmurray | This allows developers to add hooks to their own package and not having modify 2 packages. | 18:28 |
| bdmurray | < qense> QUESTION: Is there a nice table showing what packages have and | 18:29 |
| bdmurray | what haven't got Apport hooks? | 18:29 |
| bdmurray | qense: there is a likely incomplete list at https://wiki.ubuntu.com/Apport/PackageHooks | 18:30 |
| bdmurray | I'll update it soon though | 18:30 |
| bdmurray | Back to writing an apport hook - Let's take rhythmbox from that list and write a simple apport hook for it. | 18:31 |
| bdmurray | The easiest way of writing and testing a package hook is to put one in "/usr/share/apport/package-hooks" named after the appropriate package. | 18:31 |
| bdmurray | I'll create one called "source_rhythmbox.py". | 18:32 |
| bdmurray | We'll import hookutils from apport and os, then in the add_info function we'll add the rhythmdb.xml file, if it exists, and gconf settings for rhythmbox. | 18:34 |
| bdmurray | The hook as written looks like http://pastebin.ubuntu.com/262850/. | 18:34 |
| bdmurray | Its really only 2 lines of code but now we'll have some potentially useful information in every rhythmbox crash or bug report, | 18:35 |
| bdmurray | After I've put this file in "/usr/share/apport/package-hooks" I can test it using "ubuntu-bug". | 18:36 |
| bdmurray | After running "ubuntu-bug rhythmbox", apport presents the user with a dialog asking them if they want to send the problem report. | 18:37 |
| bdmurray | In this dialog box we can see the complete contents of the report and if our collected information was attached. | 18:38 |
| bdmurray | I see a key named rhythmdb.xml which contains the same information as the one on my system and a key named GconfRhythbox which contains my gconf settings for rhythmbox. | 18:39 |
| bdmurray | That's really all there is to writing and testing an apport hook! | 18:39 |
| bdmurray | In the event that you write a hook for a package that you can not upload or need help getting sponsored, please report a bug about the package missing a hook. | 18:40 |
| bdmurray | Then add the hook as a patch (or a merge proposal) and subscribe me, brian-murray, and the appropriate sponsor's team to the bug report. | 18:40 |
| bdmurray | I'll work to ensure that it gets uploaded. | 18:41 |
| bdmurray | Another thing to consider is that a hook isn't just useful for bug reports going forward, | 18:41 |
| bdmurray | it could actually help you out with your existing bug reports! | 18:42 |
| bdmurray | For example, if you write a hook for your package right now and get it included in Lucid you could then have bug reporters execute 'apport-collect' and the number of the bug they've reported. | 18:43 |
| bdmurray | This will then use the package hook to include the information you want in the already reported bug. | 18:43 |
| bdmurray | Are there any more questions about writing apport package hooks? | 18:44 |
| bdmurray | Another way to get more detailed bug reports is by setting the bug filing guidelines for packages in Launchpad. | 18:47 |
| bdmurray | These are presented when someone files a bug via the web interface at Launchpad. | 18:48 |
| bdmurray | For example if you were to file a bug about ffmpeg at https://bugs.edge.launchpad.net/ubuntu/+source/ffmpeg/+filebug ... you would be presented with the package's bug filing guidelines - http://people.canonical.com/~brian/tmp/package-bug-filing-guidelines.png. | 18:48 |
| bdmurray | This particular package's bug guidelines asks for specific information that would be rather challenging to gather with an apport hook. | 18:49 |
| bdmurray | The bug reporting guidelines for ffmpeg can be set at https://edge.launchpad.net/ubuntu/+source/ffmpeg/+edit. The url will be similar for any other package | 18:50 |
| === z is now known as Guest44291 | ||
| bdmurray | In the event that you don't have the ability to set the guidelines for a package you care about for some reason, please let me know and I'll be happy to get them set for you. | 18:50 |
| bdmurray | I see there are some questions! | 18:51 |
| bdmurray | < barcc> QUESTION: can i use apport hook for a project not included in ubuntu but hostet on launchpad? | 18:51 |
| bdmurray | barcc: possibly. you might look at the ubuntuone-client hook as an example. | 18:52 |
| bdmurray | In particular the following lines | 18:52 |
| bdmurray | if not apport.packaging.is_distro_package(report['Package'].split()[0]): | 18:52 |
| bdmurray | report['ThirdParty'] = 'True' | 18:52 |
| bdmurray | report['CrashDB'] = 'ubuntuone' | 18:52 |
| bdmurray | < vocx> QUESTION: are debug symbols required to be installed before calling apport? what are debug symbols used for? | 18:53 |
| bdmurray | No they are not required, an incomplete (w/o debugging symbols) is uploaded to Launchpad and the the retracer takes care of installing the debug symbols and retracing the bug for you | 18:54 |
| bdmurray | You could install the debugging symbols your self if you wanted to manually run gdb and get some detailed output about a crash | 18:54 |
| bdmurray | < barcc> QUESTION: is it possible to test apport hooks against lauchpad staging? | 18:55 |
| bdmurray | barcc: I believe so. If you look at /etc/apport/crashdb.conf there might be a line like #'staging': True | 18:57 |
| bdmurray | You could comment that out and then report a bug. | 18:57 |
| bdmurray | However, I've found the dialog that apport presents before filing a bug to be sufficient | 18:58 |
| bdmurray | Are there any more questions? | 18:58 |
| bdmurray | If you need any further help the apport code is very well documented and there are a few wiki pages about it - https://wiki.ubuntu.com/Apport and https://wiki.ubuntu.com/Apport/DeveloperHowTo. Additionally, you can find me in the ubuntu-bugs channel. Thanks! | 18:58 |
| bdmurray | Okay, thanks everyone and have a great day! | 19:00 |
| jelmer | Thanks! | 19:05 |
| jelmer | Hello everybody! | 19:05 |
| jelmer | I'm Jelmer Vernooij, one of the Bazaar developers and an occasional contributor to Debian and Ubuntu | 19:06 |
| jelmer | I'll be leading the next class about packaging with Bazaar | 19:06 |
| jelmer | For this class you will need the bzr-builddeb package | 19:07 |
| jelmer | Today we'll have a look at packaging branches on Launchpad, building packages from a branch and making changes to a packaging branch. | 19:09 |
| jelmer | Is everybody here familiar with Bazaar, or should I give a quick introduction to it as well? | 19:10 |
| jelmer | Great, let's dig into obtaining a packaging branch | 19:11 |
| jelmer | As of a couple of releases ago, Launchpad supports packaging branches | 19:11 |
| jelmer | You can find the packaging branches for a particular package by clicking on "Branches" when viewing a source package on Launchpad | 19:11 |
| jelmer | For example, https://code.edge.launchpad.net/ubuntu/+source/tdb contains the packaging branches for the 'tdb' package | 19:13 |
| jelmer | These branches are different from the upstream branches that live at http://code.launchpad.net/<project>, and contain upstream | 19:14 |
| jelmer | QUESTION: can you clarify what a package branch is, and how it's different from a regular branch | 19:14 |
| jelmer | A package branch is a bzr branch that contains a source package, i.e. the upstream including any changes that have been made to it and the debian/ directory | 19:15 |
| jelmer | Other than the contents there isn't anything special about a packaging branch | 19:16 |
| jelmer | https://code.edge.launchpad.net/ubuntu/+source/tdb lists all of the Ubuntu packaging branches, but James also imports the Debian archive | 19:17 |
| jelmer | for example, you can have a look at https://code.edge.launchpad.net/debian/+source/tdb which contains the Debian packaging branches | 19:17 |
| jelmer | QUESTION: how do you keep the code in a project branch and it's associated package branch in sync? | 19:17 |
| jelmer | You can use the "bzr merge-upstream" package to merge in new upstream tarballs | 19:19 |
| jelmer | This command will also take care of setting the right tags, etc | 19:19 |
| jelmer | But before we get ahead of ourselves.. | 19:20 |
| jelmer | Let's check out the tdb package and try to build it | 19:20 |
| jelmer | If you're familiar with bzr this should be no surprise to you: | 19:20 |
| jelmer | $ bzr branch lp:ubuntu/tdb tdb-lucid | 19:20 |
| jelmer | After this, you should be able to go into the branch and run "bzr builddeb" (or "bzr bd" for short) | 19:21 |
| jelmer | this will export the current working tree and try to build it | 19:21 |
| jelmer | QUESTION: What is the standard naming convention of a bzr ? lp:~ for some and it's lp:<version>/<project> for some.. | 19:22 |
| jelmer | For upstream branches, it is always ~OWNER/PROJECT/NAME | 19:22 |
| jelmer | for packaging branches, it is ~OWNER/DISTRO/SERIES/PACKAGE/NAME | 19:23 |
| jelmer | but there are some abbreviated names available - if SERIES is the development series, you can get by by just specifying the DISTOR and the PACKAGE | 19:23 |
| jelmer | QUESTION:is there a way to search the lp bzr from cli? | 19:24 |
| jelmer | There is no way to query the lists of branches on Launchpad using the bzr cli at the moment | 19:25 |
| jelmer | Mainly because nobody has had the time to add it yet, patches to add such support would certainly be welcome :-) | 19:26 |
| jelmer | (It should also be possible to run "bzr bd lp:ubuntu/tdb", but I haven't tried that in a while) | 19:26 |
| jelmer | now that we've built the package, let's have a look at making some changes | 19:27 |
| jelmer | The idea is that in the future it will be possible to upload a new version of package simply by pushing to lp:ubuntu/<packagename> | 19:27 |
| jelmer | we're not quite there yet, at the moment you will still have to manually build the sourcepackage and upload it. | 19:28 |
| jelmer | (upload it using dput, that is) | 19:29 |
| jelmer | There are importer scripts, run by james_w, that update lp:ubuntu/<package> to match whatever has been uploaded using ftp. | 19:29 |
| jelmer | QUESTION: my build failed because it couldn't sign the package | 19:29 |
| jelmer | you can use "bzr bd -- -uc -us" to skip the signing, as I assume most of you won't have access to my private GPG key ;-) | 19:29 |
| jelmer | QUESTION: Do ubuntu-stable-release updates work the same way, except one proposes a merge instead of attaching a diff to a bug? | 19:30 |
| jelmer | I'm not familiar with the ubuntu-stable-release process, can somebody else tell perhaps? | 19:31 |
| jelmer | QUESTION: If I want to create a new package branch for a package not yet in Ubuntu, where should I put it? | 19:31 |
| jelmer | That's a good question - at the moment it is not possible to push a new branch for a package that doesn't exist in Ubuntu yet | 19:32 |
| jelmer | So at the moment you will have to wait until the first version of your package has been uploaded before you can push your branches | 19:32 |
| jelmer | Supporting pushing before the upload is something that is on the todo list of the launchpad developers. | 19:33 |
| jelmer | Let's have a look at making a change and preparing an upload. | 19:34 |
| jelmer | Let's make a dummy change, perhaps fixing the outdated docs? | 19:34 |
| jelmer | $ echo "Don't panic." > docs/README | 19:34 |
| jelmer | Next, we should of course update the changelog: | 19:34 |
| jelmer | $ dch -i "Fix the documentation. " | 19:35 |
| jelmer | When we would like to commit this change, the only thing we have to do is run "debcommit" | 19:35 |
| jelmer | $ debcommit | 19:36 |
| jelmer | debcommit will create the commit in Bazaar and will automatically figure out the correct commit message based on the changes to the changelog | 19:37 |
| jelmer | Next, we should be able to build the source package that we'd like to upload | 19:37 |
| jelmer | e.g. "bzr bd -S" (-S for source package) | 19:37 |
| jelmer | QUESTION: why is debcommit separate from the bzr- builddeb commands? | 19:38 |
| jelmer | debcommit is part of the debian developer scripts, and works for a variety of version control systems | 19:38 |
| jelmer | in newer versions of bzr-builddeb you can also use just "bzr commit" | 19:39 |
| jelmer | and bzr will fill in a suggested commit message, similar to the one debcommit uses | 19:39 |
| jelmer | once you have uploaded the package and are happy with it, you should tag the version in bzr by running "bzr mark-uploaded" | 19:40 |
| jelmer | "bzr tags" should list all of the existing tags | 19:40 |
| jelmer | the tags should all you to easily navigate all of the versions of a package using bazaar | 19:42 |
| jelmer | An easy way to browse the history is by using "bzr viz" from the bzr-gtk package | 19:43 |
| jelmer | Another command that is provided by the bzr-builddeb package is "bzr import-dsc" | 19:43 |
| jelmer | This command allows you to import existing tarballs that have been published in the past | 19:44 |
| jelmer | it is the main engine behind the importer scripts that create the branches owned by ~ubuntu-branches | 19:44 |
| jelmer | bzr-builddeb also includes two commands for merging changes into your packaging branch: import-upstream and import-package | 19:45 |
| jelmer | import-upstream allows you to import a new upstream version into your package - you specify a tarball and a version string and it will take care of updating your packaging branch and update a new entry to the debian changelog | 19:46 |
| jelmer | s/update/add/ | 19:46 |
| jelmer | and s/import-upstream/merge-upstream/, obviously - sorry | 19:46 |
| jelmer | merge-package allows you to merge from another packaging branch | 19:46 |
| jelmer | for example, if you need to merge changes from a debian packaging branch you can use merge-package | 19:47 |
| jelmer | among other things, this will take care of merging debian/changelog in a smarter way than "bzr merge" does | 19:47 |
| jelmer | Is everybody keeping up so far? | 19:47 |
| jelmer | merge-package should no longer be necessary after the next bzr release, when "bzr merge" will just Do The Right Thing. | 19:48 |
| jelmer | That concludes what I wanted to talk about - are there any more questions or would you like me to expand on anything? | 19:49 |
| jelmer | QUESTION: I noticed multiple .deb packages were made, does a package branch create more than one package? | 19:50 |
| jelmer | A package branch is like a versioned source package - like a source package multiple binary packages can be created from it | 19:51 |
| jelmer | QUESTION: are there some good resources for someone new to debian packaging to better understand the structure and process? | 19:52 |
| jelmer | You might want to read the log from the getting started session: https://wiki.ubuntu.com/MeetingLogs/devweek1001/GetStarted | 19:53 |
| jelmer | There are also some good resources on the wiki and in the Debian developer corner | 19:54 |
| jelmer | QUESTION: years ago, I think I read 'bazaar' and 'bzr' were two different, closely related programs, maybe forks. Is this true nowadays? Is 'bazaar' exactly the same as 'bzr'? | 19:55 |
| jelmer | The bazaar project has had two different incarnations: "baz" (written in C, derived from arch) and "bzr" (written in Python, from scratch) | 19:55 |
| jelmer | baz used to own the package name "bazaar" but has been removed from the archive now, "bzr" is the current app that we work on. The "bzr baz-import" command can import from old repositories. | 19:56 |
| jelmer | (The developers that used to work on baz all work on bzr these days) | 19:58 |
| jelmer | Any other questions? Feature requests? Rotten tomatoes? | 19:59 |
| jelmer | QUESTION: unrelated, but is there a bzr plugin like mercurial queues? I know about loom, but it doesn't let change the order or patches, etc | 20:00 |
| jelmer | There is also bzr-pipeline, which is similar to but not entirely unlike looms. | 20:01 |
| jelmer | I'm not sure if it supports reordering of patches, but you might want to check it out. | 20:02 |
| jelmer | Thanks for your attention, happy hacking! | 20:02 |
| * kfogel claps for jelmer | 20:02 | |
| kfogel | AFAIK, I'm teaching the next class. Should I wait for some official signal to start? :-) | 20:03 |
| jelmer | Thanks Karl :-) | 20:03 |
| jelmer | I don't think there is a chair at the moment | 20:03 |
| kfogel | jelmer: did dholbach or someone lead you in? (I don't see any record of it...) | 20:04 |
| jelmer | The next speaker up is Karl Fogel; he works on Launchpad and will introduce you to the wonderful world of Launchpad development! | 20:04 |
| kfogel | Indeed! | 20:04 |
| kfogel | Thank you, Jelmer. | 20:04 |
| kfogel | Hi, everyone. To start with, I am by no means the world's greatest expert in lp development. | 20:04 |
| kfogel | I have only recently started doing it. | 20:04 |
| kfogel | That said, I'm happy to help you get started too -- at least it's fresh in my mind! | 20:04 |
| kfogel | :-) | 20:04 |
| kfogel | The intended audience is people who use Launchpad -- especially Ubuntu maintainers, but not limited to them -- and see things they might want to improve in Launchpad. Some users who have already contributed to Launchpad are listed here: | 20:05 |
| kfogel | https://dev.launchpad.net/Contributions | 20:05 |
| kfogel | You too can be listed on that page :-). | 20:05 |
| kfogel | (That list is automatically updated, so if you hack Launchpad, your name will eventually show up there too.) | 20:05 |
| kfogel | For this class, you should have a web browser open. You should have a little familiarity with Bazaar or some other decentralized version control system, plus basic comfort with Python and HTML -- but you do not need to be an expert in any of these things. | 20:05 |
| kfogel | I hope that by the end of the session you will feel like you can contribute an improvement to Launchpad whenever you see something that needs fixing. | 20:05 |
| kfogel | This isn't going to be a write-a-change-in-realtime kind of session, because there's too much to tell you about and I want to keep things moving. Instead, the idea is that you would keep the transcript of this session around, and come back to it later for a closer look (especially at the examples) if/when you're ready to start writing your own changes. | 20:06 |
| kfogel | Note that much of whath I say here will be stuff you can also find in the dev.launchpad.net wiki. But the wiki can be a little daunting. I hope this narrative presentation will give a useful overview of how things go. | 20:06 |
| kfogel | GETTING LAUNCHPAD | 20:06 |
| kfogel | You may want to start downloading the Launchpad source code if you don't have it already (although it's not really necessary for what we're going to talk about today, it's just good to have so you can explore around). | 20:06 |
| kfogel | Now, normally, the "right" way to obtain the Launchpad source code is to follow the instructions at https://dev.launchpad.net/Getting . | 20:06 |
| kfogel | But that's actually a somewhat non-trivial process, and involves running a script (I'll explain more later). So for now, if you just want to get the code fast, so you can explore it during this session, you can download some pre-packaged, compressed Bazaar repositories I've made for you :-). (But note that when you actually start hacking on Launchpad, you'll want to go back and follow the instructions at https://dev.launchpad.ne | 20:07 |
| kfogel | t/Getting .) | 20:07 |
| kfogel | So... | 20:07 |
| kfogel | To avoid overload an any one server, let's do it this way: if your family name starts with the letters "A-M", you can start the download from here: | 20:07 |
| kfogel | http://www.red-bean.com/kfogel/lp-hacking/lp-repos.tar.gz | 20:07 |
| kfogel | And if your family name starts with the letters "N-Z", get it from here: | 20:07 |
| kfogel | http://people.canonical.com/~kfogel/lp-hacking/lp-repos.tar.gz | 20:07 |
| kfogel | (How do you like my load-balancing algorithm? I'm thinking of patenting it!) | 20:07 |
| kfogel | Just kidding. I sure hope you're all on slow connections, though :-). | 20:07 |
| kfogel | Anyway, when you unpack it with | 20:07 |
| kfogel | tar zxvf lp-repos.tar.gz | 20:07 |
| kfogel | you'll get a new subdirectory named lp-repos, and inside that will be a .bzr subdirectory and a branch named "devel". "devel" is where the Launchpad sources are. To actually build and run Launchpad, you need a bunch of other dependencies -- all of which is described at (you guessed it) https://dev.launchpad.net/Getting . | 20:08 |
| kfogel | 20:08 | |
| kfogel | At this point, I will pause, both for questions, and so people who want to can start download | 20:08 |
| kfogel | downloading. | 20:08 |
| kfogel | Note it is not really necessary to have the sources for this talk. I will give you pointers to all examples. | 20:08 |
| kfogel | Anyone there? :-) | 20:08 |
| * kfogel waits | 20:08 | |
| kfogel | (or say something in #ubuntu-classroom-chat if you can't speak here) | 20:09 |
| kfogel | Heh. I'll keep going, since the transcript will be useful. If anyone has a question at any point, please just speak up here or in #ubuntu-classroom-chat. | 20:09 |
| kfogel | RUNNING A LOCAL DEVELOPMENT LAUNCHPAD INSTANCE | 20:10 |
| kfogel | Basically, do what https://dev.launchpad.net/Running says . I won't actually try to get your instance running during this class, since there are so many people and we have other stuff to cover. But do note a couple of things: | 20:10 |
| kfogel | At one point in the setup, you run a script as root. Yes, that's terrible and awful. But you can look at the script first :-). The reason to do that is that a launchpad development instance runs on your local box but using the hostname "launchpad.dev" -- to test stuff, you'll point your browser at URLs that start like this: | 20:10 |
| kfogel | https://launchpad.dev/ | 20:10 |
| kfogel | https://bugs.launchpad.dev/ | 20:10 |
| kfogel | https://code.launchpad.dev/ | 20:10 |
| kfogel | https://blueprints.launchpad.dev/ | 20:10 |
| kfogel | https://translations.launchpad.dev/ | 20:10 |
| kfogel | https://answers.launchpad.dev/ | 20:10 |
| kfogel | https://lists.launchpad.dev/ | 20:11 |
| kfogel | Those would all be running on your local box during development. | 20:11 |
| kfogel | So the setup scripts modify your /etc/hosts (and a few other files, I think) to take care of this all automatically. | 20:11 |
| kfogel | The other thing is that if you use PostgreSQL, the scripts will blow any data you have in there. If that's a problem, just run it in a virtual machine. | 20:11 |
| kfogel | So, that's how you get it and set it up to run -- just do what that page says. | 20:11 |
| kfogel | And understand why it's so complicated! | 20:12 |
| kfogel | GETTING HELP | 20:12 |
| kfogel | The most important thing to realize is, doing Launchpad development is primarily about communications, not coding. You do not need mad coding skills, you just need to be comfortable talking to people about your proposed bugfix / enhancement / new feature. | 20:12 |
| kfogel | QUESTION: is there a choice as to which DB you can run it on | 20:12 |
| kfogel | ANSWER: not right now, AFAIK. It's tied to PostgreSQL. | 20:13 |
| kfogel | Except for really trivial changes (typo fixes and such), most of your time will be spent communicating with other developers, and writing the automated tests for the change. The actual time writing the change is often the smallest part of it. | 20:13 |
| kfogel | So even before I tell you how to get the source code, I'm going to tell you how to get help -- how to find the right people to talk to! | 20:13 |
| kfogel | Start here: | 20:13 |
| kfogel | dev.launchpad.net/Help | 20:13 |
| kfogel | join the mailing list launchpad-dev {AT} lists.launchpad.net | 20:14 |
| kfogel | And use these two IRC channels here on freenode; | 20:14 |
| kfogel | #launchpad-dev and #launchpad-reviews (for reviewing changes you've submitted) | 20:14 |
| kfogel | the first channel is the main one for conversation about Launchpad development. | 20:14 |
| kfogel | It is not to be confused with #launchpad, which is for *usage* of Launchpad | 20:14 |
| kfogel | Next thing is: | 20:15 |
| kfogel | WHAT TO WORK ON? | 20:15 |
| kfogel | Most likely, your own usage of Launchpad will drive what you're interested in improving. But if you want to get an idea of what we think needs fixing, take a look at https://dev.launchpad.net/FixBugs | 20:15 |
| kfogel | However, the usual way things work is that you, as a user of Launchpad, run into something that you think should be fixed, and you fix it. | 20:15 |
| kfogel | So our list of existing bugs is interesting, but not necessarily the driving force for community developers (do check there to see if your bug has already been filed, though). | 20:16 |
| kfogel | HOW TO START TO WORK ON SOMETHING | 20:16 |
| kfogel | You'll need three key things: a bug, a branch, and a merge proposal. | 20:16 |
| kfogel | If there isn't already a bug filed for what you want to do, then discuss the proposed change first, in #launchpad-dev or on the launchpad-dev {AT} lists.launchpad.net mailing list. Once there is general agreement, file a bug describing what you plan to do. (If it's a new feature, there may be a Blueprint involved too.) | 20:16 |
| kfogel | So, let's assume that by one route or another, you have a bug filed now. | 20:17 |
| kfogel | The next step is to roughly describe your proposed solution to an experienced Launchpad developer -- this is known as the "pre-implementation call", although | 20:17 |
| kfogel | it can also be an IRC chat session instead of a phone call. | 20:17 |
| kfogel | Once you've got the basic design agreed on (which is usually a pretty quick process), you make a branch. | 20:18 |
| kfogel | A "branch" is equivalent to a patch -- it's how you submit a change to the project. In order to get the dependencies set up right, I recommend starting one with 'rocketfuel-branch' | 20:18 |
| kfogel | $ cd MY_LP_BRANCHES | 20:18 |
| kfogel | $ ls | 20:18 |
| kfogel | devel db-devel stable db-stable | 20:18 |
| kfogel | $ utilities/rocketfuel-branch 123456-improve-fish | 20:18 |
| kfogel | <<... watch output fly by ...> | 20:18 |
| kfogel | $ ls | 20:18 |
| kfogel | 123456-improve-fish devel db-devel stable db-stable | 20:18 |
| kfogel | $ | 20:18 |
| kfogel | 20:18 | |
| kfogel | By the way, if you're wondering what that "devel", "db-devel", "stable", "db-stable" thing is, | 20:19 |
| kfogel | the answer is here: | 20:19 |
| kfogel | https://dev.launchpad.net/Trunk | 20:19 |
| kfogel | Basically, Launchpad has four trunks, instead of one trunk. There's stable vs devel, and there's db-changes vs no-db-changes. | 20:19 |
| kfogel | It's all explained on that page. | 20:19 |
| kfogel | 95% of the time, you'll be working against 'devel', so when I say "trunk", that's what I mean. | 20:19 |
| kfogel | In your branch, edit the code and use Bazaar as you normally would. Make as many commits as you want! | 20:20 |
| kfogel | It's okay to make lots of small commits. This is all just in yuor local branch. | 20:20 |
| kfogel | You should also push the branch up to launchpad, so others can see it. | 20:20 |
| kfogel | just type 'bzr push' inside the top level of your new branch | 20:21 |
| kfogel | bzr will do the right thing | 20:21 |
| kfogel | and you'll get a branch visible at some place like this: | 20:21 |
| kfogel | https://code.launchpad.net/~your_name_here/launchpad/123456-improve-fish | 20:21 |
| kfogel | Note the bug number in the branch name -- this is a convention. It can go at the beginning or the end, but it's very handy to have it in there. | 20:21 |
| kfogel | You should also link the branch *to* the bug. | 20:21 |
| kfogel | If you go to the branch page, there will be a link for that. Do it immediately, so anyone who | 20:22 |
| kfogel | visits the bug can see it is being worked on. | 20:22 |
| kfogel | You might also want to mark the bug's status as "in progress". | 20:22 |
| kfogel | In a moment, I'll digress into talking about what you actually do to develop. | 20:22 |
| kfogel | But first, let's assume you're done with your branch, and you want to submit it for review. | 20:23 |
| kfogel | First make a merge proposal, in the usual way (there's a link on the branch page for that). | 20:23 |
| kfogel | I recommend going into #launchpad-reviews and seeing who the on-call reviewer is | 20:23 |
| kfogel | and entering them as a reviewer, if they agree. Or if you're already talking to another lanuchpad developer about the change, they may be available to review it. | 20:23 |
| kfogel | If your change involves a DB schema change, then click "extra options" on the merge proposa page and put "db" for review type. | 20:24 |
| kfogel | If it involves visible UI changes, then put "ui" for review type. | 20:24 |
| kfogel | (You can submit to different reviewers for different types of reviews -- ask the developers you're working with who the appropriate people might be.) | 20:24 |
| kfogel | brief pause for questions... | 20:25 |
| kfogel | okay, continuing | 20:25 |
| kfogel | Launchpad development is very test-driven. | 20:25 |
| kfogel | Most changes of any size (except maybe for trivial typo fixes and stuff) need automated tests to accompany them. | 20:26 |
| kfogel | The testing system looks complicated, but don't worry, it's actually complicated. | 20:26 |
| kfogel | There. That was comforting, right? | 20:26 |
| kfogel | It's not so bad. There are three basic kinds of tests: doc tests, story tests, and windmill (javascript/ui) tests. | 20:26 |
| kfogel | Running "bin/test -v -t bug.txt" runs the doc test lib/lp/bugs/doc/bug.txt | 20:27 |
| kfogel | bin/test -v -v -t test_bugtask runs | 20:27 |
| kfogel | lib/lp/bugs/browser/tests/test_bugtask.py | 20:27 |
| kfogel | (for example again, and this is all from inside the top of "devel", by the way) | 20:28 |
| kfogel | and bin/test -v -v -t patches-view.txt | 20:28 |
| kfogel | runs tests lib/lp/bugs/stories/patches-view/patches-view.txt . | 20:28 |
| kfogel | I won't go into depth about how those tests work, since the best way to learn them is just to look at the examples there already. Basically, they are a way to test both the Python-level interfaces you may have added, and a way to test browser behavior (the story tests actually simulate a browser) | 20:29 |
| kfogel | for more on these, see lib/canonical/launchpad/pagetests/README.txt | 20:30 |
| kfogel | and lib/canonical/launchpad/pagetests/REFERENCE.txt | 20:30 |
| kfogel | By the way, since you run tests at the command line, the MOST USEFUL TRICK IN THE WORLD is this: | 20:30 |
| kfogel | in the code, you can do the standard python debugger thing: | 20:31 |
| kfogel | import pdb | 20:31 |
| kfogel | pdb.set_trace() | 20:31 |
| kfogel | (that's in a .py file somewhere) | 20:31 |
| kfogel | and then when you run your tests, when you get to that spot, you'll be right at an interactive debugger. | 20:31 |
| kfogel | It is an incredible timesaver. Highly recommended. | 20:31 |
| kfogel | another trick is, in the story tests, you'll see how when you show browser page contents, there is a certain output expected -- it's just the raw text (with HTML tags stripped out) from the page, with whitespace all folded | 20:32 |
| kfogel | So if you want to see the output you're actually getting, just put a bunch of blank lines where the output would be expected, and you'll get a nice diff showing what the browser put out instead | 20:33 |
| kfogel | Okay, now really on to some examples, which I hope will be good references. | 20:33 |
| kfogel | EXAMPLES | 20:33 |
| kfogel | I'm going to use two examples from code changes I happened to make recently. | 20:34 |
| kfogel | I think these are actually fairly clean examples of two types of changes you might find yourself having to make. | 20:34 |
| kfogel | The first is "What does making a change to the DB schema look like?" | 20:34 |
| kfogel | Here's a branch that makes a very small change to the schema (and it landed recently, I believe): | 20:34 |
| kfogel | https://code.edge.launchpad.net/~kfogel/launchpad/509194-506514-bug-heat-db-fixes | 20:34 |
| kfogel | Note the two bug numbers both in the branch name. | 20:34 |
| kfogel | (of course, if there had been many bugs, I would have not done that, or would have had an overarching master bug) | 20:35 |
| kfogel | I'll pause so people can take a look at that code. | 20:35 |
| kfogel | Note how the SQL file we changed was database/schema/patch-2207-27-0.sql. When the change first started, the file we added and edited was: | 20:36 |
| kfogel | database/schema/patch-2207-99-0.sql | 20:36 |
| kfogel | For complicated, historical reasons, DB changes come in distinct SQL files, and the "99" convention tells the DB merge person "fit this one in wherever you have the next slot". | 20:37 |
| kfogel | So by the end, when we merged, we were assigned "27". But when we were working, it was "99". | 20:37 |
| kfogel | ("we" is Abel Deuring and myself, btw -- he's adeuring in IRC) | 20:37 |
| kfogel | That change went into the db-devel branch, of course, and so our merge proposal was targeted to that. | 20:37 |
| kfogel | Regarding merge proposals, there are two good wiki links about them (thanks to ajmitch for these): | 20:38 |
| kfogel | https://dev.launchpad.net/PatchSubmission | 20:38 |
| kfogel | https://dev.launchpad.net/CoverLetters | 20:38 |
| kfogel | A "merge proposal cover letter" is the sort of summary description of the change, that helps the reviewer know what they're looking at and how to test it. | 20:39 |
| kfogel | It is important to give the reviewer the context they need to properly review the change. | 20:39 |
| kfogel | There will be some more about that in the next example, | 20:39 |
| kfogel | which is about making a UI change. | 20:39 |
| kfogel | This next example comes from https://bugs.edge.launchpad.net/malone/+bug/506018 | 20:40 |
| kfogel | which is about making a new "+patches" view available (you put "+patches" on the end of certain Launchpad URLs, and you get a listing of patches available for, say, that source package, or project, or whatever). | 20:40 |
| kfogel | The code branch is here: | 20:40 |
| kfogel | https://code.edge.launchpad.net/~kfogel/launchpad/506018-patch-report | 20:40 |
| kfogel | And the merge proposal is here: | 20:40 |
| kfogel | https://code.edge.launchpad.net/~kfogel/launchpad/506018-patch-report/+merge/18181 | 20:41 |
| kfogel | That's a terrific example of what you can expect from launchpad code review, by the way (above). | 20:41 |
| kfogel | Note how the merge proposal describes the change, makes it clear to the reviewer how to test it. | 20:41 |
| kfogel | And the bug itself contains a screenshot (http://launchpadlibrarian.net/38192596/screenshot-2010-01-21_rich_popup.png) so the reviewer knows what to look for. | 20:41 |
| kfogel | <PAUSE FOR QUESTIONS> | 20:42 |
| kfogel | Okay, so there are two more things I want to talk about. | 20:42 |
| kfogel | The first is, how does a change actually land? | 20:42 |
| kfogel | The second is just some standard Launchpad development "gotchas" that tend to bite newbies. Maybe I can save you some time by telling you about them now! | 20:43 |
| kfogel | HOW TO LAND A CHANGE | 20:43 |
| kfogel | Well, right now, once your branch has passed review (and note that review is something *all* Launchpad developers get, including full-time experts) | 20:44 |
| kfogel | an existing Launchpad developer has to land it. What they'll do is use the Bazaar PQM plugin: | 20:44 |
| kfogel | https://edge.launchpad.net/bzr-pqm | 20:44 |
| kfogel | and run a command like this: | 20:44 |
| kfogel | $ bzr pqm-submit -m "[r=REVIEWER_NICKNAME][ui=none] Description of change here..." | 20:44 |
| kfogel | inside a branch with your changes. | 20:45 |
| kfogel | Your commits will be landed, even though they were gated through by someone else (and by the PQM -- "Patch Queue Manager" system -- which does some build sanity checks before putting the change on trunk. | 20:45 |
| kfogel | ) | 20:45 |
| kfogel | After that, your name will show up at https://dev.launchpad.net/Contributions , within a day or so :-). | 20:46 |
| kfogel | To see what other people's submit messages and commit messages look like, I highly recommend running 'bzr log -n0 > log-n0.out' in one of the four trunk branches, say, "devel". | 20:46 |
| kfogel | It gives you a good sense for how development goes. | 20:46 |
| kfogel | Okay, I'll pause for a moment before launching into the final section of this class... | 20:47 |
| kfogel | any questions? | 20:47 |
| kfogel | Okay, it's time for... | 20:47 |
| kfogel | STANDARD LAUNCHPAD DEVELOPMENT GOTCHAS | 20:47 |
| kfogel | Watch out for 'make schema' -- it will blow away any test data you generated while using your instance. | 20:47 |
| kfogel | So how do you save previously-generated test data? Easy: 'make newsampledata' | 20:47 |
| kfogel | like this: | 20:48 |
| kfogel | $ pwd | 20:48 |
| kfogel | ...../lp-branches/devel/database/sampledata | 20:48 |
| kfogel | $ ls -l current* newsampledata* | 20:48 |
| kfogel | -rw-r--r-- 1 kfogel kfogel 1635602 2009-12-07 14:11 current-dev.sql | 20:48 |
| kfogel | -rw-r--r-- 1 kfogel kfogel 1583524 2010-01-18 09:01 current.sql | 20:48 |
| kfogel | -rw-r--r-- 1 kfogel kfogel 1712790 2010-01-18 09:54 newsampledata-dev.sql | 20:48 |
| kfogel | -rw-r--r-- 1 kfogel kfogel 1583524 2010-01-18 09:54 newsampledata.sql | 20:48 |
| kfogel | 20:48 | |
| kfogel | that file, newsampledata-dev.sql, contains the sample data. | 20:48 |
| kfogel | This is very useful for your reviewers, too! Save it, attach it where they can use it, so they don't have to generate this data themselves. | 20:48 |
| kfogel | 20:48 | |
| kfogel | Who are the test users to use in your "launchpad.dev" instance? | 20:49 |
| kfogel | log in as "test@canonical.com" for a standard test user (I think the password is "test") | 20:49 |
| kfogel | and log in as "foo.bar@canonical.com" for an admin-privs user (same pw I think) | 20:49 |
| kfogel | 20:49 | |
| kfogel | Hmm, I'll start numbering the tips now, so you know where the boundaries are. | 20:49 |
| kfogel | Tip 3: if things aren't working, you should check the "launchpad.log" file in the top level of your branch | 20:50 |
| kfogel | that is, when you do a "make run" and then visit the launchpad.dev instance in your browser, | 20:50 |
| kfogel | your code changes may or may not have the effect you expected. If they don't have the effect you expected, the answers are often to be found in launchpad.log. | 20:50 |
| kfogel | Also, /var/log/postgresql/postgresql-8.3-main.log has all your DB queries | 20:51 |
| kfogel | 20:51 | |
| kfogel | Tip 4: Visiting foo.launchpad.dev for the first time may bring up a cert failure thing. Just accept the cert. | 20:51 |
| kfogel | 20:51 | |
| kfogel | Tip 5: You don't need to restart on a template change (a .pt file). You DO need to restart on a .py file change. | 20:51 |
| kfogel | These templates are Zope templates, by the way. | 20:51 |
| kfogel | If you're new to Zope (I am!) then http://www.owlfish.com/software/simpleTAL/tal-guide.html is your friend. | 20:51 |
| kfogel | And we use the Yahoo User Interface (YUI) library for javascript. | 20:52 |
| kfogel | If you're new to that, see | 20:52 |
| kfogel | http://developer.yahoo.com/yui/3 and http://developer.yahoo.com/yui/3/api/module_event.html | 20:52 |
| kfogel | 20:52 | |
| kfogel | Tip 5: terminology. You may be used to seeing "project" to mean a software program. | 20:52 |
| kfogel | But inside the code, "project" means "project group", and "product" means what you normally think of as a project. E.g, firefox is a "product" object in the code. | 20:53 |
| kfogel | Watch out for confusion. Yes, we're sorry. No, it's too late to change it now - that would be a massive string-replace :-). | 20:53 |
| kfogel | 20:53 | |
| kfogel | Tip 6: use staging.launchpad.net to test behaviors. the database gets reset every night or so, so you can't hurt anything. | 20:53 |
| kfogel | 20:53 | |
| kfogel | Tip 7: most of your page UI will be defined by .pt (template) files, for example see lib/lp/bugs/templates/bugtarget-patches.pt | 20:54 |
| kfogel | (the new file added by that 506018 branch I referred to earlier). | 20:54 |
| kfogel | In a template file, "context/foo" accesses the model, whereas "view/foo" accesses the view. If you already know Zope, you probably know that. For the rest of us, it's a pretty basic thing. | 20:55 |
| kfogel | Also, in the code, such as in a View class (like "BugsPatchesView(LaunchpadView):"), self.context also accesses the model. | 20:55 |
| kfogel | 20:55 | |
| kfogel | Tip 8: we observe an "XXX" convention. If you leave an XXX comment in the code, meaning "this isn't quite right, but I'm going to leave it for now so I can just finish this feature", then the deal is, file a bug about the XXX, and go back and mark the XXX with your name, the date, and the bug number. | 20:56 |
| kfogel | For example: "# XXX Jill Random 2010-01-28 bug=123456 We really should use a database query here instead of looping over the objects in Python." | 20:56 |
| kfogel | 20:57 | |
| kfogel | QUESTION: what is, or what was the deal with open sourcing Launchpad, exactly? To the average user, and many experienced ones, Launchpad is not software, it's just a web page. | 20:57 |
| kfogel | 20:57 | |
| kfogel | Interesting question. | 20:57 |
| kfogel | Couple of reasons: | 20:57 |
| kfogel | One, many people become more willing to use Launchpad, and invest time & data into it, if they feel that they are not locked in -- that is, no matter what happens, the source code is open so it can't be lost. | 20:58 |
| kfogel | We heard from people that they didn't want to host their free software on a proprietary system. We did want them to use it, so we solved it by making the system not proprietary :-). | 20:58 |
| kfogel | The second reason is that, although it is a web service, it's one that has a very technically skilled user base, and there's no reason they shouldn't be able to fix problems when they see them. | 20:59 |
| kfogel | So we're doing something a bit unusual: forming a development community around a piece of software that runs as a web service, but forming the community around that actual service -- around Launchpad.net itself. | 20:59 |
| kfogel | The code is open, but we try to shepherd all changes onto the running service as quickly as we can. | 21:00 |
| kfogel | We have a monthly release cycle, and changes become visible on edge.launchpad.net even sooner for beta testers. | 21:00 |
| kfogel | QUESTION: why are there only 14 people outside Canonical with registered contributions? | 21:00 |
| kfogel | ANSWER: because one of them is William Grant, who seems to actually about 10 people himself? :-) | 21:00 |
| kfogel | No, seriously, I think the answer is that Launchpad is complex, and it takes people a while to get up to speed. | 21:01 |
| kfogel | It's not clear to me whether 14 is too high or too low. I hope it goes up, of course; but we didn't know what to expect when we open sourced. | 21:01 |
| kfogel | Remember also that we only open sourced six or seven months ago! | 21:01 |
| kfogel | I think that's it. | 21:01 |
| kfogel | I hope this helped those who might consider hacking on Launchpad. It's been a pleasure; find me in #launchpad-dev if ever have questions of course. | 21:02 |
| Nahsei | Hello | 23:45 |
| Nahsei | I have some questions... can someone help me please? | 23:45 |
Generated by irclog2html.py 2.7 by Marius Gedminas - find it at mg.pov.lt!
