=== BobJonkman is now known as BobJonkman_AFK
=== BobJonkman_AFK is now known as BobJonkman
=== MickE is now known as Guest34399
=== nigel is now known as nigelb
=== yofel_ is now known as yofel
=== raju1 is now known as raju
=== coalwater_ is now known as coalwater
=== [1]damageboy is now known as damageboy
=== raju1 is now known as raju
=== Pendulum_ is now known as Pendulum
thanos713_hi to everyone11:05
coalwaterhello thanos713_11:07
thanos713_When the next classrooom is going to be held?11:08
karthick87Can anyone help me with apt-cacher-ng setup?11:20
astraljavakarthick87: Support in #ubuntu, as per /topic12:55
dholbachHELLO EVERYBODY! Welcome to Day 3 of Ubuntu Developer Week!15:50
dholbachWe still have 10 minutes left until we start, but I just wanted to bring up a few organisational bits and pieces first:15:50
dholbach - If you can't make it to a session or missed one, logs to the sessions that already happened are linked from https://wiki.ubuntu.com/UbuntuDeveloperWeek15:51
dholbach - If you want to chat or ask questions, please make sure you also join #ubuntu-classroom-chat15:51
dholbach - If you ask questions, please prefix them with QUESTION:15:51
dholbach   ie: QUESTION: dpm, How hard was it to learn German?15:52
dholbach - if you are on twitter/identi.ca or facebook, follow @ubuntudev to get the latest development updates :)15:52
dpm(answer: difficult if you start learning in the Swabian dialect)15:52
dholbachha! :)15:52
dholbachthat's it from me15:53
dholbachyou still hvae 7 minutes until David "dpm" Planella will kick off today and talk about Launchpad Translations Sharing!15:53
dholbachHave a great day!15:53
=== Andre_Go` is now known as Andre_Gondim
=== ChanServ changed the topic of #ubuntu-classroom to: Welcome to the Ubuntu Classroom - https://wiki.ubuntu.com/Classroom || Support in #ubuntu || Upcoming Schedule: http://is.gd/8rtIi || Questions in #ubuntu-classroom-chat || Event: Ubuntu Developer Week - Current Session: Getting Translations Quicker into Launchpad: Upstream Imports Sharing - Instructors: dpm
ClassBotLogs for this session will be available at http://irclogs.ubuntu.com/2011/07/13/%23ubuntu-classroom.html following the conclusion of the session.16:01
dpmhey everyone!16:02
dpmwelcome all to the 3rd day of Ubuntu Developer Week16:03
dpmand welcome to this UDW talk about one of the coolest features of Launchpad Translations: upstream imports sharing16:03
dpmMy name is David Planella, and I work on the Community team at Canonical as the Ubuntu Translations Coordinator16:03
dpmAs part of my job, I get the pleasure of working with the awesome Ubuntu translation teams and with the equally awesome Launchpad developers16:04
dpmI also use Launchpad for translating Ubuntu on a regular basis, as part of my volunteer contribution to the project,16:04
dpmwhich is why I'm excited about this feature and I'd like to share it with you guys16:04
dpmSo, without further ado, let's roll!16:05
dpmoh, btw, I've set aside some time for questions at the end, but feel free to ask anything during the talk.16:05
dpmjust remember to prepend your questions with QUESTION: and ask them on the #ubuntu-classroom-chat channel16:05
dpm 16:06
dpmWhat is message sharing16:06
dpm 16:06
dpmBefore we delve into details, let's start with some basics: what's all this about?16:06
dpmIn short, message sharing is a unique feature in Launchpad with which translations for identical messages are essentially linked into one single message16:07
dpmThis means that as a translator, you just need to translate that message once and your translations will appear automatically in all other places where that message appears.16:07
dpmThe only requirements are that those messages are contained within a template with the same name and are part of different series of a distribution or project in Launchpad.16:08
dpmThis may sound a bit abstract, so let's have a look at an example:16:08
dpmLet's say you're translating Unity in Ubuntu Oneiric:16:08
dpm(you translate it from that URL in Launchpad)16:09
dpmAnd you translate the message "Quit" into your language16:09
dpmInstantly your translation will propagate to the Unity translation in _Natty_ (and all other releases):16:09
dpmSo it will appear translated here:16:10
dpmwithout you having had to actually translate it there16:10
dpmSo basically Launchpad is doing the work for you! :-)16:10
dpmIt also works when you want to do corrections:16:11
dpmLet's say you are not happy with the translation of "Quit" in Unity, and you change it in the Natty series in Launchpad16:12
dpmSo you go to: https://translations.launchpad.net/ubuntu/natty/+source/unity/+pots/unity/16:12
dpmand change it16:12
=== tavish_ is now known as tavish
dpmactually, I could point you to the actual message:16:13
dpmanyway, so you fix the "Quit" translation16:13
dpmperhaps there was a typo, perhaps you were not happy with the current translation, etc.16:13
dpmThat change will also be propagated to all other Ubuntu series, so that you don't have to manually fix each one of them16:15
dpmIsn't that cool?16:15
dpmAnd as you see, it works both ways: from older series to new ones, and viceversa. The order does not really matter at all.16:15
dpmwe've got a question:16:15
ClassBotshnatsel|busy asked: If there was a change in terminology between the series, e.g. I translated "torrent" with one word, then the app changed (some strings changed, some strings added) and now I use a different word, is there a way to prevent the new terminology partly leaking to the older series and making a terrible mess there?16:15
dpmthat's a good point16:16
dpmhowever, with the current implementation this will not happen16:16
dpmmessage sharing works only with identical messages16:16
dpmthis means that if in the first series the original string was "torrent",16:17
dpmand in the next series the original string was "torrent new",16:17
dpmthere won't be any message sharing at all, avoiding inadvertently translating strings you don't want to be translated automatically16:18
ClassBotgpc asked: So, "torrent" and "torrent." are not the same string?16:19
dpmthat's correct, even with such a small change as this, they're not considered the same string16:20
dpmthey need to be identical16:20
dpmso rather than a fuzzy match, message sharing works only on identical matches16:20
dpmfuzzy matching would need further development in Launchpad16:21
dpmIf you're interested on how difficult it would be to implement it, or even in implement it yourself,16:21
dpmI'd recommend asking on the #launchpad channel16:21
dpmthat's where the Launchpad developers hang out16:21
dpmand they're always happy to help16:22
dpmAnyway, let'w move on...16:22
dpmSo far we've only talked about distributions, and in particular Ubuntu. But this equally works within series of projects in Launchpad16:22
dpmBut more on that later on...16:22
dpmYou'll find out more about sharing here as well:16:22
dpm 16:23
dpmBenefits of messsage sharing16:23
dpmContinuing with the basics: what good is this for?16:23
dpmIn terms of Launchpad itself, it makes it much more effective to store the data: rather than storing many translations for a same message, only one needs to be stored.16:24
dpmBut most importantly:16:24
dpmFor project maintainers: when they upload a template to a new release series and it gets imported, will instantly get all existing translations from previous releases shared with the new series and translators won’t have to re-do their work. They won’t have to worry about uploading correct versions of translated PO files, and can just care about POT files instead.16:24
dpmFor translators: they no longer have to worry about back-porting translation fixes to older releases anymore, and they can simply translate the latest release: translations will automatically propagate to older releases. Also, this works both ways, so if you are translating a current stable release, newer development release will get those updates too!16:25
dpmAny other questions so far?16:26
dpmOk, let's continue then16:27
dpm 16:27
dpmWhat's new in message sharing16:27
dpmUntil recently, message sharing had only worked within the limits of a distribution or of a project.16:27
dpmThat is, messages could be shared amongst all the series in a distribution or amongst all series of a project.16:28
dpmAs cool as it already was, that was it: data could not flow across projects and distributions, and each one of these Launchpad entities behaved like isolated islands with regards to message sharing.16:28
dpmBut before going forward, let me recap quickly on some other basic concepts in Launchpad. When we're talking about message sharing, we're interested mostly in two types of Launchpad entities16:29
dpm* Projects: these are standalone projects whose translations are exposed in Launchpad. If these projects are packaged in a distribution, we often refer to the actual project at a location such as https://launchpad.net/some-project as to the upstream project. An example is the Chromium project at https://translations.launchpad.net/chromium-browser/. Upstream projects can host their translations in Launchpad or externally. In the latter case, translat16:30
dpmions can still be imported into Launchpad, but more on that later on16:30
dpm* Distributions: these are collections of source packages, each one of which is also exposed for translation. The most obvious example is Ubuntu. Here's an example of the Natty series of the Ubuntu distribution: https://translations.launchpad.net/ubuntu/natty16:31
dpmSo the news, and the main subject of this talk, is that from now on translations can be shared, given the right permissions, across projects and distributions.16:32
dpmAgain, let's take an example:16:32
dpm• The Synaptic project is translatable in Launchpad: https://translations.launchpad.net/synaptic16:32
dpm• At the same time, the Ubuntu distribution has got a Synaptic package which is translatable in Launchpad: https://translations.staging.launchpad.net/ubuntu/natty/+source/synaptic/16:33
dpm• Now, given that the upstream maintainer has enabled sharing and has set the right permissions, now translators can translate Synaptic in Ubuntu and their translations will seamlessly flow into the upstream project!16:33
dpm• This works again both ways: if one translates Synaptic in the upstream project, translations will appear in the Ubuntu distribution16:33
dpmSo no more backporting translations or exporting them and committing them back and forth.16:34
ClassBotashams asked: So why there is a separate set of Templates for each release of Ubuntu, i.e. (in Natty: https://translations.launchpad.net/ubuntu/natty/+source/unity/+pots/unity/) in Oneric: https://translations.launchpad.net/ubuntu/oneiric/+source/unity/+pots/unity/)16:35
dpmThis is due to the fact that for each series of Ubuntu you not only get new applications with new templates (or some go away), but also you get different messages in the templates16:36
dpmThis is how releases of projects work, it hasn't got anything to do with message sharing16:36
dpmi.e. we need different sets of templates for each series, otherwise if we had one single set, it will overwrite the old ones on each release16:37
dpmAnyway, combine the previous example with automatic translation imports and exports, and project maintainers get a fully automated translation workflow, which is really really awesome :-)16:38
dpmMore on automatic imports/exports:16:38
dpmSo far we've covered projects hosted in Launchpad - what happens with projects hosted externally?16:39
dpmIf translations of a given project are hosted externally, you won't get the benefit of full integration into Launchpad, but you'll still get some important advantages:16:39
dpm• Translations will need to be imported from a mirror branch into a Launchpad upstream project16:40
dpm• They will then be regularly imported to Launchpad Translations16:40
dpm• From there, they will flow quickly, and on a regular basis, into the Ubuntu distribution16:40
dpm• Up until here, the end result is the same as for upstream projects hosting translations in Launchpad16:40
dpm• However, translations will only flow in the direction upstream -> Ubuntu, as we don't have a way to automatically send translations to the external project yet16:40
dpm• The big benefit here is that translations will be imported reliably and quickly on a regular basis16:41
dpmFor an overview on message sharing across projects and distributions, check out this UDS presentation by Henning Eggers, one of the Launchpad developers who implemented this feature, and myself:16:42
dpm 16:43
dpmHow to enable message sharing16:43
dpmThe cool thing to know is that within a project or a distribution message sharing is already enabled16:43
dpmThere are no steps that the project maintainers need to follow: every new series will automatically share messages with all the others16:44
dpmHowever, if you want to share messages between an upstream project and a distribution (e.g. Ubuntu), there are a set of steps that need to be performed first:16:45
dpm* Enable translations in the upstream project, setting the right permissions and translations group16:45
dpm* If the project you want to enable sharing for is hosting translations externally, you'll need to request a bzr import, so that translations can get imported from an external branch16:46
dpm* Finally, you'll need to link the upstream project to the corresponding source package in Launchpad16:47
dpmRight now just a few projects and source packages are linked this way, but this cycle we're planning a community effort to enable sharing for all supported packages.16:47
dpmI've prepared a table with an overview of the supported packages here:16:48
dpmAnd will soon announce how the community can help in enabling these for sharing.16:48
dpmStay tuned to the Ubuntu translators list for more info:16:48
dpmOk, I think that's all I had on content, so let's go for questions!16:49
dpm 16:49
dpmQ & A16:49
ClassBotyurchor asked: Why does translation sharing behaves strange (diffs are really weird)? Ex.: https://translations.launchpad.net/ubuntu/natty/+source/avogadro/+pots/libavogadro/uk/+translate?show=new_suggestions16:50
ClassBotThere are 10 minutes remaining in the current session.16:51
dpmI think that particular case is a project in which there was some data that needed migration (i.e. Ubuntu translations exported and uploaded in the upstream package) and that did not happen.16:51
dpmI'd suggest pointing this out in the #launchpad channel, where the Launchpad devs can have a look at it in more detail16:51
ClassBotyurchor asked: What if the project does not have repository with translations (like Fedora's libvirt, etc. on Transifex, translations are generated at package creation)? What will be imported from upstream?16:52
dpmI'm not familiar with how translations are stored in Fedora's libvirt. The only layout that's supported for external repositories is PO files stored in an external version control system that can be imported as a bzr branch (e.g. git, mercurial, svn, etc)16:53
dpmQUESTION: For example, I'm translating a BitTorrent client. I had a translation of the term "torrent" that I used in all strings that contain it. Then a new major release arrived that has some strings added, some strings removed and some strings (like "New torrent") intact. For the new version, I find a better translation of the term "torrent", and change it in all those strings in the newer series. But then the new translation of "New torrent", wit16:54
dpmh the new term to describe "torrent" will leak to the older series, while some other strings in there still use the old term. How can I prevent it?16:54
dpmoh, I understand what you mean now16:55
ClassBotThere are 5 minutes remaining in the current session.16:55
dpmUnfortunately, there is no way to detect this in Launchpad, as there is no way to link the "New torrent" to the "torrent" messages16:56
dpmIn this particular case, one thing you can do is to export the translation as a PO file, and replace all the "New torrent" translations with the new term16:56
dpmand then reupload in Launchpad16:57
dpmOne thing I forgot to mention, and it might be useful if you want to keep the old terminology in older series,16:57
dpmis that you can explicitly choose individual messages to diverge16:57
dpmso you can call "torrent" as "a" in a series and "b" in another series, bypassing message sharing16:58
dpmOk, I think there is not much time for more questions, so we can probably wrap it up here16:59
dpmIf you've got other questions, feel free to ask me on the #ubuntu-translators channel on Freenode16:59
dpmSo thank you everyone for listening and for participating with your questions16:59
dpmI hope you enjoyed the talk and see you soon!17:00
=== ChanServ changed the topic of #ubuntu-classroom to: Welcome to the Ubuntu Classroom - https://wiki.ubuntu.com/Classroom || Support in #ubuntu || Upcoming Schedule: http://is.gd/8rtIi || Questions in #ubuntu-classroom-chat || Event: Ubuntu Developer Week - Current Session: Debugging the Kernel - Instructors: jjohansen
ClassBotLogs for this session will be available at http://irclogs.ubuntu.com/2011/07/13/%23ubuntu-classroom.html following the conclusion of the session.17:01
jjohansenHi, before we start I figured I would introduce myself,  I am John Johansen an Ubuntu contributor and member of the Kernel team17:01
jjohansenFeel free to ask questions through out in he #ubuntu-classroom-chat channel, and remember to prepend QUESTION: to your question17:01
jjohansenSo lets get started17:01
jjohansenDebugging Kernels is a vast topic, and there is lots of documentation out there.  Since we only have an hour I thought I would cover a few thing specific to the Ubuntu kernel.  So perhaps the topic really should have been changed to debugging the Ubuntu kernel.  I am not going to walk through a live example as we don't have time for multiple kernel builds, installing and testing, working with the kernel takes time.17:01
jjohansenFirst up is the all important documentation link https://wiki.ubuntu.com/Kernel17:02
jjohansenIt has a lot of useful information buried in its links.17:02
jjohansenit takes a while to read through but its really worth doing if you are interested in the kernel17:02
jjohansenThe Ubuntu kernels are available from git://kernel.ubuntu.com/ubuntu/ubuntu-<release>.git17:02
jjohansenie. for natty you would do17:02
jjohansen  git clone git://kernel.ubuntu.com/ubuntu/ubuntu-natty.git17:02
jjohansengives the full details if you need more17:03
jjohansenThe Ubuntu kernel uses debian packaging and fakeroot to build, so you will need to pull in some dependencies17:03
jjohansen  sudo apt-get build-dep linux-image-$(uname -r)17:03
jjohansenonce you have the kernel you can change directory into the tree and build it17:04
jjohansen  fakeroot debian/rules clean17:04
jjohansen  fakeroot debian/rules binary-headers binary-generic17:04
jjohansenNote, a couple of things, 1. the clean must be done before your first attempt at building, its sets up some of the environment.  2 we are building the one kernel flavour in the above example, and the end result should be some debs.17:04
jjohansenalso if you ever see me use fdr its an alias I set up for fakeroot debian/rules, so I can get a way with doing17:04
jjohansen  fdr clean17:04
jjohansenit saves on typing, I'll try not to slip up but just incase ...17:04
jjohansenEveryone good so far?17:04
jjohansenAlright onto bisecting17:05
jjohansenWe need to cover a little more information about how the Ubuntu kernel is put together.  Each release of the Ubuntu linux kernel is based on some version of the upstream kernel.  The Ubuntu kernel carries a set of patches on top of the upstream linux kernel.  The patches really can be broken down into 3 catagories, packaging/build and configs, features/drivers that are not upstream (see the ubuntu directory for most of t17:05
jjohansenDuring the development cycle, the Ubuntu kernel is rebased on top of the current linux kernel, as the linux kernel is updated so is the development Ubuntu kernel; it occasionally gets rebased on top of newly imported linux kernels.  This means a couple of things patches that have been merged upstream get dropped, and that the Ubuntu patches stay at the top of the stack.17:05
jjohansenDuring the development cycle we hit a point called kernel freeze, where we stop rebasing on upstream, and from this point forward only bug fixes are taken with commits going on top.17:06
jjohansenSo why mention all of this?  Because it greatly affects how we can do kernel bisecting.  If a regression occurs after a kernel is released (say the natty kernel), and there is a known good natty kernel, then bisecting is relatively easy.  We can just checkout the natty kernel and start a git bisect between the released kernel tags.17:06
jjohansenHowever bisecting between development releases or different released versions (say maverick and natty) of the kernel becomes much more difficult.  This is because there is no continuity because of the rebasing, so bisect doesn't work correctly, and if you are lucky and have continuity the bisecting may remove the packaging patches.17:06
jjohansenSo how do you bisect bugs in the Ubuntu kernel then?  We use the upstream kernel of course :)17:07
jjohansenThere are two ways to do this, the standard upstream build route and using a trick to get debian packages.17:07
jjohansenThe upstream route is good if you are just doing local builds and not distributing the kernels17:07
jjohansenbut if you want other people to install your kernels you are probably best of using the debian packaging route17:08
jjohansenSo the trick to get debian packaging is pretty simple17:08
jjohansenyou checkout the upstream kernel17:09
jjohansencheckout an Ubuntu kernel17:09
jjohansenidentify a good and bad upstream kernel17:09
jjohansenyou can do this by using the ubuntu mainline kernel builds available from the kernel team ppa17:10
jjohansenthat saves you from having to build kernel17:11
jjohansennow copy the debian/ and debian.master/ directories from the ubuntu kernel into the upstream kernel17:12
jjohansenyou do not want to commit these17:12
jjohansenas that will just make them disappear with the bisect17:12
jjohansenyou can the change directory into the upstream kernel17:12
jjohansenedit debian.master/changelog, the top of the file should be something like17:12
jjohansen  linux (2.6.38-10.44) natty-proposed; urgency=low17:12
jjohansenyou want to change the version to something that will have meaning to you17:14
jjohansenand that can be easily replaced by newer kernels17:14
jjohansenyou use a debian packaging trick to do this17:14
jjohansenchange 2.6.38-10.44 to something like 2.6.38-10.44~jjLP645123.117:15
jjohansenthe jj indicates me, then the launchpad bug number and I like to use a .X to indicate how far into the bisect17:16
jjohansenyou can use what ever make sense to you17:16
jjohansenthe important part is the ~ which allows kernels with higher version numbers to install over the bisect kernel without breaking things17:17
jjohansenif you are going to upload the kernel to a ppa you will also want to update the release info17:18
jjohansenie. natty-proposed in this example17:18
jjohansenif you are using the current dev kernel it will say UNRELEASED and you need to specify a release pocket17:19
jjohansenie. natty, maverick, ...17:19
=== Cas071 is now known as Cas
jjohansenhowever ppa builds are slow and I just about never use them, at least not for regular bug testing17:19
=== Cas is now known as Cas07
jjohansennow you can build the kernel17:20
jjohansen  fakeroot debian/rules clean17:20
jjohansen  fakeroot debian/rules binary-headers binary-generic17:20
jjohansenthis will churn through and should build some .debs that can be installed using17:21
jjohansen  dpkg -i17:21
jjohansennow on to doing the actual bisect17:21
jjohansenso bisecting is basically a binary search, start with a known good point and bad, cut the commits in half, build a kernel test if its good, rinse lather, and repeat17:23
jjohansengit bisect is just a tool to help you do this17:23
jjohansenit is smarter than just doing the cut in half, it actually takes merges and other things into account17:24
jjohansenthe one important thing to note, for these bisects is if you look at the git log etc, you may find yourself in kernel versions outside of your bisect range17:24
jjohansenthis is because of how merges are handled, don't worry about it, git bisect will handle it for you17:25
jjohansenso the basics of git bisect are17:25
jjohansengit bisect start <bad> <good>17:26
jjohansenwhere bad is the bad kernel and good is the good kernel17:26
jjohansenthe problem is how do you know which kernel versions to use17:26
jjohansenif you are using the upstream kernels for a bisect then the ubuntu version tags are not available to you17:27
jjohansenyou need to use either a commit sha, or tag in the upstream tree17:27
jjohansensorry lost my internet there for a minute17:29
jjohansenif you used the mainline kernel builds to find a good and bad point then you can just use the kernel tags for those, ie. v2.6.3617:31
jjohansenif you used and ubuntu kernel then you can find a mapping of kernel versions here17:31
jjohansenalright so I was asked what the debian.master directory is about17:32
jjohansenin the Ubuntu kernel we have two directories to handle the debian packaging17:32
jjohansendebian and debian.master/17:32
jjohansenthis allows abstracting out parts of the packaging17:33
jjohansenwhen you setup a build, the parts of debian.master/ are copied into debian/ and that is used17:34
jjohansenthe difference between the two isn't terribly important for most people, think of debian as the working directory for the packaging, and master as the reference17:35
jjohansenwhen I had you edit debian.master/changelog above I could have changed things around and had you edit debian/changelog17:36
jjohansenfakeroot debian/rules clean17:36
jjohansenwill endup copying debian.master/changelog into debian/17:36
jjohansenthus if you change debian/change log you have to do a full edit on it every time you do a clean17:37
jjohansenso if you are editing debian/ you do17:37
jjohansen  fdr clean17:37
jjohansen  edit debian/changelog17:37
jjohansenwhich is the reversion of doing it to debian.master/changelog17:38
jjohansen  edit debian.master/changelog17:38
jjohansen  fdr clean17:38
jjohansenfor me editing debian.master/changelog keeps me from making a mistake and building a kernel with out my edits to the kernel version17:38
jjohansenhopefully that is enough info on the debian/ and debian.master/ for now17:40
jjohansenand we will jump back to bisecting for a little longer17:40
jjohansenso assuming you have your kernel version for good and bad you start, your bisection17:41
jjohansengit will put you on a commit roughly in the middle17:41
jjohansenthen you can do17:41
jjohansen  fdr clean17:41
jjohansen  fakeroot debian/rules binary-headers binary-generic17:42
jjohansensorry caught myself using fdr17:42
jjohansenthis will build your kernel you can install and test17:42
jjohansenand then you can input your info into git bisect17:43
jjohansen  git bisect good17:43
jjohansen  git bisect bad17:43
jjohansenthe import thing to remember is to not, commit any of your changes to git17:43
jjohansenI tend to edit the debian.master/changelog and update the .# at the end of my version string every iteration of the bisection17:44
jjohansenyou don't have to do this17:44
jjohansenyou can get away with just rebuilding, straight or if you want doing a partial build17:45
jjohansenthe partial build is a nice trick if you don't have a monster build machine but it doesn't save you anything early on in the bisect, when git is jumping lots of commits and lots of files are getting updated17:46
jjohansenthe trick to doing a partial build in the Ubuntu build system is removing the stamp file17:46
jjohansenwhen the kernel is built there are some stamp files generated and placed in17:47
jjohansen  debian/stamps/17:47
jjohansenthere is one for prepare and one for the actual build17:47
jjohansenif you build a kernel, and the build stamp file is around, starting a new build will just use the build that already exists and package it into a .deb17:48
jjohansenyou don't want to do this17:48
jjohansenso after you have stepped you git bisect (git bisect good/bad)17:49
jjohansen  rm debian/stamps/stamp-build-generic17:49
jjohansenthis will cause the build system to try building the kernel again, and make will use its timestamp dependencies to determine what needs to get rebuilt17:50
ClassBotThere are 10 minutes remaining in the current session.17:51
jjohansenif the bisect is only stepping within a driver or subsystem this can save you a log of time on your builds, however if the bisect updates lots of files (moves lots of commits) or updates some common includes, you are going to end up doing a full kernel build17:51
jjohansenso now for the other tack, what do you do if you don't want to mess with the .deb build and just want to build the kernel old fashioned way17:52
jjohansenwell you build as you are familiar with.17:52
jjohansen  make17:52
jjohansen  make install17:52
jjohansen  make modules_install17:52
jjohansenthen you need to create a ramdisk, and update grub17:53
=== AndroUser is now known as daemonTutorials
jjohansen  sudo update-initramfs -c k <kernelversion>17:54
jjohansenwill create the ram disk you need if you don't want to mess with the kernel version17:54
jjohansen  sudo update-initramfs -c -k all17:54
jjohansenthen you can do17:54
jjohansen  sudo update-grub17:55
jjohansenand you are done17:55
jjohansenso QUESTION: After rm debian/stamps/stamp-build-generic do you still do a fakeroot debian/rules clean when doing the incremental build?17:55
jjohansenthe answer is no17:55
ClassBotThere are 5 minutes remaining in the current session.17:55
jjohansendoing a clean will remove all the stamp files, and remove your .o files which will cause a full build to happen17:56
jjohansenso with only a couple minutes left I am not going to jump into a new topic but will mention something I neglected about about the Ubuntu debian builds17:57
jjohansenour build system has some extra checks for expected abi, configs, and modules17:58
jjohansenwhen building against an upstream kernel you will want to turn these off17:58
jjohansenyou can do this by setting some variables on the command line17:58
jjohansen  fakeroot debian/rules binary-headers binary-generic17:58
jjohansen  skipabi=true skipconfig=true skipmodule=true fakeroot debian/rules binary-headers binary-generic17:59
jjohansenthis can also be used when tuning your own configs etc/17:59
jjohansenI think I will stop there18:00
jjohansenthanks for attending, drop by #ubuntu-kernel if you have any questions18:00
=== ChanServ changed the topic of #ubuntu-classroom to: Welcome to the Ubuntu Classroom - https://wiki.ubuntu.com/Classroom || Support in #ubuntu || Upcoming Schedule: http://is.gd/8rtIi || Questions in #ubuntu-classroom-chat || Event: Ubuntu Developer Week - Current Session: dotdee - break a flat file into dynamically assembled snippets - Instructors: kirkland
ClassBotLogs for this session will be available at http://irclogs.ubuntu.com/2011/07/13/%23ubuntu-classroom.html following the conclusion of the session.18:01
kirklandhowdy all18:01
kirklandthis session is on dotdee18:02
kirklandthere will be a live streamed demo at: http://bit.ly/uclass18:02
kirklandi invite you to join me there18:02
kirklandthe username and password is guest/guest18:02
kirklandfor those interested, this is a tool called ajaxterm18:03
kirklandwhich embeds a terminal in a web browser18:03
kirklandi've set up a byobu/screen session for the guest user (which is readonly for all of you)18:03
kirklandI'll drive the demo18:03
kirklandand annotate it here18:04
kirklandalternatively, you can ssh guest@ec2-50-19-128-105.compute-1.amazonaws.com18:04
kirklandwith password guest18:04
kirklandokay, on to dotdee :-)18:05
kirklandif you've ever configured a Linux/UNIX system, you're probably familiar with the /etc directory18:05
kirklandand inside of /etc, there are many directories that end in a ".d"18:05
kirklandwatch the terminal while I find a few in /etc18:05
kirklandthere's a bunch!18:06
kirklandthis is a very user friendly way of offering configuration to users18:06
kirklandusually, files in a .d directory are concatenated, or executed sequentially18:06
kirklandit give users quite a bit of flexibility for editing or adding configurations18:07
kirklandto some software or service18:07
kirklandit also helps developers and packagers of software18:07
kirklandas it often allows them to drop snippets of configuration into place18:07
kirklandbut not every configuration file is setup in this way18:07
kirklandin fact, most of them are really quite "flat"18:08
kirklanda few months ago, I found myself repeatedly needing to converted some flat configuration files18:08
kirklandto .d style ones18:08
kirklandthis was for software I was working with, as a developer/packager18:09
kirklandbut not software that I had written myself18:09
kirklandideally, I would just ask the upstream developers to change their flat .conf file18:09
kirklandto a .d directory18:09
kirklandand they would magically do it18:09
kirklandand test it18:09
kirklandand release it18:09
kirklandimmediately :-)18:09
kirklandthat rarely happens though :-P18:10
kirklandso I wrote a little tool that would generically to that for me!18:10
kirklandand that tool is called "dotdee"18:10
kirklandso let's take a look!18:10
kirklandover in the terminal, i'm going to install dotdee, which is already in Ubuntu oneiric18:10
kirklandsudo apt-get install dotdee18:10
kirklandfor older ubuntu distros, you can 'sudo apt-add-repository ppa:dotdee/ppa'18:11
kirklandand update, and install it there too18:11
kirklandcool, so now i have a /usr/sbin/dotdee executable18:11
kirklandlet's take a flat file, and turn it into a dotdee directory!18:11
kirklandI'm going to use /etc/hosts as my example18:12
kirklandfirst, I need to "setup" the file18:12
kirklandi can first verify that /etc/hosts is in fact a flat file,18:12
kirkland-rw-r--r-- 1 root root 296 2011-07-13 12:14 /etc/hosts18:12
kirklandI'm going to set it up like this:18:13
kirklandsudo dotdee --setup /etc/hosts18:13
kirklandINFO: [/etc/hosts] updated by dotdee18:13
kirklandnow let's look at what dotdee did....18:13
kirkland$ ll /etc/hosts18:13
kirklandlrwxrwxrwx 1 root root 27 2011-07-13 18:13 /etc/hosts -> /etc/alternatives/etc:hosts18:13
kirklandso /etc/hosts is now a symlink18:13
kirklandpointing to an alternatives link18:13
kirkland$ ll /etc/alternatives/etc:hosts18:14
kirklandlrwxrwxrwx 1 root root 22 2011-07-13 18:13 /etc/alternatives/etc:hosts -> /etc/dotdee//etc/hosts18:14
kirklandwhich is pointing to a flat file in /etc/dotdee18:14
kirkland$ ll /etc/dotdee//etc/hosts18:14
kirkland-r--r--r-- 1 root root 296 2011-07-13 18:13 /etc/dotdee//etc/hosts18:14
kirklandif I look at the contents of that file, I see exactly what I had before18:14
kirklandbut let's go to that directory, /etc/dotdee18:15
kirklandinside of /etc/dotdee, there is a file structure that mirrors the same file structure on the system18:15
kirklandimportantly, we now have a .d directory18:15
kirklandthat refers exclusively to our newly managed file18:15
=== daker is now known as daker_
kirklandnamely, /etc/dotdee/etc/hosts.d18:16
kirklandin that directory, all we have is a file called 50-original18:16
kirklandwhich is the original contents of our /etc/hosts18:16
kirklandbut let's say we want to "append" a host to that fie18:16
kirklandlet's create a new file in this directory18:16
kirklandand call it 60-googledns18:16
kirklandso I edit the new file (as root)18:17
kirklandadd the entry, googledns18:17
kirklandand I'm going to write the file18:17
kirklandbut before i write the file, let me split the screen18:17
kirklandso that we can watch it get updated, automatically, in real time!18:18
kirklandso i ran 'watch -n1 cat /etc/hosts'18:18
kirklandwhich is just printing that file every 1 second18:18
kirklandnow i'm going to save our 60-googledns file18:18
kirklandand voila!18:18
kirklandwe have a new entry appended to our /etc/hosts18:19
kirklandthrough the magic of inotify :-)18:19
kirklandwhich is a daemon that monitors filesystem events18:19
kirklanddotdee comes with a configuration (which is dotdee managed, of course!) that adds and removes patterns18:19
kirklandas you setup/remove dotdee management18:19
kirklandlet's prepend a host18:20
kirklandwe'll call this one 40-foo18:20
kirklandand see it land at the beginning of the file18:20
kirklandnow it's at the beginning of our /etc/hosts18:20
kirklandso adding/removing a flat text file is one way of affecting our managed file18:22
kirklandflat text files are just appended or prepended, based on its alpha-numeric positioning18:23
kirklandbut there are 2 other ways as well!18:23
kirklandyou can also put executables in this .d directory18:23
kirklandwhich operate on the standard in and out18:23
=== Cas07 is now known as cas07
kirklandif you want to modify a flat file by "processing" it18:23
kirklandfor instance18:24
kirklandlet's make this file all uppercase18:24
kirklandokay, there we go18:25
kirklandwhich brings me to the --update command :-)18:25
kirklanddotdee --update can be called against any managed file18:26
kirklandto update it immediately18:26
kirklandin case the inotify bit didn't pick up the change18:26
kirklandin any case, i just did that, and now our /etc/hosts is all uppercase18:26
kirklandbecause of our 70-uppercase executable18:26
kirklandwhat happens if we move it from 70-uppercase to 10-uppercase?18:27
kirklandor, rather, how about 51-uppercase?18:27
kirklandsee the output now18:27
kirklandnote that 51-uppercase was applied against the "current state" of the output, as of position 5118:28
kirklandbut 60- was applied afterward18:28
kirklandso it wasn't affected18:28
kirklandso that's two ways we can affect the contents of the file18:28
kirklanda) flat text files, b) scripts that process stdin and write to stdout18:28
kirklandthe third way is patches or diff files18:29
kirklandgiven that this is a developer audience, we're probably familiar with quilt18:29
kirklandand directories of patches18:29
kirklandthis is particularly useful if you need to do some 'surgery' on a file18:29
kirklandlet's say I want to "insert" a line into the middle of this file18:29
kirklandinto the middle of 50-original, for instance18:30
kirklandokay, so i've added "        hello-there" to the middle of a copy of this file18:31
kirklandand i'm going to use diff -up to generate a patch18:31
kirklandthere we go18:31
kirklandokay, let's put that in this .d dir18:31
kirklandnote that I have to add .patch or .diff as the file extension18:32
kirklandand now i can cat /etc/hosts and see that the patch has been applied!18:32
kirklandi could stack a great number of these patches here18:33
kirklandmuch like a quilt directory18:33
kirklandokay, so now let's undo this configuration18:33
kirklandoh, first18:34
kirklandsudo dotdee --list /etc/hosts18:34
kirkland$ echo $?18:34
kirklandthis verifies that /etc/hosts is in fact dotdee managed18:34
kirklandif i try this against some other file18:34
kirkland$ sudo dotdee --list /boot/vmlinuz-3.0-3-virtual18:35
kirklandERROR: [/boot/vmlinuz-3.0-3-virtual] is not managed by dotdee18:35
kirkland(I don't recommend dotdee'ing your kernel :-)18:35
kirklandbut we can undo our /etc/hosts18:35
kirkland$ sudo dotdee --undo /etc/hosts18:35
kirklandupdate-alternatives: using /etc/dotdee//etc/hosts.d/50-original to provide /etc/hosts (etc:hosts) in auto mode.18:35
kirklandINFO: [/etc/hosts] has been restored18:35
kirklandINFO: You may want to manually remove [/etc/dotdee//etc/hosts /etc/dotdee//etc/hosts.d]18:35
kirklandand now our /etc/hosts is back to being whatever we saved in 50-original18:36
kirklandso ...18:36
kirklandthat's how it works18:36
kirklandand the /etc/hosts example is only marginally useful18:36
kirklandwhat I would *really* like to use it for is in configuration file management in Debian/Ubuntu packaging18:37
kirklandin the case where the upstream daemon or utility has a single, flat .conf file18:37
kirklandbut I really would prefer it to be a .d directory18:37
kirklandso i just did a find on /etc18:37
kirklandsudo find /etc/ -type f -name "*.conf"18:38
kirklandand chose one at random18:38
kirkland /etc/fonts/fonts.conf18:38
kirklandwhich happens to be XML18:38
kirklandand I just thought about this18:38
kirklandi should have mentioned it in our previous section18:38
kirklandXML is tougher than a linear file, like a shell script18:39
kirklandin that you can't just append, or prepend text18:39
kirklandyou have to surgically insert the bits you want18:39
kirklandin which case the latter two methods I mentioned, the executable and the diff/patch will be your friend!18:39
kirklandnow I would *really* like to see dpkg learn just a little bit about dotdee18:40
kirklandI'd like for it to be able to determine *if* a file is managed by dotdee18:41
kirkland(easy to check using dotdee --list, or just checking if the file is a symlink itself)18:41
kirklandand if so, then it would use $(dotdee --original ...) to find the 50-original file path18:41
kirklandand dpkg would write its changes to that location (the 50-original file)18:42
kirklandsuch that local admin, or even other packages, could dabble in the .d directory, without causing conffile conflicts or .dpkg-original files18:42
kirklandokay, anyway, let's take a break for questions18:43
kirklandI think I've demo'd most of what I'd like to show you18:43
kirklandany questions?18:43
kirkland<coalitians> Question: Is there any issues when you upgrade or update due to dotdee?18:44
kirklandcoalitians: great question !18:44
kirklandcoalitians: right, so that's what I was saying about dpkg needing to "learn" about dotdee18:44
kirklandcoalitians: let's look at an example over in our test system18:44
kirklandcoalitians: I'm going to dotdee --setup that font xml18:45
kirklandcoalitians: okay, as you can see, i've made a change18:47
kirklandcoalitians: let's upgrade (or reinstall) the package that owns this file18:47
kirkland$ sudo apt-get install --reinstall fontconfig-config18:47
kirklandlrwxrwxrwx  1 root root   38 2011-07-13 18:45 fonts.conf -> /etc/alternatives/etc:fonts:fonts.conf18:47
kirkland-rw-r--r--  1 root root 5287 2011-07-01 12:12 fonts.conf.dpkg-new18:47
kirklandunfortunately, dpkg dump fonts.conf.dpkg-new here :-(18:48
kirklandi have toyed with another inotify/iwatch regex that would look for these :-)18:48
kirklandslurp them up, and move them over to 50-original18:48
kirklandwhich works reasonably well18:48
kirklandexcept that I don't yet have the interface for the merge/conffile questions, like dpkg does18:49
kirklandcoalitians: so to answer your question, upgrades are not yet handled terribly gracefully18:49
kirklandcoalitians: and would take a little work within dpkg itself18:49
kirklandcoalitians: sorry;  but great question!18:49
kirklandany others?18:49
kirklandI see roughly 19# in the byobu session :-)18:50
kirklandthat's what the red-on-white 19# means18:50
kirkland19 ssh sessions18:50
kirklanddid the web interface work for anyone?18:50
kirklandthis is the first time I've tried it18:50
ClassBotThere are 10 minutes remaining in the current session.18:51
zygaoh, already? :-)18:51
zygakirkland, is your session over now?18:51
kirklandokay, I reckon my session is over18:51
kirklandno more questions18:51
kirklandzyga: sure, you can have it ;-)18:51
kirklandthanks all18:51
zygaawesome, thanks18:51
ClassBotThere are 5 minutes remaining in the current session.18:55
=== ChanServ changed the topic of #ubuntu-classroom to: Welcome to the Ubuntu Classroom - https://wiki.ubuntu.com/Classroom || Support in #ubuntu || Upcoming Schedule: http://is.gd/8rtIi || Questions in #ubuntu-classroom-chat || Event: Ubuntu Developer Week - Current Session: Introduction to LAVA - Instructors: zyga
ClassBotLogs for this session will be available at http://irclogs.ubuntu.com/2011/07/13/%23ubuntu-classroom.html following the conclusion of the session.19:01
zygawelcome everyone :)19:01
zygaI'm glad to be able to tell you something about LAVA today19:01
zygamy name is Zygmunt Krynicki, I'm one of the developers working on the project19:02
zygafeel free to ask questions at any time, check the topic for instructions on how to do so19:02
zygalet's get started19:02
zygaSo first off, what is LAVA?19:02
zygaLAVA is an umbrella project, created by Linaro, that focuses on overall quality automation19:03
zygayou can think of it as a growing collection of small, focused projects that work well together19:04
zygathe overall goal of LAVA is to improve quality that developers perceive while working on ARM-based platforms19:05
zygawe're trying to do that by building tools that can be adopted by third party developers and Linaro members alike19:05
zygaAs I mentioned earlier LAVA is a collection of projects, I'd like to enumerate the most important ones that exist now19:06
zygafirst of all all of our projects can be roughly grouped into two bins "server side" and "client side"19:06
zygawhere client is either client of the "server" or a unrelated non-backend computer (like your workstation or a device being tested)19:07
zygathe key project on the server is called lava-server, it acts as a entry point for all other server side projects19:08
zygaessentially it's an extensible application container that simplifies other projects19:08
zyganext up we have lava-dashboard - a test result repository with data mining and reporting19:09
zygalava-scheduler - scheduler for "jobs" for lava-dispatcher19:09
zygalava-dispatcher - automated deployment, environment control tool that can remotely run tests on ubuntu and android images19:09
zygathe last one is really important and is getting a lot of focus recently19:10
zygaessentially it's something that knows how to control physical devices so that you can do automated image deployment, monitoring and recovery on real hardware19:10
zygaon the client side we have a similar list:19:11
zygalava-tool is a generic wrapper for other client side tools, it allows you to interact with server side components using command line instead of the raw API exposed by our services19:11
zygalava-dashboard-tool talks to the dashboard API19:11
zygalava-scheduler-tool talks to the scheduler API19:12
zygaand most important of all: lava-test, it's a little bit different as it is primarily an "offline" component19:12
zygait's a wrapper framework for running tests of any kind and processing the results in a way that lava-dashboard can consume19:12
zygaall of those projects are full of developer friendly APIs that allow for easy customization and extensions19:13
zygawe use the very same tools to build additional features19:13
zygalava-test is also important because it is a growing collection of wrapper for existing tests19:13
zygausing our APIs you can easily wrap your test code so that the test can be automated and processed in our stack19:14
zygasome test definitions are shipped in the code of lava-test but more and more are using the out-of-tree API to register tests from 3rd party packages19:14
zygaif you are an application author you could easily expose your test suite to lava this way19:15
zygathat's the general overview19:15
zyganow for two more things:19:15
zyga1) what can LAVA give you today19:15
zyga2) how can you help us if you are interested19:15
zygaWhile most of our focus is not what typical application developers would find interesting (arm? automation? testing? ;-)19:16
zygasome things are quite useful for a general audience19:16
zygayou can use lava-server + lava-dashboard to trivially deploy a private test result repository19:16
zygathe dashboard has a very powerful data system, you could store crash reports, user-submitted benchmark measurements, results from CI systems that track your development trees19:17
zygain general anything that you'd like to retain for data mining and reporting that you (perhaps) currently store in a custom solution that you need to maintain yourself19:18
zygaall of lava releases are packaged in our ppa (ppa:linaro-validation/ppa) and can be installed on ubuntu lucid+ with a single command19:18
zygathe next thing you could use is our various API layers: you could integrate some test/benchmark code in your application and allow users to submit this data to your central repository for analysis19:19
zygaif you are really into testing you could wrap your tests in lava-test and benefit from the huge automation effort that we bring with the lava-dispatcher19:20
zygain general, as soon as testing matters to you and you are looking for a toolkit please consider what we offer and how that might solve your needs19:21
zygaduring this six month cycle a few interesting thins are planned to land19:22
zygafirst of all: end user and developer documentation19:22
zygaoverview of lava project, various stack layers, APIs and examples19:23
ClassBotjykae asked: any projects that use successfully lava tools?19:23
zygajykae, linaro is our primary consumer at this time but ubuntu QA is looking at what we produce in hope for alignment19:23
zygajykae, next big users are ARM vendors (all the founding members of linaro) that use lava daily and contribute to various pieces19:24
zygajykae, finally I know of one big user, also from the ARM space, expect some nice announcement from them soon - they are really rocking (with what they do with LAVA and in general)19:25
zygajykae, but I hope to build LAVA in a way that _any_ developer can just deploy and start using, like a bug tracker that virtually all pet projects have nowdays19:25
zygajykae, we need more users and we will gladly help them get started19:25
zygaok, back to the "stuff coming this cycle"19:26
zygaso documentation is the number one thing19:26
zygaanother thing in the pipe is email notification for test failures and benchmark regressions19:26
zygathis will probably land next month19:26
zygawe are also looking at better data mining / reporting features, currently it's quite hard to use this feature, this will be somewhat improved with good documentation but we still think it can be more accessible19:27
zygathe goal is to deliver a small IDE that allows users to do data mining and reporting straight from their browsers19:27
zygathis is a big topic but small parts of it will land before 11.1019:27
zygafinally we are looking at some build automation features so that LAVA can help you out as a CI system19:28
zygaand of course: more tests wrapper in lava-test, more automaton (arm boards, perhaps x86)19:29
ClassBotjykae asked: do you have irc channel for lava?19:29
zygajykae, yes, we use #linaro for all lava talks19:29
zygajykae, a lot of people there know about it or use it and can help people out19:30
zygajykae, also all the core developers are lurking there so it's the best place to seek assistance and chat to us19:30
zygaa few more things:19:30
zyga1) I already mentioned our PPA, we have a policy of targeting Ubuntu 10.04 LTS for our server side code19:31
zygayou should have no problems in installing our packages there19:31
zygaif you want more modern system (we also support all the subsequent ubuntu releases, except for 11.10 which will be supported soon enough)19:32
zygaif you want most of the code is also published on pypi and can be installed on any system with pip or easy_install19:32
zyga2) We have a website at http://validation.linaro.org where you can find some of the stuff we are making in active production19:32
zygaThe most prominent feature there is lava-server with dashboard and scheduler19:33
zyga(the dispatcher is also there but has no web presence at this time)19:33
zygaThere is one interesting thing I wanted to show to encourage you to browse that site more: http://validation.linaro.org/lava-server/dashboard/reports/benchmark/19:34
zygathis is a simple report (check the source code button to see how it works) that shows a few simple benchmarks we run daily on various arm boards19:34
=== yofel_ is now known as yofel
zygathere are other reports but they are not as interesting (pictures :-) unless you know what they show really19:35
zygaanother URL I wanted to share (it's not special, just one I selected now): http://validation.linaro.org/lava-server/dashboard/streams/anonymous/lava-daily/bundles/7c0da1d8765e806102c6f8a707ff22b99a43c485/19:36
zygathis shows a "bundle" which is the primary thing that dashboard stores19:36
zygabundles are containers for test results19:37
zygafrom that page click on the bundle viewer tab to see how a bundle really looks like19:37
zygain the past whenever we were talking about "dashboard bundles" people had a hard time understanding what those bundles are and this is a nice visual way to learn that19:38
zygaone more thing before I'm done19:38
zygawhat we'd like from You19:38
zyga1) Solve your problem, tell us about what you need and how LAVA might help you reach your goal (or what is preventing you from using it effectively), work with us to make that happen19:39
zyga2) Testing toolkit authors: consider allowing your users to save test results in our format19:40
zyga3) Application authors: if you care about quality please tell us what features you'd like to see the most19:41
zyga4) Coders: help us implement new features, we are a friendly and responsible upstream19:41
zygathat's all I wanted to broadcast, I'm happy to answer any questions now19:42
zyganobody into quality it seems :-)19:45
zygaokay, guess that's it -- thanks everyone :-)19:49
ClassBotThere are 10 minutes remaining in the current session.19:51
ClassBotThere are 5 minutes remaining in the current session.19:55
=== ChanServ changed the topic of #ubuntu-classroom to: Welcome to the Ubuntu Classroom - https://wiki.ubuntu.com/Classroom || Support in #ubuntu || Upcoming Schedule: http://is.gd/8rtIi || Questions in #ubuntu-classroom-chat || Event: Ubuntu Developer Week - Current Session: Introduction to Upstart - Instructors: marrusl
ClassBotLogs for this session will be available at http://irclogs.ubuntu.com/2011/07/13/%23ubuntu-classroom.html following the conclusion of the session.20:01
marruslHi folks!20:01
marruslI have a secret to admit.20:01
marruslI'm not actually an Ubuntu Developer.20:01
marruslQuick introduction:20:01
marruslI work for Canonical as a system support engineer helping customers with implementing and supporting Ubuntu, UEC, Landscape, etc.20:02
marruslOn a scale from dev to ops, I'm pretty firmly ops.20:02
marruslHowever, for that very reason, I have a keen interest in what is managing the processes on my systems and how those systems boot and shutdown.20:02
marruslLast thing before I really start...20:03
marruslif you're not very familiar with Upstart, this might be a bit dense with new concepts.20:03
marruslBut to paraphrase Upstart's author, Scott James Remnant:  thankfully this is being recorded, so if it doesn't make complete sense now, you can read it again later!20:03
marruslThe best way to start is probably to define what Upstart is.  If you visit http://upstart.ubuntu.com, you'll find this description:20:04
marruslvices during boot, stopping them during shutdown and supervising them while the system is running.”20:04
marrusllet me try that again20:04
marrusl“Upstart is an event-based replacement for the /sbin/init daemon which handles starting of tasks and services during boot, stopping them during shutdown and supervising them while the system is running.”20:04
marruslMost of that definition applies to any init system, be it classic System V init scripts, SMF on Solaris, launchd on Mac OS X, or systemd.20:05
marruslWhat sets Upstart apart from the others is that it is "event-based" and not "dependency-based".20:05
marrusl(note: launchd is not dependency-based, but it's also not event-based like Upstart.  I could explain why, but we're all here to talk about Linux, right? :)20:06
marruslSo let's unpack those terms:20:06
marruslA dependency-based system works a lot like a package manager.20:07
marruslIf you want to install a package, you tell the package manager to install your "goal package".20:07
marruslFrom there, your package manager determines the required dependencies (and the dependencies of those dependencies and so on) and then installs everything required for your package.20:07
marruslLikewise, in a dependency-based init system, you define a target service and when the system wishes to start that service, it first determine and starts all the dependent services and completes dependent tasks.20:08
marruslFor example, depending on configuration, a mysql installation might depend on the existence of a remote file system.20:09
marruslThe remote filesystem in turn would require networking to be up.20:09
marruslNetworking requires the local filesystems to be mounted, which is carried out by the mountall task.20:09
marruslThis works fairly well with a static set of services and tasks, but it has trouble with dynamic events, such as hot-plugging hardware.20:10
marruslTo steal an example from the Upstart Cookbook (http://upstart.ubuntu.com/cookbook) let's say you want to start a configuration dialog box whenever an external monitor is plugged in.20:11
marruslIn a dependency-based system you would need to have an additional daemon that polls for hardware being plugged.20:11
marruslWhereas Upstart is already listening to udev events and you can create a job for your configuration app to start when that event occurs.20:12
marruslCertainly this requires udev to be running, but there's no need to define that dependency.20:12
marruslSometimes we refer to this as "booting forward".  A dependency-based system defines the end goals and works backwards.20:13
marruslIt meets all of the goal service's dependencies before running the goal service.20:13
marruslUpstart starts a service when its required conditions are met.20:14
marruslIt's a subtle distinction, hopefully it will become clearer as we go.20:14
marruslA nice result of this type of thinking is that when you want to know why "awesome" is running (or not running) you can look at /etc/init/awesome.conf and inspect its start and stop criteria (or on Natty+ run `initctl show-config -e awesome`).20:15
marruslThere's no need to grep around and figure out what other other service called for it to start.20:15
marruslBut enough about init models...  let's get to the real reason I suspect you're here:  how to understand, modify, and write Upstart jobs.20:16
marruslUpstart jobs come in two main forms: tasks and services.20:16
marruslA task is a job that runs a finite process, complete it, and ends.20:16
marruslCron jobs are like tasks, whereas crond (the cron daemon itself) is a service.20:17
marruslSo like other service jobs, it's a long running process that typically is not expected to stop itself.20:17
marruslssh, apache, avahi, and network-manager are all good examples.20:17
marruslNow events...20:18
marruslAn event is a notification sent by Upstart to any job that is interested in that event.20:18
marruslBefore Natty, there were for main types of events: init events, mountall events, udev events and what I'll call "service events".20:19
marruslIn Natty that was expanded to socket events  (UNIX or TCP/IP) and D-Bus events.20:19
marruslEventually this will include time-based events (for cron/atd functionality) and filesystem events (e.g. when this file appears, do stuff!).20:20
marruslYou an type `man upstart-events` on natty or oneiric to see a tabular summary of all "well-known events" along with information about each.20:20
marruslWe're going to mostly focus on the service events, of which there are four.  These are the events that start and stop jobs.20:21
marrusl1. Starting.  This event is emitted by Upstart when a job is *about* to start.20:21
marruslIt's the equivalent of Upstart saying "Hey! In case anyone cares, I'm going to start cron now, if you need to do something before con starts, you'd better do it now!"20:22
marrusl2. Started. This event is emitted by Upstart when a job is now running.20:22
marrusl"Hey!  If anyone was waiting for ssh to be up, it is!"20:22
marrusl3. Stopping.  Like the starting even, this event is emitted when Upstart is *about* to stop a job.20:23
marrusl4. Stopped.  "DONE!"20:23
marruslNote that "stopping" and "stopped" are also emitted when a job fails.  It is possible to establish the manner in which they fail to.  See the man pages for more details.20:24
marrusl(and yes, Upstart shouts everything)20:24
marruslThese events allow other Upstart jobs to coordinate with the life cycle of another job.20:24
marruslIt's probably time to look at an Upstart job to see how this works.20:25
marruslSince I couldn't find a real job that takes advantage of each phase of the cycle, I've created a fake one to walk through.20:25
marruslPlease `bzr branch lp:~marrusl/+junk/UDW` and open the file "awesome.conf"20:25
marruslIf you don't have access to bzr at the moment, you can find the files here:20:26
marruslWhile we look at awesome.conf, it might also help to open the file "UpstartUDW.pdf" and take a look at the second page.20:26
marruslHopefully this will make the life cycle more clear.20:26
marruslAwesome is a made-up system daemon named in honor of our awesome and rocking and jamming Community Manager (please see:  http://mdzlog.alcor.net/2010/03/19/introducing-the-jonometer/)20:27
marruslI mentioned start and stop criteria earlier... well those are the first important lines of the job.20:28
marruslWhat we are saying here is "if either the jamming or rocking daemons signal that they are ready to start, awesome should start first".20:28
marruslIf I want to make sure that awesome runs *after* those services, I would have used "start on started" instead of "starting".20:29
marruslSo let's say Upstart emits "starting jamming", this will trigger awesome to start.20:29
marruslUpstart will emit "starting awesome" and now the pre-start stanza will run.20:29
marruslSome common tasks you might consider putting into "pre-start" are things like loading a settings file into the environment or cleaning up any files or directories that might have been left if the service dies abnormally.20:30
marruslOne more key use of the pre-start is if you want some sanity checks to see if you should even run (are the required files in place?)20:31
marruslAfter pre-start, now we are ready to eithe exec a binary or run a script.  Here we are executing the daemon.20:31
marruslIn most cases, this is when Upstart would emit the "started" event.  In this example, we have one more thing to do: the post-start stanza.20:31
marruslYou might want to use the post-start stanza when waiting for the PID to exist isn't enough to say that the service is truly ready to respond.20:32
marruslFor example, you start up mysql, the process is running, but it might be another moment or two before mysql has finished loading your databases and is ready to respond to queries.20:32
marruslIn my example, I essentially ripped something out of the CUPS upstart job because it illustrates the point well enough.20:33
marruslThis post-start stanza waits for the /tmp/awesome/ directory to exist.  But it doesn't wait forever, it checks every half second for 5 seconds.20:33
marruslIf awesome isn't ready to go by the, something is very wrong and I want it to exit.20:34
marruslSince that script exits with a non-zero status, Upstart will stop the service.20:34
marruslThis might be a good place to mention that all shell fragments run with `sh -e` which means two things...20:34
marruslYour scripts will run with the default system shell, and unless you've changed it, this is by default linked to /bin/dash.20:35
marruslSo do remember to avoid "bashisms" (though you can use "here files" to use any interpreter, please ask later if you'd like to know how, but it's really better form to use only POSIX-complaint sh, imo).20:36
marruslThe other thing it means, is that if any command fails in the script it will exit.  You really can't be too careful running scripts as root.20:36
marruslStopping a service is essentially the reverse... Upstart emits "stopping awesome", exexutes the pre-stop stanza (notice I used an exec in place of a script, you can do this in any of the other stanzas as well).20:37
marruslNow it tires to SIGTERM the process, if that takes longer than the "kill timeout", it will then send a SIGKILL.20:37
marruslI should point out that a well-written daemon probably doesn't need pre-stop.  It should handle SIGTERM gracefully and if it needs to flush something to disk it does so itself.20:38
marruslIf 5 seconds (the default) isn't enough, specify a longer setting in the job as I did here.  In a real job you wouldn't likely be upping the kill timeout _and_ using a “pre-stop” action, I just wanted to illustrate both methods.20:38
marruslOnce post-stop has run (if present), Upstart emits "stopped awesome".20:39
marruslAnd the cycle is complete!20:39
marruslNow, I've covered the major sections of a job, but there are some important additional keywords I'd like to introduce (this is not an exhaustive list):  task, respawn, expect [fork or daemon], and manual.20:39
marrusl“task”.  This keyword, as you might suspect, should be present in task jobs.  There's no argument to it, just put it on a line by itself.20:39
marruslThis keyword lets Upstart know that this process will run its main script/exec and then should be stopped.  Some good examples of task jobs on a standard Ubuntu system are:  procps, hwclock, and control-alt-delete.20:40
marrusl“respawn”.  There are a number of system services that you want to make sure are running constantly, even if they crash or otherwise exit.  The classic examples are ssh, rsyslog, cron, and atd.20:40
marrusl“expect [fork|daemon]”.  Classic UNIX daemons, well, daemonize... that is they fork off new processes and detach from the terminal they started from.  “expect fork” is for daemons that fork *once*, “expect daemon” will expect the process to fork exactly *twice*.20:40
marruslIn many cases, if your service has a “don't daemonize” or “run in foreground” mode, it's simpler to create an Upstart job without “expect” entirely.  You may just have to try both approaches to find out which works best for your service.20:41
marruslWell, unless you are the author, in that case, you probably already know. :)20:41
marrusl“manual”.  The essence of manual is that it disables the job from starting or stopping automatically.  Another way of putting that (and more precise) is that if the word “manual” appears by itself on a line, anywhere in a job, Upstart will *ignore* any previously specified “start on” condition.  So, assuming “manual” appears after the “start on” condition, the service will only run if the20:41
marrusladministrator manually starts it.20:41
marruslNote that were an administrator to start the job by running, “start myjob”, Upstart will still emit the same set of 4 events automatically. So, starting a job manually may cause other jobs to start.20:41
marruslNote too that it is good practise to specify a “stop on” condition since if you do not, the only reasonable manner to stop the job is to kill it at some unspecified time/ordering when the system is shut down.20:42
marruslBy specifying a “stop on”, you provide information to Upstart to enable it to stop the job in an appropriate fashion and at an appropriate time.20:42
marrusladding “manual” seems like a clunky way to disable jobs, doesn't it?  I'd rather not have to hack conf files to disable a job.20:42
marruslAnd what happens to my modified job if there is a new version of the package released and I update?20:43
marruslI'll tell you, your changes will be clobbered.20:43
marrusl(ok, actually you'll be prompted by dpkg to confirm or deny the changes, but that is still pretty annoying and can be confusing for new administrators).20:43
marruslWhich is a nice segue into “override” files, which first appear in Natty.  Override files allow you to change an Upstart job without needing to modify the original job.20:43
marruslWhat override files really accomplish is...  if you put the word “manual” all by itself into a file called /etc/init/awesome.override, it will have the same effect as adding “manual” to awesome.conf.20:44
marruslSo now you can disable a job from starting with a single command:20:44
marruslecho manual >> /etc/init/awesome.override20:44
marruslnote: this is as root only.  Shell redirection doesn't really play nice with sudo.20:45
marruslo disable a job as an admin user:20:45
marruslecho manual | sudo tee -a /etc/init/awesome.override20:45
marruslSince the override file won't be owned the awesome package, dpkg won't object and you can cleanly update it without having to worry about your customizations.  Yay!20:45
marruslI don't really know, but I suspect the original purpose of override files was just to make disabling jobs cleaner.  But then a lightbulb went off somewhere...  why not let administrators override any stanza in the original job?20:45
marruslLet's change awesome's start criteria to make it start *after* rocking or jamming.20:45
marruslSimply create /etc/init/awesome.override and have it contain only this:20:46
marrusl“start on (started rocking or started jamming)”20:46
marruslNow Upstart will use all of the original job file with only this one stanza changed.  This works for any other stanza or keyword.  Want to tweak the kill timeout?  Customize the pre-start?  Add a post-stop?20:46
marruslOverride files can do that.20:46
marruslOn to the last topic of this presentation:  an example of converting a Sys V script to Upstart.20:46
marrusl(looks like it will have to be fast!)20:46
marruslIn the files you branched or downloaded, I've included the Sys V script for landscape-client and my first attempt at an Upstart job to do the same thing (landscape-client.conf).20:46
marruslFirst, some disclaimers... this is *not* any sort of official script, I'm not suggesting anyone use it.  I haven't gotten feedback from the landscape team yet, or properly tested it myself.20:47
marruslBut so far, it seems to be working for me fine. :)20:47
marruslAnd yet, I'm pretty sure I've overlooked something.  I mentioned I wasn't a developer, right?20:47
marruslNot knowing the internals of how landscape-client behaves, I started by trying “expect fork” and “expect daemon”.20:47
marruslBoth allowed me to start the client fine, but failed to stop them cleanly (actually the stop command never returned!).20:47
marruslClearly I picked the wrong approach.  In the end, running it in the foreground (no expect) allowed me to start and stop cleanly.20:48
marruslNow, if you compare the two scripts side-by-side, the most obvious difference is the length. The Upstart job is about 65% fewer lines.20:48
marruslThis is because Upstart does a lot of things for you that had to be manually coded in Sys V scripts.20:48
marruslIn particular it eliminates the need for PID file management and writing case statements for stop, start, and restart.20:48
marruslWell, depending on your previous experience with upstart, that was probably quite a bit of information and new concepts.  I know it took me ages to grok Upstart, and Ubuntu is my full-time job!20:48
marruslSo let me wrap up the formal part of this session with suggestions on the best ways to learn more about Upstart.  They are:20:49
marrusl“man 5 init”20:49
marrusl“man upstart-events”20:49
marruslThe Upstart Cookbook (http://upstart.ubuntu.com/cookbook)20:49
marruslThe Upstart Development blog (http://upstart.at)20:49
marruslYour /etc/init directory.20:49
marrusl(Looking through the existing jobs on Ubuntu is incredibly helpful.)20:49
marruslAnd of course.... #upstart on freenode.20:50
marruslwait... jcastro will kill me if I don't mention http://askubuntu.com/questions/tagged/upstart20:50
marruslWith that...  questions?20:50
ClassBotThere are 10 minutes remaining in the current session.20:50
marruslI'd also like to encourage people to open questions on askubuntu... for the sheer knowledgebase win.20:52
marruslthis link will open a new question and tag it "upstart" for you:20:52
marruslThanks for your time and attention, folks.  HTH.  :)  I'll be around on freenode for a while if something pops up.20:53
ClassBotThere are 5 minutes remaining in the current session.20:55
marrusllborda asks...  first of all, Thank you for the presentation! second, what about debugging upstart services?20:58
marruslThere are a couple levels... debugging upstart itself with  job events, and debugging individual jobs.20:59
marruslThe best techniques are in the Cookbook.  Please see: http://upstart.ubuntu.com/cookbook/#debugging20:59
marruslI guess that's a full wrap.  Take care.21:00
ClassBotLogs for this session will be available at http://irclogs.ubuntu.com/2011/07/13/%23ubuntu-classroom.html21:00
=== ChanServ changed the topic of #ubuntu-classroom to: Welcome to the Ubuntu Classroom - https://wiki.ubuntu.com/Classroom || Support in #ubuntu || Upcoming Schedule: http://is.gd/8rtIi || Questions in #ubuntu-classroom-chat ||

Generated by irclog2html.py 2.7 by Marius Gedminas - find it at mg.pov.lt!