| SpinachHead | anyone in Cali? | 01:56 |
|---|---|---|
| pleia2 | SpinachHead: might want to try #ubuntu-california | 01:57 |
| SpinachHead | just trying to figure out what 1600 UTC time is here. I am stuck at work on a win computer and can't use terminal | 01:58 |
| nigel_nb | @now | 01:58 |
| ubottu | Current time in Etc/UTC: January 27 2010, 01:58:56 | 01:58 |
| cjohnston | 8? | 01:59 |
| cjohnston | i tihkn | 01:59 |
| SpinachHead | okay, so the classes haven't started? | 01:59 |
| cjohnston | They started Monday.. | 01:59 |
| cjohnston | Each day for 5 hours | 01:59 |
| cjohnston | For the US, todays classes are already over | 02:00 |
| SpinachHead | okay... I wanted the Ubuntu One class guess I missed | 02:00 |
| maco | that was like 7 hours ago | 02:00 |
| nigel_nb | SpinachHead, https://wiki.ubuntu.com/UbuntuDeveloperWeek see logs here | 02:01 |
| SpinachHead | thanks | 02:01 |
| ardchoille | nigel_nb: Thanks | 02:01 |
| nigel_nb | np :) | 02:01 |
| ardchoille | Is there a "wish list" or some other way to request a certain type of class? | 02:04 |
| cjohnston | ardchoille: what are you looking for? | 02:05 |
| ardchoille | For instance, I'd like to learn how to write apps in python-gtk and use glade to design the ui and have the glade file usable in the app. I've tried online tutorials but can never understand them | 02:05 |
| ardchoille | It's be nice to have a class that starts with "this is a glade file and this is how you call the file componenets in a pygtk app: or something similar | 02:06 |
| ardchoille | Just anidea for a class | 02:06 |
| cjohnston | ardchoille: have you checked out the past classes to see if anything similar has been taught? | 02:06 |
| cjohnston | that would help in finding someone who could teach it | 02:06 |
| ardchoille | I don't recall anything like that but I should check | 02:06 |
| ardchoille | Is there a list of past classes? | 02:07 |
| cjohnston | in the topic there is a link to the Classroom page... | 02:07 |
| cjohnston | from the classroom page there is an archive area | 02:07 |
| nigel_nb | ardchoille, you can check MOTU school first https://wiki.ubuntu.com/MOTU/School and previous dev week session https://wiki.ubuntu.com/UbuntuDeveloperWeek/Previous | 02:08 |
| ardchoille | The problem is that all of the "getting started with ubuntu development" classes I find always deal with packaging an app and that's it. I never see a "how to write an app in pygtk" type of thing. While packaging is a great skill to have, it's also good to know how to write an app | 02:11 |
| maco | but thats not what ubuntu development is... | 02:11 |
| maco | that'd be upstream development | 02:12 |
| ardchoille | Ah, ok, I misunderstood the meaning of ubuntu development" | 02:12 |
| maco | many ubuntu developers dont know how to program beyond maybe a simple bash script to run at the end of package installation | 02:12 |
| ardchoille | I did not know that | 02:13 |
| maco | you might be more interested in ... i think its called gnomelove | 02:13 |
| maco | i think thats the name of their wannabe-gnome-dev mentoring stuff | 02:14 |
| maco | kde has "junior jobs" which are simple bugs for newcomers, but i dont know of any mentoring for "hi i want to learn qt" | 02:14 |
| maco | and then there's kernelnewbies for people who want to learn to hack the kernel | 02:14 |
| ardchoille | maco: Thank you for the information | 02:15 |
| maco | np | 02:15 |
| ardchoille | I always thought the packagers were programming guru's too | 02:17 |
| maco | i know one guy who does no programming and another that knew cobol 25 years ago but didnt learn a modern language til after becoming an ubuntu dev | 02:18 |
| ardchoille | maco: Thank you! | 02:30 |
| ardchoille | maco: http://live.gnome.org/GnomeLove is an excellent starting point for what I want to do :) | 02:30 |
| maco | great! | 02:30 |
| ardchoille | Hmm.. I wonder if I get good at this, I might be able to teach a class in here at some point in the future | 02:32 |
| cjohnston | ardchoille: sounds good to me | 02:32 |
| maco | i need to learn where kde keeps equivalent docs. been a kde user for a year, and i still only know how to do gnome programming | 02:33 |
| linuxman | someone speek portuguese? | 03:24 |
| cjohnston | hggdh: i think if hes around | 03:25 |
| hggdh | I am here | 03:35 |
| cjohnston | do you speak portuguese? | 03:35 |
| hggdh | cjohnston: yes, I do | 03:36 |
| hggdh | linuxman: why you need Portuguese? | 03:36 |
| linuxman | yes speek | 03:36 |
| linuxman | vc so falam ingles | 03:36 |
| hggdh | linuxman: PVT | 03:37 |
| hggdh | linuxman: a liŋua oficial por cá é o ɪŋles | 03:37 |
| cyborg_7 | no hacking i are noobs | 03:58 |
| cyborg_7 | -.- | 04:00 |
| cyborg_7 | :( | 04:00 |
| luzbelito92 | hi? | 04:18 |
| === jamalta is now known as jamalta-afk | ||
| === Yos_ is now known as Yos | ||
| === nixternal_ is now known as nixternal | ||
| === SmrtySsa is now known as SmartSsa | ||
| === jarlen_ is now known as jarlen | ||
| bittin | http://fc08.deviantart.net/fs71/f/2010/027/7/1/eeedebian_wmii_by_bittin.png me likes <3 | 13:20 |
| === jamalta-afk is now known as jamalta | ||
| === breiner is now known as breinera | ||
| === cjohnston is now known as FFEMTcJ | ||
| === FFEMTcJ is now known as cjohnston | ||
| dholbach | WELCOME EVERYBODY TO ANOTHER FINE DAY OF UBUNTU DEVELOPER WEEK! | 15:59 |
| dholbach | If you are a bit confused, have a look at - it will explain, why you are here: https://wiki.ubuntu.com/UbuntuDeveloperWeek | 15:59 |
| dholbach | On an organisational note: if you have questions, please ask in #ubuntu-classroom-chat and prefix them with QUESTION: so they stand out. | 16:00 |
| dholbach | Also, if you have questions but are afraid to ask in English head to one of the channels mentioned at the bottom of https://wiki.ubuntu.com/UbuntuDeveloperWeek | 16:00 |
| dholbach | First up is the fantastic John Arbash Meinel - he'll talk about "Writing good test-cases"! | 16:00 |
| dholbach | jam: the floor is yours! | 16:00 |
| jam | greetings all! | 16:01 |
| jam | I'm happy to see so many names on the channel.... hopefully you all aren't just lurking | 16:01 |
| jam | waiting for the next speaker :) | 16:01 |
| jam | I'm going to be pretty loose with the discussion | 16:01 |
| jam | so feel free to ask questions | 16:01 |
| jam | cjohnston has graciously offered to help me keep track | 16:02 |
| cjohnston | o/ | 16:02 |
| jam | So what is a 'good' test case | 16:02 |
| jam | L | 16:02 |
| jam | ? | 16:02 |
| jam | Generally speaking, a test case is meant to verify that something is happening correctly. | 16:03 |
| jam | Or looking at it differently, when something *isn't* happening correctly, the test case should fail. | 16:04 |
| jam | However, there are still a lot of subtle aspects. | 16:04 |
| jam | I would say that the desirable features of a test case break down into | 16:04 |
| jam | 1) Sensitivity (the likelyhood that the test case will break when something goes wrong) | 16:04 |
| jam | 2) Specificity (the likelyhood that the test case will break and what you are testing has not gone wrong) | 16:05 |
| jam | 3) Performance | 16:05 |
| jam | It is easy to focus on say 1, but to have a *good* test, all 3 are important | 16:05 |
| jam | It is great to have a huge test suite that covers every possible aspect of your code, and all permutations | 16:06 |
| jam | but if it takes 3 weeks to run, it is often not very useful for development | 16:06 |
| jam | Likewise, a test with low specificity will be breaking all the time | 16:06 |
| jam | and won't really help you isolate the problem | 16:06 |
| jam | You can argue that 4) coverage is a property, but I would argue that it is a desirable property of the whole test suite, and not a great property of a single test | 16:07 |
| jam | Personally, I think there are a range of 'good' tests, generally dependent on what aspect you are focusing on | 16:10 |
| jam | I personally think that having lots of focused tests is better than having a single 'test' that is testing lots of aspects | 16:10 |
| jam | but integration tests are still useful an needed | 16:11 |
| jam | and | 16:11 |
| jam | So how about we put together an example | 16:12 |
| jam | I'll use python, since that is my preferred language | 16:13 |
| jam | and it has a decent unittest suite | 16:13 |
| cjohnston | QUESTION: Even after weeks of testing by most experience testers, very robust apps breakdown all of a sudden, what might be the reasons behind it? | 16:13 |
| jam | ANSWER: I think that complexity is generally a major factor causing 'bugs' in software. | 16:14 |
| jam | It is often considered inherent in any sufficiently developed program. | 16:15 |
| cjohnston | < Omar871> QUESTION: according to what I learned in college, a 'good' test is that one that makes that system break/crash, to show where the problem is, how true could that be? | 16:15 |
| jam | Generally, this means there will be some sort of permutation of objects which has not been tested | 16:15 |
| jam | directly | 16:15 |
| jam | A goal of development is to manage complexity (generally by defining apis, separation of concerns, etc) | 16:16 |
| jam | good software can then have good tests that test a subset, without *having* to manage the permutation problem | 16:16 |
| jam | (but generally, abstractions can 'leak', and ... boom) | 16:16 |
| jam | ANSWER <omar>: I think I understand you to mean the "inverse" | 16:17 |
| jam | which is that a good test is one that provokes a problem in the system | 16:17 |
| jam | I think that for a regression style test, it is certainly important that the test would trigger the bug that you are trying to fix | 16:18 |
| jam | However, once a bug is fixed, you would certainly expect it to not provoke the problem anymore | 16:18 |
| jam | So it is certainly important that a test exposes a weakness | 16:19 |
| jam | I guess put another way... | 16:19 |
| jam | If I had a bug, and wrote a test case, and it *didn't* trigger the bug, that test doesn't have a lot of worth (for fixing that bug) | 16:19 |
| jam | (Which often accidentally happens if you fix the bug before writing the test) | 16:20 |
| jam | n3rd: We were just discussing in our group about "coding creates bugs... are we reducing the number of bugs faster than we are creating them?" | 16:22 |
| jam | n3rd also brings up a decent point | 16:24 |
| jam | users often find bugs that developers don't think of | 16:24 |
| jam | often because they use software in ways that weren't anticipated | 16:24 |
| jam | often this goes back to the permutation issue | 16:25 |
| jam | it isn't possible to test every possible permutation | 16:25 |
| jam | (well, rarely possible) | 16:25 |
| cjohnston | < n3rd> jam, so the users are passive tester? | 16:26 |
| jam | A file with just 20 bytes has 8^20 = ~10^48 permutations | 16:26 |
| jam | Well, I would say that users are often pretty *active* testers | 16:27 |
| jam | however, they don't make good automated test suites | 16:27 |
| jam | I suppose that would be my: | 16:27 |
| jam | 4) Reproducible | 16:27 |
| jam | (the chance that running the same thing now will give you the same thing it gave you before) | 16:28 |
| jam | It is somewhat related to specificity | 16:28 |
| jam | As an unreproducible test has low specificity | 16:28 |
| jam | (it breaks for reasons that you aren't trying to test) | 16:28 |
| jam | I guess I should also mention... if a user used every intermediate version of my program, they'd run into a lot more bugs | 16:29 |
| jam | As a developer I fix a huge amount of things before it gets released | 16:29 |
| jam | it is just that often the set of *remaining* bugs | 16:29 |
| jam | are ones that I had not anticipated very well | 16:30 |
| jam | Anyway, I think it is useful to remember what the strengths of a given style of testing are. | 16:34 |
| jam | You can have automated tests (unit and integration tests), manual (interactive) testing, foisting off the software on your users | 16:34 |
| jam | etc | 16:34 |
| jam | I do think that testing at each level is useful | 16:34 |
| jam | and trying to test things at a the wrong level introduces more pain than benefit | 16:35 |
| jam | Having an absolutely bulletproof piece of software that doesn't do what your users want isn't particularly beneficial | 16:35 |
| jam | So having user testing in your feedback loop is certainly a good thing | 16:36 |
| jam | However, giving them a buggy PoS is also not going to make them very happy | 16:36 |
| jam | I'm certainly a fan of multi-tier testing, including automated testing | 16:36 |
| jam | having a small fast test suite that is likely to expose bugs is nice on 'must pass before making it to trunk' | 16:36 |
| jam | having a slower but more thorough "must pass before releasing to users" | 16:37 |
| jam | and for some systems adding a "must be tested by a human interaction" can be inserted in there as well | 16:37 |
| jam | If the first one takes much more than 5 minutes, it often causes grief when trying to get development done | 16:38 |
| jam | but the second can take a day, and still not slow down your release cycle | 16:38 |
| cjohnston | < Omar871> QUESTION: Could the efficiency and effectivenes of the testing process depend on the licensing type of the software we are making? | 16:44 |
| jam | ANSWER: I don't think it would change dramatically | 16:45 |
| jam | If the license is open, it does allow users to do introspective testing which would just not be possible otherwise | 16:46 |
| jam | however, few users can really develop your software anyway, so it certainly shouldn't be relied upon as a source of improving correctness | 16:46 |
| jam | even if your users are very intelligent they almost always | 16:46 |
| jam | 1) don't know your code | 16:46 |
| jam | 2) don't have time to be doing your work for you :) | 16:46 |
| jam | I think Linus gets a bit of a boost, because there are lots of developers on the project, not just users | 16:47 |
| jam | Certainly a "lot of eyeballs" requires eyeballs that can see the code | 16:48 |
| jam | and with enough, you can have a person to specialize for any given subset, which also helps in defect localization (hence 'shallow') | 16:49 |
| cjohnston | < hggdh> QUESTION: are there any considerations for *usability* testing (this is one thing that users would certainly be able to perform)? | 16:50 |
| jam | ANSWER: I think that usability testing is certainly important | 16:51 |
| jam | (10:35:23 AM) jam: Having an absolutely bulletproof piece of software that doesn't do what your users want isn't particularly beneficial | 16:51 |
| jam | There is an argument whether this specifically falls under the standard category of 'testing' | 16:51 |
| jam | (market research is certainly important to developing software, but it isn't "testing" :) | 16:52 |
| cjohnston | < strycore89> QUESTION : how is testing of graphical applications done ? (For example PyGTK apps) | 16:56 |
| jam | IME, you can test PyGTK (and PyQt) without actually showing dialogs | 16:56 |
| jam | both of them support updating widgets by programattically setting values | 16:57 |
| jam | and then triggering events | 16:57 |
| jam | In that case, they can be tested in the same fashion as any other unit testing | 16:57 |
| jam | however, it doesn't test the visual representation | 16:57 |
| jam | which is a valid level to test | 16:57 |
| jam | There are also gui testing suites that can be used | 16:58 |
| jam | I forget the exact name of one (suikili?) | 16:59 |
| jam | Which uses screenshots | 16:59 |
| jam | and some work for marking the regions you care about | 16:59 |
| jam | http://groups.csail.mit.edu/uid/sikuli/ | 17:01 |
| dholbach | alrightie! | 17:02 |
| dholbach | thanks jam for giving this great talk! | 17:02 |
| * dholbach hugs jam | 17:02 | |
| dholbach | next up are henninge and adiroiban, the dynamic duo that'll explain "Launchpad Translations under the hood" to you! | 17:02 |
| henninge | Hi everybody, thanks for coming! | 17:03 |
| henninge | Today we want to help you understand the inner workings of the Launchpad Translations application (Rosetta) and take you for a walk through the source code. We hope that this will enable you to scratch your own itches you have about Rosetta and to contribute to its source. | 17:03 |
| henninge | So the intendend audience is | 17:05 |
| henninge | * Developers wanting to contribute to Launchpad Translations but are not yet familiar with the internal structure of the application. | 17:05 |
| henninge | * Interested maintainers of translations in Launchpad and translators that want to have a better understanding of how and why Launchpad Translations does what it does. | 17:05 |
| henninge | We hope that by the end of the session you have a good overview of | 17:05 |
| henninge | # how translation data is stored in LP translation (db schema) | 17:05 |
| henninge | # how the source code is organized | 17:05 |
| henninge | # what to expect when diving into the source code | 17:05 |
| henninge | # where to start when trying to hack on Launchpad Translations. | 17:05 |
| henninge | It is not the goal of this session to introduce you to Launchpad development in general. That will be covered in a different session by Karl Fogel. | 17:06 |
| henninge | We encourage you to visit that on Friday. | 17:06 |
| henninge | Thursday, sorry | 17:06 |
| henninge | Anyway, we are happy to take questions. We will keep the session open towards the end to see what questions come up and what your interests are. | 17:07 |
| henninge | We will be taking questions in intervalls, so cjohnston will nicely hold them until we ask for them. Thanks. | 17:08 |
| henninge | Go, adiroiban ;-) | 17:08 |
| adiroiban | Hi | 17:08 |
| adiroiban | I will continue with Gettext Basics | 17:08 |
| adiroiban | You need to understand how gettext is used to internationalize computer software. You should be familiar gettext documenation but we will give you a short run-through of those parts that are important for Rosetta. | 17:08 |
| adiroiban | = PO files = | 17:09 |
| adiroiban | Gettext stores translations in so-called portable object files, abreviated as PO files. They contain data sets of msgid and msgstr, the former containing the English original string, the later containing the translations of that string. They may be prepended by special comments that convey information about the string that is being translated, like in which source file it was found. Here is an example: | 17:09 |
| adiroiban | #: src/coding.c:123 | 17:09 |
| adiroiban | msgid "Thank you" | 17:09 |
| adiroiban | msgstr "Merci" | 17:09 |
| adiroiban | Gettext states that msgid could be anything to indentify the string in the source code and not necessarily the English original string. Using the full English original string as the msgid, though, has proven to be the most convenient way to work on translations and is _the only form_ that is fully supported by Rosetta. | 17:10 |
| adiroiban | The first msgid in a PO file is empty and its msgstr contains meta information about the file. The minimum information here is the MIME Content-type of the file but usually a lot of other information is included, too. | 17:10 |
| adiroiban | Here it is | 17:11 |
| adiroiban | mmsgid "" | 17:11 |
| adiroiban | msgstr "" | 17:11 |
| adiroiban | "Project-Id-Version: PACKAGE VERSION\n" | 17:11 |
| adiroiban | "Report-Msgid-Bugs-To: \n" | 17:11 |
| adiroiban | "POT-Creation-Date: 2009-01-26 12:28+0000\n" | 17:11 |
| adiroiban | "PO-Revision-Date: 2009-01-26 12:28+0000\n" | 17:11 |
| adiroiban | "Last-Translator: Foo Bar <foo.bar@canonical.com>\n" | 17:11 |
| adiroiban | "Language-Team: French <fr@li.org>\n" | 17:11 |
| adiroiban | "MIME-Version: 1.0\n" | 17:11 |
| adiroiban | "Content-Type: text/plain; charset=UTF-8\n" | 17:11 |
| adiroiban | "Content-Transfer-Encoding: 8bit\n" | 17:11 |
| adiroiban | The standard naming convention for PO files is to use the language code, so in this case fr.po. | 17:11 |
| adiroiban | = Translation templates = | 17:11 |
| henninge | Questions so far? | 17:12 |
| cjohnston | henninge / adiroiban you aren't using slides are you? | 17:12 |
| adiroiban | no | 17:13 |
| cjohnston | < Navaneeth> QUESTION: Is gettext portable to other OS? | 17:13 |
| adiroiban | no slides | 17:13 |
| adiroiban | sorry | 17:13 |
| adiroiban | the content is on the wiki | 17:13 |
| henninge | Navaneeth: The PO format is machine independent. | 17:14 |
| adiroiban | Gettext is available on many operating systems | 17:15 |
| adiroiban | I don't think you need to worry for gettext portability | 17:15 |
| adiroiban | I will continue with PO Templates | 17:15 |
| adiroiban | When translatable strings are extracted from source code using xgettext or intltool, they are put into a file which is commonly referred to as the translation template. Its format is identical to that of a PO file but all the msgstr lines do not contain any translations. These files are intended to be used for creating new PO files, so they also contain the header information but with most fields left with empty or generic values. | 17:15 |
| adiroiban | Since a PO template is not really a separate file format it does not find much mention in the gettext documentation. Also, because its content can be generated from source any time (like during a build), most projects don't include it in their repository. Only PO files contain valuable information for a project, the translations themselves, and are therefore included in the source code repository. | 17:16 |
| adiroiban | The standard naming convention for PO templates is to use the translation domain with the extension .pot, for example myproject.pot. | 17:17 |
| adiroiban | How translations are done in the simple gettext context: | 17:18 |
| adiroiban | 1) To start a translation into a new language for a project, the following steps are necessary: | 17:18 |
| adiroiban | * Either the project maintainer or the translator creates a template from source code. | 17:19 |
| adiroiban | * The translator fills out the template with the translations for each msgid. | 17:19 |
| adiroiban | * The translator saves the file in the source tree as languagecode.po (lsee above), ususally in a directory called po. | 17:19 |
| adiroiban | * The translator or somebody with commit rights commits the file to the repository. | 17:19 |
| adiroiban | * Whenever the package is built, the translations are processed so that they are available at run-time (out of scope here). | 17:19 |
| adiroiban | 2) To change translations, the steps are simpler: | 17:20 |
| adiroiban | Sorry. Launchpad Workflow :) | 17:20 |
| adiroiban | When using Launchpad to translate a project, the steps are slightly different because the PO files are kept in Launchpad for the translators to work with. From Launchpad they are mirrored into the source tree to be used at build time. | 17:20 |
| adiroiban | * The project maintainer uploads the PO template file to Launchpad. | 17:21 |
| adiroiban | * Translators go to Launchpad to translate the English strings that now appear in the web interface. | 17:21 |
| adiroiban | * The project maintainer downloads all PO files whenever they want to, usually to prepare a release of the software. | 17:21 |
| adiroiban | * Nowadays the upload and download should happen automatically from and to Bazaar branches in Launchpad so that the maintainer always has a mirror of the latest translations in the branch, while changes to the PO template are automatically propagated to Launchpad. The next step will be automatic generation of PO templates from the source code in a Bazaar branch. | 17:21 |
| henninge | Questions? | 17:21 |
| henninge | cjohnston: Questions? ;) | 17:22 |
| cjohnston | I don't see any | 17:22 |
| henninge | cool | 17:22 |
| cjohnston | one just came | 17:22 |
| cjohnston | < Navaneeth> QUESTION: PO files need to deployed along with the application? | 17:22 |
| henninge | cjohnston: actually, they are converted to a binary format first "MO". | 17:23 |
| adiroiban | You will deploy the MO files (machine object) | 17:23 |
| henninge | Navaneeth: ^ | 17:23 |
| henninge | ;) | 17:23 |
| henninge | Next, we'll go into how this structure is mapped into the Launchpad database. | 17:24 |
| henninge | jtv drew up this amazing diagram: | 17:24 |
| henninge | https://wiki.ubuntu.com/UbuntuDeveloperWeek/Sessions/LaunchpadTranslationsUnderTheHood?action=AttachFile&do=get&target=TranslationsSchema.png | 17:24 |
| cjohnston | < Omar871> QUESTION: What's a PO file? | 17:25 |
| henninge | Omar871: the actual translations are stored in PO files. It contains English strings and their translation. | 17:25 |
| henninge | s | 17:25 |
| henninge | That's what translators work with when translating. | 17:26 |
| henninge | Back to database schema. | 17:26 |
| henninge | You can see the main tables used for Launchpad Translations in the digram. PO templates are mapped into the database using four tables. | 17:26 |
| henninge | POMsgID is a look-up table for all the English strings that are being translated. | 17:27 |
| henninge | So it gives each string a numeric ID. | 17:27 |
| henninge | POTMsgSet holds all the data related to an original English string as found in a PO template, one database entry per msgid entry in the file. It refers to the actual English strings only by their IDs in POMsgID. This represent one paragraph/entry from a PO template (msgid, msgid_plural, context, comments). | 17:27 |
| henninge | POTemplate holds the meta data related to a PO template as it has been imported, most notably the original path name, the translation domain, the original header and a flag if this template is active or not. | 17:28 |
| henninge | TranslationTemplateItem is a linking table because of the n:m realtionship between POTMsgSet and POTemplate which message sharing introduces. Not only does a PO template file contain multiple msgid entries, the same msgid may also appear in multiple PO template files, if the same template is used accross different series of a a project. | 17:28 |
| henninge | Questions? | 17:28 |
| henninge | cjohnston: ^ | 17:28 |
| cjohnston | < n3rd> QUESTION: Can we not give auto suggestions/transitions like Rosy with the DB and PO ? | 17:29 |
| henninge | n3rd: I don't know Rosy but we do have "external suggestions" in Rosetta. | 17:29 |
| henninge | That is one of the key features of it, actually. | 17:30 |
| henninge | If a string has already been translated in a different package or project, it's will be suggested to the translator. | 17:30 |
| henninge | next question | 17:31 |
| cjohnston | < bullgard45> QUESTION: How do you take care of that the same English phrase needs to be translated differently depending on the program at hand? | 17:31 |
| adiroiban | Just add a new translation | 17:31 |
| henninge | bullgard45: that's why they are just presented as "suggestions". It still takes a reviewer to acutally accept it as a translation to use. | 17:31 |
| adiroiban | bullgard45: a program will use a certain PO file | 17:32 |
| adiroiban | and you can have the same string in 2 PO files | 17:33 |
| adiroiban | and for each PO file to have different translations | 17:33 |
| henninge | in database terms: | 17:33 |
| henninge | You can have multiple TranslationMessage entry related to one POTMsgSet entry. I will come to that in a minute. | 17:34 |
| henninge | next question? | 17:34 |
| cjohnston | < n3rd> QUESTION:Would it not result in overhead when there are many revisions for the same sentence, and finally refined to perfect translation? | 17:34 |
| henninge | n3rd: it does ;-) | 17:35 |
| henninge | n3rd: we have a lot of entries in the POTranslation table. | 17:35 |
| henninge | but since we consider all those refining steps as contributions we keep them for reference's sake. | 17:36 |
| henninge | next questions? | 17:36 |
| cjohnston | < n3rd> QUESTION : So the alternative is ? | 17:36 |
| henninge | n3rd: none I am aware of atm... | 17:37 |
| henninge | We have been cleaning out translations of discontinued Ubuntu series. | 17:37 |
| henninge | Also, for most cases there are not that many iterations, especially if the translations are imported from upstream. | 17:38 |
| henninge | more questions? | 17:38 |
| cjohnston | < Emilien> Question: in a project that is oenly translated, if I modify an already translated string, does it overwrite the old translation, or is it stored as a new entry? | 17:38 |
| henninge | Emilien: new entry. | 17:38 |
| henninge | A new TranslationMessage and POTranslation entry is created. | 17:39 |
| henninge | Let me add my explanations for the remaining four tables that all deal with this: | 17:39 |
| henninge | POTranslation is a simple look-up table and holds the actual translated strings | 17:39 |
| henninge | TranslationMessage holds all the information about a translation to a specific language, like when it was done and by whom, if it was translated in Launchpad or imported from elsewhere, if it is currently used or just a suggestion, etc. For each POTMsgSet there may be multiple entries in this table, even for the same language, because any translation ever made is stored in the database, even if only the latest is actually used. | 17:39 |
| henninge | The actual translation strings are referred to by their id in POTranslation. | 17:39 |
| henninge | Language is the set of all languages known in Launchpad. This table is not specific to Launchpad Translations as it is used in other parts of Launchpad, too. | 17:40 |
| henninge | POFile represents the set of translations into a certain language for a POTemplate. If it was created by importing a PO file, it also holds some information about that file. It is not linked directly to any translation but this relationship can be derived through the Language table. | 17:40 |
| henninge | Questions? | 17:40 |
| cjohnston | < bullgard45> QUESTION: I have seen a lot of po files in the WWW. How are they related to Launchpad? | 17:40 |
| henninge | bullgard45: Can you be more specific about WWW, please? | 17:41 |
| henninge | as in: I did not understand the question ... | 17:42 |
| henninge | another question ? | 17:42 |
| cjohnston | That's it for nwo | 17:42 |
| cjohnston | now | 17:42 |
| henninge | adiroiban, take us into the code, please. | 17:42 |
| adiroiban | http://bazaar.launchpad.net/~launchpad-pqm/launchpad/db-devel/files/head:/lib/lp/translations/ | 17:43 |
| adiroiban | OK. we should see the Launchpad Translation folder structure | 17:43 |
| adiroiban | I will not describe them in alphabetic order | 17:44 |
| adiroiban | == Model == | 17:44 |
| adiroiban | * _interfaces_ contains the Zope interface definitions and schema for the objects used by the application. You will find interfaces for each of the database tables described earlier. For example, potemplate.py contains IPOTemplate. | 17:44 |
| adiroiban | * _doc_ contains function tests for code from model. Written as doctests. | 17:44 |
| adiroiban | * _model_ contains objects mapping to relational database using storm. | 17:44 |
| adiroiban | * _tests_ contains unit tests for code from model. | 17:44 |
| adiroiban | Next we have the folder containing the view layer | 17:45 |
| adiroiban | == View == | 17:45 |
| adiroiban | * _browser_ contains classes dealing with presentation logic and user interaction. | 17:45 |
| adiroiban | * _browser/tests_ contains unit tests for code from browser. | 17:45 |
| adiroiban | * _emailtemplates_ contains templates for various emails issued by Rosetta. | 17:45 |
| adiroiban | * _help_ contains documentation and help pages integrated with Rosetta. | 17:46 |
| adiroiban | * _stories_ constains functional tests for code from browser. Written as doctests. | 17:46 |
| adiroiban | * _templates_ contains Zope Page Templates use by the objects from browser. | 17:46 |
| adiroiban | * _windmill_ contains tests for javascript code using Windmill Python API. | 17:46 |
| adiroiban | * _lib/canonical/launchpad/javascript/translations/_ contains YUI 3 javascript code. | 17:46 |
| adiroiban | and finaly some utilities | 17:46 |
| adiroiban | == Utilities == | 17:46 |
| adiroiban | * _scripts_ contains various helping scripts used in cronjobs or doing other utility and integration jobs. | 17:46 |
| adiroiban | * _scripts/tests_ contains tests for code from scripts. | 17:46 |
| adiroiban | * _utilities_ contains utility classes used in model and browser code, mostly data conversion related. | 17:46 |
| adiroiban | Hope you have noticed that almoust all code has the coresponding tests | 17:47 |
| henninge | Questions? | 17:48 |
| cjohnston | 12:43 < bullgard45> hennige: I have seen many .po files having English text of GNOME programs and their German translations. I could make good use of some of them. I do not have a particular URL at hand just now but I can provide one in a moment. | 17:48 |
| cjohnston | 12:43 < bullgard45> hennige: I have seen many .po files having English text of GNOME programs and their German translations. I could make good use of some of them. I do not have a particular URL at hand just now but I can provide one in a moment. | 17:48 |
| henninge | cjohnston: you can find those files officially on the Damned LIes website of gnome. | 17:49 |
| henninge | http://l10n.gnome.org/ | 17:49 |
| henninge | http://l10n.gnome.org/teams/de | 17:50 |
| henninge | They are all imported into Launchpad to be included in the Ubuntu language packs. | 17:50 |
| henninge | We only edit them in Launchpad Translations if we need to differ from standard Gnome. | 17:50 |
| henninge | More questions? | 17:51 |
| cjohnston | < Emilien> QUESTION: (but maybe keep it for later, as it's not relevant right now): When there is only one series, and one translation template, why couldn't we get the template page as the default | 17:51 |
| cjohnston | translations page. Example: with project BaShare, you have 3 translation pages: root https://translations.launchpad.net/bashare , series | 17:51 |
| cjohnston | https://translations.launchpad.net/bashare/trunk/+translations and template https://translations.launchpad.n | 17:51 |
| cjohnston | < Emilien> et/bashare/trunk/+pots/bashare . The last page is IMHO the most useful, as it contains links to untranslated, need review or changed strings. This could be the default page when there is | 17:51 |
| cjohnston | only one series and one template... What is your opinion about that (and sorry for long question) | 17:51 |
| henninge | Emilien: that sounds like a good idea as long as it is clear to viewer what is going on. | 17:52 |
| henninge | Emilien: You can describe that in a bug, discuss it with us and even go about and fix it ... ;-) | 17:53 |
| henninge | More questions? | 17:53 |
| cjohnston | < bullgard45> QUESTION: hennige: An example for a po file that I just have found in the WWW, is http://www.mail-archive.com/xfce4-commits@xfce.org/msg08020.html | 17:53 |
| henninge | That is actually a diff of PO file ... ;) | 17:54 |
| henninge | This should be imported into Launchpad if they'd want to use it for translations. https://edge.launchpad.net/xfce | 17:55 |
| henninge | But AFAIR they decided against using Launchpad for translations. | 17:55 |
| henninge | Another question? | 17:56 |
| cjohnston | < Emilien> Question: I like how all the parts have their own tests. Are they automatically executed in some way? | 17:56 |
| henninge | Ah, the great Launchpad test suite, Emilien! | 17:56 |
| henninge | Run "make check" to run the whole test suite. Takes a few hours. | 17:57 |
| henninge | Each branch that is submitted to the LP tree needs to have had this run. | 17:57 |
| henninge | Then we have the buildbot that continously runs the test suite on the trunk with every new commit. | 17:58 |
| henninge | One more question? | 17:58 |
| cjohnston | < bullgard45> QUESTION: I understood that you considered the translations provided by Launchpad as proposals or a help for a human translator. This somewhat contradicts the attitude of a German | 17:58 |
| cjohnston | translator team to admit or restrict would-be human helpers. (But I cannot provide the document that proves such a censorship.) I do understand that a high standard of qualtiy | 17:58 |
| cjohnston | translations must be maintained. | 17:58 |
| henninge | bullgard45: I think what you are referring to is this: | 17:59 |
| henninge | Anybody can submit suggestions in Launchpad but only those on the translation team are reviewers and can actually accept them to be used. | 18:00 |
| henninge | This requeires some deal of organization to keep the translations consistent. | 18:00 |
| henninge | But actually, bullgard45, I don't see a question in there ... ;-) | 18:01 |
| henninge | But, we have run out of time. | 18:01 |
| cjohnston | Thanks henninge and adiroiban ! | 18:01 |
| henninge | Thank you all for coming. Come and talk to us on #launchpad any time. Thanks. | 18:01 |
| cjohnston | Next up is smoser | 18:02 |
| adiroiban | Thanks! | 18:02 |
| smoser | Hello all | 18:02 |
| smoser | Ok, lets get started. | 18:02 |
| smoser | This is my first time doing one of these, so I ask for forgiveness in advance | 18:02 |
| smoser | I'm Scott Moser (https://launchpad.net/~smoser). | 18:02 |
| smoser | For the past few months I've been working on maintaining and improving Ubuntu's presense on Amazon's Cloud Offering, known as "EC2" | 18:02 |
| smoser | In this session, I hope to explain : | 18:02 |
| smoser | * A little bit about what EC2 is (at least from my point of view) | 18:03 |
| smoser | * What you need to do to use it | 18:03 |
| smoser | * What you can do with it | 18:03 |
| smoser | * Some warnings about issues you might find | 18:03 |
| smoser | * Some miscellaneous things I'd like to mention | 18:03 |
| smoser | and then, have some time for questions | 18:03 |
| smoser | * Questions | 18:03 |
| smoser | first off, I guess I'd like to kind of see a show of hands on who is interested in this session | 18:04 |
| smoser | if you are, simply a 'o/' or 'i am' .... | 18:04 |
| smoser | well, seeing none , we'll just go on | 18:05 |
| smoser | If you have questions along the way, please feel free to interrupt. If it becomes too much, then we might have cjohnston help moderate. | 18:05 |
| cjohnston | There's quite a efw | 18:05 |
| cjohnston | quite a few | 18:05 |
| smoser | == A little bit about what EC2 is (at least from my point of view) == | 18:05 |
| smoser | """Amazon Elastic Compute Cloud (Amazon EC2) is a web service that provides resizable compute capacity in the cloud. It is designed to make web-scale computing easier for developers.""" | 18:05 |
| smoser | What that means to me is: | 18:06 |
| smoser | * Virtual Servers, as many "clean instances" as you want, provisioned on your command | 18:06 |
| smoser | * ability to access permenant professional grade storage from inside the virtual servers (EBS or S3) | 18:06 |
| smoser | * access to fast network resources (much better than my cable modem) | 18:06 |
| smoser | You can use the above, and do whatever you want, then throw it away. | 18:07 |
| smoser | You get charged by the unit (hour/transfer/disk-size) for the resources as you are using them. | 18:07 |
| smoser | Depending on what your interest is, it may be tempting to simply calculate the charges for a month of m1.small (0.085 * 24 * 30 = $61.20), compare that to a virtual private server at http://www.linode.com and stop looking. | 18:08 |
| smoser | However, EC2 and "Cloud" is something quite different. The biggest difference being the ability to start up and tear down instances in very little time. | 18:08 |
| smoser | Any questions so far? | 18:08 |
| cjohnston | < acicula> Question: define little, do they have spinning instances running or something? | 18:09 |
| smoser | acicula, 'little' is defined in terms of seconds , i'd say on average less than 100 | 18:09 |
| smoser | i dont have hard numbers, and those vary based on time of day and load of ec2, and also (I think) popularity of the instance being requested | 18:10 |
| smoser | (the more popular an instance, the faster you get a copy) | 18:10 |
| smoser | moving on | 18:11 |
| smoser | === What you need to do to use it === | 18:11 |
| smoser | Assuming you weren't completely turned off from my explanation above lets move on to how you can use this. | 18:11 |
| smoser | I'm really only going to cover command line usage via the ec2-api-tools, but there are several other ways to manage ec2 | 18:12 |
| smoser | First thing you need is a credit card, and an AWS account with EC2 access. | 18:12 |
| smoser | The EC2Starters Guide (https://help.ubuntu.com/community/EC2StartersGuide) has a good description on how to get signed up. Make sure to generate, and download your X509 certificate, and key and keep them safe. | 18:12 |
| smoser | Next, you'll need the ec2 api tools. The latest version is available in Lucid and backports are available for karmic and hardy at https://launchpad.net/~ubuntu-on-ec2/+archive/ec2-tools . Install those tools by enabling multiverse and then: | 18:13 |
| smoser | $ apt-get install ec2-api-tools ec2-ami-tools | 18:13 |
| smoser | (note, for just "using" the cloud, the api-tools are probably enough. if you get interested in creating content for it, you will need to get the ec2-ami-tools) | 18:14 |
| smoser | Commands I show will assume that you have EC2_PRIVATE_KEY and EC2_CERT in your environment. | 18:14 |
| smoser | the EC2StartersGuide mentions how to set that up also | 18:14 |
| smoser | Access to instances "in the cloud" is via network only, and the primary method being ssh (at least here) | 18:15 |
| smoser | In order to access instances via ssh, you need to create a keypair. | 18:15 |
| smoser | This is a normal ssh private/public key, but amazon makes public portion available to the instance when you run it. | 18:15 |
| smoser | Create that keypair with: | 18:16 |
| smoser | $ ec2-add-keypair ec2-keypair.us-east-1 > ec2-keypair.us-east-1.pem | 18:16 |
| smoser | $ SSH_KEY=$PWD/ec2-keypair.us-east-1.pem | 18:16 |
| smoser | At this point, you're ready to go. You can launch a new instance of the latest release of karmic on i386 with: | 18:17 |
| smoser | $ ec2-run-instances --instance-type m1.small --key ec2-keypair.us-east-1 ami-1515f67c | 18:17 |
| cjohnston | < bullgard45> QUESTION: What is an AWS account? | 18:17 |
| smoser | bullgard45, sorry, AWS Account is Amazon Web Services (http://aws.amazon.com/) | 18:18 |
| smoser | expanding on m1.small | 18:18 |
| cjohnston | < kamalmostafa> QUESTION: Please expound on "m1.small". | 18:19 |
| smoser | the above command says "give me a machine of type m1.small" and make the keypair that i just created able to access it. | 18:19 |
| smoser | m1.small is described at http://aws.amazon.com/ec2/instance-types/ | 18:19 |
| smoser | and pricing information available at http://aws.amazon.com/ec2/#pricing | 18:20 |
| smoser | m1.small is the cheapest, and is i386 virtual hardware | 18:20 |
| smoser | You will see output similar to the example at https://wiki.ubuntu.com/UbuntuDeveloperWeek/Sessions/GettingStartedWithEC2 . | 18:21 |
| smoser | The magic 'ami-xxxxxx' number above is the image-id. You can see a list of available images via: | 18:21 |
| smoser | $ ec2-describe-images --all > images.list | 18:21 |
| smoser | I redirected output to a file because the list is quite long. Another interface for looking at available machine images is http://thecloudmarket.com/ | 18:22 |
| smoser | Official Ubuntu images are named such that you can easily identify them. For more information on that see: https://wiki.ubuntu.com/UEC/Images/NamingConvention | 18:23 |
| smoser | You'll note that after creation, you got an 'instance id' for the new instance (i-xxxxxx) | 18:23 |
| smoser | and that it was in 'pending' state. | 18:24 |
| smoser | At some point, your instance will move from 'pending' state to 'running' and will have a hostname associated with it. | 18:24 |
| smoser | you can watch the state with | 18:24 |
| smoser | $ ec2-describe-instances | 18:24 |
| smoser | Once you see 'running' and a hostname, then you can ssh to the instance | 18:24 |
| smoser | $ ssh -i ${SSH_KEY} ubuntu@ec2-174-129-61-44.compute-1.amazonaws.com | 18:24 |
| smoser | Ubuntu images accept only the 'ubuntu' user out of the box | 18:25 |
| smoser | and that user has passwordless sudo configured. | 18:25 |
| smoser | Hoorah! now you've got root access to your very own machine in the cloud, and it cost you $0.085 USD | 18:25 |
| smoser | now, you can shut that instance down with 'shutdown' or | 18:26 |
| smoser | $ ec2-terminate-instances i-xxxxxx | 18:26 |
| smoser | we'll pause for questions on the above | 18:26 |
| smoser | (i realize theres a lot for "getting started") | 18:26 |
| smoser | ok. going on then. | 18:27 |
| smoser | === What you can do with it === | 18:27 |
| smoser | So, now that you've created this new machine and have access to it, what can you do? | 18:27 |
| smoser | * Test : We make night nightly builds of the development release, and semi-nightly builds of previous releases available. | 18:27 |
| smoser | This makes it extremely easy to verify a change in a development release without any risk to your system | 18:28 |
| smoser | * develop | 18:28 |
| smoser | If you don't have access to hardware supporting kvm, or just don't want to bother with keeping multiple images around of old releases or development, you can easily start up an instance, and start developing on it. | 18:28 |
| smoser | When you're done, you just throw it away. A new one is only seconds away. | 18:28 |
| smoser | As an ubuntu developer, those two items above are the most interesting to me. My laptop is not normally in a situation where it has a clean environment available for test or build. | 18:29 |
| smoser | * Make your own images / devpay | 18:29 |
| smoser | You can make your own images and register them, and then boot your personal image. | 18:29 |
| smoser | You can "Make Money!" using DevPay (http://aws.amazon.com/devpay/). If you add value to an image, enough that people will pay to use it, its easy to charge them for using their image. | 18:29 |
| smoser | I've not experimented at all with devpay, but it is a really easy way that you can make software appliances available to others and get paid! | 18:30 |
| smoser | I'll pause again for questions... | 18:30 |
| smoser | === Some warnings about issues you might find === | 18:31 |
| smoser | The thing that most sticks out as different from other environments is that the root filesystem is *gone* once you shut a system down. | 18:31 |
| smoser | Amazon makes basically no promises for data stored on the "ephemeral storage" of an instance. | 18:31 |
| smoser | hardware failure that causes an instance to die would cause data there to be lost | 18:32 |
| smoser | If you want to keep data that you create on an instance, you have a few options: | 18:32 |
| smoser | * Use S3 storage (from amazon) | 18:32 |
| smoser | * Use EBS (and connect an ebs volume to your instance with ec2-attach-volume) | 18:32 |
| smoser | * Use a (new) EBS Root instance. | 18:32 |
| smoser | In this new type of instance, your / is on a EBS volume, and you can stop your instance, and access that volume from another instance. | 18:32 |
| smoser | EBS Root Instances make the EC2 cloud much more similar to a "pay as you go Virtual Private Server" | 18:33 |
| smoser | We are planning on creating EBS volume IDs for Lucid, and Eric Hammond has made EBS boot AMIs available for Karmic and Hardy (http://alestic.com/2010/01/public-ebs-boot-amis-for-ubuntu-on-amazon-ec2) | 18:33 |
| smoser | The "best practice" for using ec2 instances is to make all setup of your environment automated | 18:34 |
| smoser | so that you can start up new instances on failure or demand. | 18:34 |
| smoser | pause again. questions ? | 18:34 |
| smoser | === Some Miscellaneous Things I'd like to mention === | 18:35 |
| smoser | * User Data | 18:35 |
| smoser | Booting an exact copy of an instance over and over again is hardly any fun, and arguably not much use | 18:35 |
| smoser | In almost all cases, you will want to make some modifications to the instance immediately after boot. | 18:35 |
| smoser | You can do this by waiting util the machine is running, and ssh'ing in and running a set of commands. | 18:36 |
| smoser | But that may be less than idea. | 18:36 |
| smoser | This is where "User Data" comes in. User data is provided to the instance via the '--user-data' or '--user-data-file' arguments to ec2-run-instances. | 18:36 |
| smoser | When the instance boots, it reads the user-data an can change its behavior based on that content. | 18:37 |
| smoser | When an ubuntu instance starts, if user-data begins with '#!' then it will be executed like an rc.local script. | 18:37 |
| smoser | This means that you can put together a shell script to turn the "base AMI" into your personal development box and pass that script as user-data to the instance on startup. Then, when you ssh in, it will be all set up. | 18:38 |
| smoser | This is obviously quite useful for all sorts of purposes | 18:38 |
| smoser | We're adding features to how user-data is handled for lucid (https://wiki.ubuntu.com/ServerLucidCloudBoothooks and https://wiki.ubuntu.com/ServerLucidCloudConfig for more information). | 18:38 |
| smoser | The big new feature will be the ability to configure the image on first boot via "config" syntax rather than a shell script. | 18:38 |
| smoser | * Spot Instances | 18:39 |
| smoser | If you're like me, you're cheap. Spot instances allow you to run an instance in basically the same way as "normal instances" but at a reduced price. | 18:39 |
| smoser | The catch is that if there is high demand, your instance may be terminated. | 18:39 |
| smoser | I've began using spot instances for most of my tests. | 18:39 |
| smoser | When you make a request for a "spot instance" you state the maximum you will pay for that instance. The rate you are charged varies based on demand for those resources. | 18:40 |
| smoser | if the rate goes above your maximum, your instance is terminated. | 18:40 |
| smoser | * Desktop In the Cloud | 18:40 |
| smoser | In the lucid cycle, the Desktop team wanted to take advantage of the easily created test/reproduce/verify environment | 18:41 |
| smoser | so we started creating "desktop in the cloud" builds. | 18:41 |
| smoser | Those are created daily and registered to ec2 | 18:41 |
| smoser | http://uec-images.ubuntu.com/desktop/lucid/ for relevant AMIs. | 18:42 |
| smoser | These images have a full gnome desktop installed and can be accessed via an NX client. If you've never used NX before, it is surprising how well it performs. Literally, you may forget that you are using a remote desktop. | 18:42 |
| smoser | http://www.stgraber.org/2009/12/12/ubuntu-desktop-cloud-now-working-lucid has information on how to use these. | 18:43 |
| smoser | QUESTION: smoser, when you use spot instances for your testing, what rate do you typically offer to pay, and how often do your instances get terminated? | 18:43 |
| cjohnston | < kamalmostafa> QUESTION: smoser, when you use spot instances for your testing, what rate do you typically offer to pay, and how often do your instances get terminated? | 18:43 |
| smoser | I've never actually had one terminated. | 18:44 |
| smoser | http://cloudexchange.org/ has graphs of spot prices historically | 18:44 |
| smoser | and you can get that same data with : ec2-describe-spot-price-history | 18:44 |
| smoser | what I do is take the high price available and raise it by 10% | 18:45 |
| smoser | you'll notice spikes in the graphs at cloudexchange | 18:45 |
| smoser | this is where (I think) amazon had higher demand for non-spot instances and explicitly kicked off most spot instances | 18:46 |
| smoser | On average, spot instances run ~ 40% of the regular price. | 18:46 |
| smoser | This is all I had prepared, so the floor is open to questions | 18:47 |
| smoser | Questions | 18:47 |
| smoser | oh. One other thing I should mention | 18:48 |
| smoser | i can't believe i didn't plug UEC | 18:48 |
| smoser | UEC (Ubuntu Enterprise Cloud) is based on Eucalyptus and allows you to easily run a ec2 compatible cloud inside your own data center. | 18:48 |
| smoser | This has all sorts of neat uses. | 18:49 |
| cjohnston | < zul> QUESTION: how are the images built? | 18:49 |
| smoser | zul, thanks for asking :) | 18:49 |
| smoser | We build our images using vmbuilder | 18:49 |
| smoser | The complete setup for building identical images is available starting at : https://code.launchpad.net/~ubuntu-on-ec2/vmbuilder/automated-ec2-builds | 18:50 |
| smoser | and code we use to publish the builds is at https://code.edge.launchpad.net/~ubuntu-on-ec2/ubuntu-on-ec2/ec2-publishing-scripts | 18:50 |
| cjohnston | < dhillon-v10> smoser: how is the upcomming HTML5 going to affect the server cloud | 18:51 |
| smoser | hm.. I dont know. | 18:51 |
| smoser | Do you think it will increase demand for server resources? or reduce demand? | 18:51 |
| smoser | I guess I need more context on the question | 18:52 |
| cjohnston | < dhillon-v10> smoser: probably decrease the demand because now users can do more stuff offline | 18:52 |
| smoser | Maybe, but by the same note, the more you depend on "the cloud" the more important reliable resources are | 18:53 |
| smoser | Well, I guess to wrap up... | 18:54 |
| cjohnston | < dhillon-v10> smoser: possibly :) | 18:54 |
| cjohnston | < kamalmostafa> QUESTION: Can EC2 machine images be used by UEC installations? (either with our without incurring charges at Amazon)? | 18:54 |
| smoser | kamalmostafa, yes | 18:54 |
| smoser | The only real difference is that UEC uses KVM and EC2 is xen based. | 18:55 |
| smoser | so the end result is that you'll have a different kernel, but ubuntu creates "ubuntu sauce" kernels for both varieties | 18:55 |
| smoser | i see UEC as very useful for developing machine images that can then be deployed publically on ec2 | 18:56 |
| smoser | (and for other purposes too) | 18:56 |
| cjohnston | < kamalmostafa> QUESTION: Does one need an Amazon AWS account at all, in order to use those images in a UEC installation? | 18:56 |
| smoser | kamalmostafa, no | 18:56 |
| smoser | UEC is completely stand alone. | 18:56 |
| smoser | The requirements for it are basically just having hardware available. | 18:56 |
| smoser | you can create a "cloud" with 2 systems | 18:56 |
| smoser | and the ubuntu-server ISOs have an easy option for doing so. | 18:57 |
| smoser | I've probably represented UEC poorly here... maybe next developer week we need a UEC class | 18:57 |
| smoser | wrapping up... | 18:58 |
| cjohnston | < lbrinkma> QUESTION: Whats the api command for creating a spot instance? | 18:59 |
| smoser | If you're a ubuntu developer, I strongly urge you to take a look at using EC2 for easily running releases of ubuntu that you have no worry about destroying. | 18:59 |
| smoser | ec2-request-spot-instances is the api command | 18:59 |
| smoser | it basically looks like 'ec2-run-instances' with some extra args. | 18:59 |
| smoser | I've done sone work to make a wrapper 'ec2-cheap-run-instances'. It needs some work, but then is a dropin replacement for ec2-run-instances | 19:00 |
| smoser | http://smoser.brickies.net/git/?p=misc-starter-tools.git;a=blob;f=ec2/ec2-cheap-run-instances;hb=HEAD | 19:00 |
| smoser | If you're not a ubuntu developer but want to do "real stuff" with EC2, there are loads of people running businesses and servers on EC2's cloud. I urge you to look into it further, and,of course, I urge you to start with the Official Ubuntu Images | 19:01 |
| smoser | If you have questions or need assistance, please feel free to ping me #ubuntu-server or elsewhere. | 19:02 |
| cjohnston | Thanks smoser ! | 19:02 |
| cjohnston | kirkland, your next | 19:03 |
| kirkland | cjohnston: hi, thanks. | 19:03 |
| kirkland | alright, so I'm here to talk about KVM | 19:03 |
| kirkland | which is the recommended virtualization hypervisor for Ubuntu | 19:03 |
| kirkland | it's similar, at least in principle to other hypervisors like VMWare, Xen, or VirtualBox (among many others) | 19:04 |
| kirkland | this talk is structured as follows ... | 19:04 |
| kirkland | a) a brief description/explanation about Virtualization, and how KVM works | 19:04 |
| kirkland | b) a few simple examples for launching virtual machines with KVM (that you can try at home now) | 19:05 |
| kirkland | c) and finally, some "KVM hacks", to do some fun, or more interesting things with your VMs | 19:05 |
| kirkland | so for starters, check to see if your CPU supports KVM | 19:06 |
| kirkland | egrep "flags.*:.*(svm|vmx)" /proc/cpuinfo | 19:06 |
| kirkland | you need to see if your CPU supports either the svm or the vmx flag | 19:06 |
| kirkland | these are the AMD and Intel designations for hardware supported virtualization | 19:06 |
| kirkland | quick poll in #ubuntu-classroom-chat, hands up o/ if you have cpu support for kvm | 19:07 |
| kirkland | if you don't have hw support for KVM, you'll probably need to use something else (like Xen or VirtualBox) to do virtualization | 19:07 |
| kirkland | unfortunately, I'm not going to cover those in this talk | 19:08 |
| kirkland | okay so KVM ... | 19:08 |
| kirkland | kvm is actually supported through a driver in the linux kernel | 19:08 |
| kirkland | so if you have cpu support, you can "sudo apt-get install qemu-kvm" | 19:08 |
| kirkland | once that installs, you should have a device, /dev/kvm | 19:09 |
| kirkland | this is an interface that the kernel provides to the userspace kvm program | 19:09 |
| kirkland | basically, this allows virtual machines to runs some instructions *directly* on your real cpu | 19:10 |
| kirkland | rather than running those instructions on an emulated CPU | 19:10 |
| kirkland | which is what qemu alone (ie, without kvm) does | 19:10 |
| kirkland | needless to say, emulation is slow | 19:10 |
| kirkland | running on real hardware is much faster! | 19:10 |
| kirkland | for this reason, KVM is really smoking fast virtualization | 19:10 |
| kirkland | however, we still need parts of QEMU | 19:11 |
| kirkland | QEMU = "quick emulator" | 19:11 |
| kirkland | it provides what we call "the driver model" for KVM | 19:11 |
| kirkland | it emulates the rest of the things that aren't accelerated (yet) | 19:11 |
| kirkland | like the sound card, and the video card | 19:11 |
| kirkland | so there are a lot of ways to launch virtual machines with kvm | 19:12 |
| kirkland | you can launch them through the command line | 19:12 |
| kirkland | which is what I generally do, as a developer | 19:12 |
| kirkland | or through one of the graphical front ends | 19:12 |
| kirkland | virt-manager is probably the most popular front end | 19:13 |
| kirkland | i encourage you to try virt-manager out, if gui's are your thing | 19:13 |
| kirkland | it's pretty straight-forward, wizard/menu driven | 19:13 |
| kirkland | it prompts you for various options about your vm creation | 19:13 |
| kirkland | and then, ultimately, just runs a really long kvm line :-) | 19:14 |
| kirkland | it does all of this using the libvirt library | 19:14 |
| kirkland | libvirt provides a common library interface to multiple different virtualization backends (like kvm, xen, and others) | 19:14 |
| kirkland | to get a quick example up and running, let's use "testdrive" | 19:14 |
| kirkland | anyone here already running Lucid on the machine they're going to use to try out these examples? (o/ hands up in -chat) | 19:15 |
| kirkland | if so, just "sudo apt-get install testdrive" | 19:15 |
| kirkland | otherwise, you'll need to install testdrive from the PPA linked to from http://launchpad.net/testdrive | 19:16 |
| kirkland | testdrive is just a wrapper that rsync's or zsync's an Ubuntu ISO and launches it in a KVM with a good set of options | 19:16 |
| kirkland | it's a nice way of "testdriving" the daily Ubuntu desktop or server ISOs in a virtual machine | 19:17 |
| kirkland | hopefully you have an Ubuntu *.iso file lying around somewhere | 19:18 |
| kirkland | otherwise, you're going to need to download one to do any of the following examples | 19:18 |
| kirkland | fortunately, I do ... | 19:18 |
| kirkland | and testdrive can either run against a http/ftp/rsync able ISO | 19:18 |
| kirkland | or against a local file | 19:18 |
| kirkland | $ testdrive -u ./lucid-desktop-amd64.iso | 19:19 |
| kirkland | so that's what I'm running right now | 19:19 |
| kirkland | and so the desktop live cd installer then pops up in a new window | 19:19 |
| kirkland | let's look at what KVM was launched with | 19:20 |
| kirkland | $ ps -ef | grep kvm | 19:20 |
| kirkland | kirkland 11395 11364 99 13:19 pts/12 00:00:35 kvm -m 512 -cdrom /local/virt/iso/lucid-desktop-amd64.iso -drive file=/home/kirkland/.cache/testdrive/img/testdrive-disk-nvqebh.img,if=virtio,index=0,boot=on -usb -usbdevice tablet -net nic,model=virtio -net user -soundhw es1370 | 19:20 |
| kirkland | the options are mostly straightforward | 19:20 |
| kirkland | but testdrive launched with a decent set of options without you having to go figure all of these out | 19:20 |
| kirkland | but since you're here to learn about KVM, let's look at them :-) | 19:21 |
| kirkland | -m 512 says give this virtual machine 512MB of memory | 19:21 |
| kirkland | which is enough to use the LiveCD | 19:21 |
| kirkland | my VM is now up and running the liveCD | 19:21 |
| kirkland | i can launch a terminal in the VM and run "free" | 19:21 |
| kirkland | and see that yes, this VM has ~512MB of memory | 19:21 |
| kirkland | -cdrom tells KVM where to find the ISO to boot | 19:22 |
| kirkland | -drive tells KVM where to find the hard disk | 19:22 |
| kirkland | now there's a bit more to this | 19:22 |
| kirkland | in particular, take note of if=virtio | 19:22 |
| kirkland | this VM is using "virtio" for the hard disk driver | 19:22 |
| kirkland | this is *much* faster for KVM to use, than say, scsi | 19:23 |
| kirkland | virtio is a "paravirtual" disk driver | 19:23 |
| kirkland | kvm is able to write to the backing disk image more directly, through a special, virtio feature in the host's kernel | 19:23 |
| kirkland | you can also see where the .img file is stored (in ~/.cache/testdrive/*img in this case | 19:23 |
| kirkland | it's index=0 which puts this disk at /dev/vda | 19:24 |
| kirkland | (note that the disk would be /dev/sda if the if=scsi | 19:24 |
| kirkland | ) | 19:24 |
| kirkland | and boot=on makes kvm able to boot from this disk | 19:24 |
| kirkland | -usb -usbdevice tablet is just a neat little hack that allows you to move your mouse cursor in and out of the KVM sdl window more smoothly | 19:25 |
| kirkland | totally optional, but I really like it | 19:25 |
| kirkland | -net nic,model=virtio is the network adapter | 19:25 |
| kirkland | again, we're using virtio for networking | 19:25 |
| kirkland | which, like virtio disk, is a paravirtual networking driver, allowing the vm to talk through the host's network adapter *much* faster | 19:26 |
| kirkland | like 10x faster | 19:26 |
| kirkland | -net user is the type of networking, there are a lot of different options here, but this is the simplest for quick outgoing network access | 19:26 |
| kirkland | and -soundhw is the emulated sound device | 19:26 |
| kirkland | okay ... was anyone able to get a desktop LiveCD VM launched (either through testdrive or on your own)? hands up in -chat o/ | 19:27 |
| kirkland | ah, okay, I see a question about KVM in the BIOS | 19:27 |
| kirkland | let me address that .... | 19:27 |
| kirkland | first, run "kvm-ok" on your host | 19:28 |
| kirkland | that should help detect *most* issues why KVM might not work | 19:28 |
| kirkland | first, it does that grep I asked you to do earlier | 19:28 |
| kirkland | next, it does a few sanity checks, including grepping for a really annoying message in BIOS | 19:28 |
| kirkland | some desktop/laptop manufacturers ship machines that *have* CPUs that support virtualization (VT) | 19:29 |
| kirkland | *but* they disable it in BIOS | 19:29 |
| kirkland | the widespread rumour is that M$ pressures them to do so | 19:29 |
| kirkland | in any case, we all suffer with this | 19:29 |
| kirkland | so if you enter your bios, look in CPU options, and you should see something about "enabling VT, Virtualization Technology" | 19:29 |
| kirkland | turn that on, physically power off, then back on | 19:30 |
| kirkland | and then re-run kvm-ok, and you should be good to go | 19:30 |
| kirkland | cool, looks like we have a few people with running VMs | 19:30 |
| kirkland | now, go ahead and perform an installation (if you like) | 19:30 |
| kirkland | and your desktop image should be installed into that single disk image | 19:30 |
| kirkland | it'll take a little while to complete | 19:31 |
| kirkland | okay, mine's going in the background | 19:32 |
| kirkland | the fun thing is that you can do *anything* in this VM without affecting your stable desktop | 19:32 |
| kirkland | for that reason, KVM is wonderful for developers and testers | 19:32 |
| kirkland | i did all of my development of Ubuntu's Encrypted Home Directory feature, for instance, in hundreds of VMs that I created and destroyed | 19:33 |
| kirkland | have you ever wanted to cd / && sudo rm -rf / ? | 19:33 |
| kirkland | be my guest, in your VM :-) | 19:33 |
| kirkland | anyway, you can do some really constructive (or destructive) things in VMs | 19:33 |
| kirkland | let's pause for a few questions, now, while our installations are going, and before we get into some advanced features | 19:34 |
| kirkland | please post your questions in #ubuntu-classroom-chat, prepended with: QUESTION | 19:34 |
| kirkland | <rmunn|lernid> QUESTION: If I have an existing VM created in VirtualBox and/or VMWare, is it possible to convert it to run under qemu-kvm instead? | 19:34 |
| kirkland | rmunn|lernid: possibly ... or at least you can get very, very close | 19:35 |
| kirkland | rmunn|lernid: mainly, you need to track down the disk image, where it installed into | 19:35 |
| kirkland | rmunn|lernid: assuming it's a standard disk format, you should be able to launch | 19:35 |
| kirkland | rmunn|lernid: you might have to recreate some of the meta data, though | 19:36 |
| kirkland | rmunn|lernid: like how much memory the VM gets | 19:36 |
| kirkland | rmunn|lernid: but if you find the disk image itself, make a backup copy of the disk image (while the VM is *not* running), cp it to /tmp/foo.img, for instance | 19:36 |
| kirkland | rmunn|lernid: and then just run "kvm -m 512 -hda foo.img" | 19:36 |
| kirkland | rmunn|lernid: if that works (or mostly works), you'll then just need to play with getting the rest of the kvm options you want in there (like adding networking, sound, etc) | 19:37 |
| kirkland | rmunn|lernid: as for something more automated, it might well exist ... though I don't know of it | 19:37 |
| kirkland | rmunn|lernid: i think that would be a cool idea | 19:37 |
| kirkland | rmunn|lernid: file a brainstorm request, and/or a wishlist bug | 19:37 |
| kirkland | <rmunn|lernid> QUESTION: How do you save a "snapshot" of your VM's state under qemu-kvm, then revert to that snapshot later, discarding any changes since the snapshot? | 19:38 |
| kirkland | rmunn|lernid: *great* question ... i'll try to get to that in the advanced examples section | 19:38 |
| kirkland | rmunn|lernid: in case I don't get all the way through to it (as I didn't prepare that particular example), see: http://manpages.ubuntu.com/manpages/lucid/en/man1/kvm.1.html | 19:39 |
| kirkland | rmunn|lernid: search in that manpage for snapshot= | 19:39 |
| kirkland | rmunn|lernid: and -snapshot | 19:39 |
| kirkland | <kjele> QUESTION: What benefit does kvm have over virtualbox? | 19:40 |
| kirkland | kjele: for one thing, KVM is really, really, really fast | 19:40 |
| kirkland | kjele: though it requires hardware support to be really, really, really fast | 19:40 |
| kirkland | kjele: KVM is mostly maintained by the Ubuntu Server team | 19:40 |
| kirkland | kjele: and our target hardware (mostly server class hardware) has such support pretty ubiquitously | 19:41 |
| kirkland | kjele: personally, I think VirtualBox is the best alternative for users who *don't* have hardware support | 19:41 |
| kirkland | kjele: and VirtualBox has a pretty GUI | 19:41 |
| kirkland | kjele: ideally, someone in the Ubuntu Desktop team would perhaps support VirtualBox | 19:42 |
| kirkland | kjele: but as of now, it's a project in Ubuntu Universe that mostly works pretty well | 19:42 |
| kirkland | <yltsrc> QUESTION: is kvm faster than virtualbox and vmware? | 19:42 |
| kirkland | yltsrc: in my experience, and on hardware that has VT, yes, and yes. | 19:42 |
| kirkland | yltsrc: in particular, if KVM is using virtio networking and disk, yes, and yes ;-) | 19:43 |
| kirkland | <kamalmostafa> QUESTION: Does testdrive create a blank .img of a particular size before booting the specified iso? Or does the .img file somehow "grow" as needed? | 19:43 |
| kirkland | kamalmostafa: good question | 19:43 |
| kirkland | kamalmostafa: yeah, it totally does, and I'm glad you reminded me to mention the command | 19:43 |
| kirkland | kvm-img create -f qcow2 $DISK_FILE $DISK_SIZE | 19:43 |
| kirkland | that's the command testdrive uses | 19:43 |
| kirkland | if you're creating/launching custom VMs outside of testdrive, you'll need to use "kvm-img create" to create a backing disk image | 19:44 |
| kirkland | there are several supported -f formats | 19:44 |
| kirkland | usually, I use either "raw" or "qcow2" | 19:44 |
| kirkland | I strongly recommend "qcow2" whenever possible | 19:45 |
| kirkland | qcow = quick copy on write format 2 | 19:45 |
| kirkland | which allows the backing disk image to be *much* smaller than the total allocation | 19:45 |
| kirkland | testdrive randomly generates the filename | 19:45 |
| kirkland | and uses 6G for the backing disk size, by default | 19:45 |
| kirkland | (both of which can be overriden by the user (see the testdrive manpages) | 19:46 |
| kirkland | okay, my install has completed | 19:46 |
| kirkland | has anyone else's installation completed? (hands up, or % done in -chat) | 19:46 |
| kirkland | <lernid_luislopez> QUESTION: Is it possible to have live migration in KVM? | 19:46 |
| kirkland | lernid_luislopez: yes! we'll try that out shortly, actually :-) | 19:47 |
| kirkland | <yltsrc> QUESTION: is kvm ready for production? | 19:47 |
| kirkland | yltsrc: another good question ... | 19:47 |
| kirkland | yltsrc: i would say "if you're on a modern version, yes" :-) | 19:47 |
| kirkland | yltsrc: let me qualify that ... | 19:47 |
| kirkland | yltsrc: Ubuntu 8.04 LTS (Hardy) was the first of the enterprise distros to ship with KVM as the hypervisor | 19:48 |
| kirkland | yltsrc: however, the kvm version back then (and more importantly the kernel support) was very young | 19:48 |
| kirkland | yltsrc: for this reason, I backported kvm-84 to hardy-backports | 19:48 |
| kirkland | yltsrc: I'd say that version (which is Jaunty's KVM) is pretty good | 19:48 |
| kirkland | yltsrc: however, Karmic's (and Lucid's) is far more stable | 19:48 |
| kirkland | yltsrc: and KVM is ultimately what UEC (Ubuntu Enterprise Cloud and Eucalyptus) are using for cloud VMs | 19:49 |
| kirkland | yltsrc: which is production ready ;-) | 19:49 |
| kirkland | <statik-lernid> QUESTION: I used testdrive-select-iso last night and ran through a lucid desktop installation, but then i turned it off this morning. is there a way for me to get back to that machine without having to do a whole new install? | 19:49 |
| kirkland | statik-lernid: if you want to launch the backing image, yeah, sure ... you just need to find the disk image | 19:50 |
| kirkland | statik-lernid: cd ~/.cache/testdrive | 19:50 |
| kirkland | statik-lernid: cd ~/.cache/testdrive/img | 19:50 |
| kirkland | statik-lernid: ls -thalF *.img | 19:50 |
| kirkland | statik-lernid: look at the timestamps, and the file sizes, and you should be able to find the one you want | 19:50 |
| kirkland | statik-lernid: and then just launch KVM | 19:51 |
| kirkland | okay, let me tear through the rest of this | 19:51 |
| kirkland | I have a few fun hacks I'm going to just paste and tell you about, as I'm running low on time | 19:51 |
| kirkland | kvm -m 1024 -cdrom http://cdimage.ubuntu.com/daily/current/lucid-alternate-amd64.iso | 19:51 |
| kirkland | this one is fun ... | 19:51 |
| kirkland | you can actually stream the ISO over http, without downloading it :-) | 19:51 |
| kirkland | i have a pretty good ISO mirror here local on my gigabit network | 19:51 |
| kirkland | which is obviously faster than going over the internet | 19:52 |
| kirkland | so I can launch ISOs without having them locally on my small SSD hard drive in my laptop | 19:52 |
| kirkland | for *really* fast VMs, put your ISO and/or hard disk image in a tmpfs in memory | 19:52 |
| kirkland | i use tmpfs for my /tmp | 19:52 |
| kirkland | (you have to have a lot of RAM, really, to do this well) | 19:53 |
| kirkland | /etc/fstab:tmpfs /tmp tmpfs rw | 19:53 |
| kirkland | that's the line you add to /etc/fstab | 19:53 |
| kirkland | kvm -m 2048 -cdrom /tmp/lucid-desktop-amd64.iso | 19:53 |
| kirkland | that launches an ISO for me, with a *lot* of memory | 19:53 |
| kirkland | where the backing ISO is itself in my RAM :-) | 19:53 |
| kirkland | needless to say, it's pretty quick :-) | 19:53 |
| kirkland | kvm-img can create disks of *any* size | 19:54 |
| kirkland | and qcow2 will make them sparse | 19:54 |
| kirkland | so you can create, let's say, a 2PB (petabyte) disk | 19:54 |
| kirkland | kvm-img create -f qcow2 /tmp/petabyte.img 1024T | 19:54 |
| kirkland | launch a KVM with that as the backing disk and do a "sudo fdisk -l" in the VM | 19:54 |
| kirkland | it's fun to see a 1PB backing disk | 19:54 |
| kirkland | long before they're on the market :-D | 19:55 |
| kirkland | you can dynamically add a disk to a running VM | 19:55 |
| kirkland | in the vm, "sudo modprobe acpiphp" | 19:55 |
| kirkland | (note that we're enabling this by default in Lucid, so that you don't have to modprobe it on the server) | 19:55 |
| kirkland | now, in your running KVM, drop to the QEMU console, where there are all sorts of fun, interesting commands | 19:56 |
| kirkland | ctrl-alt-2 | 19:56 |
| kirkland | that'll put you at a QEMU prompt | 19:56 |
| kirkland | pci_add auto storage file=/tmp/petabyte.img,if=virtio | 19:56 |
| kirkland | then go back to your VM | 19:56 |
| kirkland | ctrl-alt-1 | 19:56 |
| kirkland | sudo fdisk -l | 19:56 |
| kirkland | that should show you your very big disk :-) | 19:57 |
| kirkland | finally, let's do a live migration ... | 19:57 |
| kirkland | now we'll do the live migration from one VM to another VM, running on the same system (your localhost) | 19:57 |
| kirkland | but if you have two machines running the same KVM version, you should be able to do this across your network too | 19:58 |
| kirkland | so first, launch a vm | 19:58 |
| kirkland | kvm -m 512 -hda /local/virt/img/lucid-desktop.img | 19:58 |
| kirkland | or use your running one | 19:58 |
| kirkland | next, launch a second kvm process WITH THE EXACT SAME PARAMETERS | 19:59 |
| kirkland | plus the -incoming option | 19:59 |
| kirkland | kvm -m 512 -hda /local/virt/img/lucid-desktop.img -incoming tcp:0.0.0.0:1025 | 19:59 |
| kirkland | and the IP address and Port | 19:59 |
| kirkland | i'm using local, so I have 0.0.0.0 | 19:59 |
| kirkland | and I arbitrarily chose 1025 | 19:59 |
| kirkland | In the source, drop to the QEMU console | 19:59 |
| kirkland | ctrl-alt-2 | 19:59 |
| kirkland | and run: | 19:59 |
| kirkland | migrate tcp:localhost:1025 | 19:59 |
| kirkland | and in a few seconds, you should see your vm move from one process to the other | 20:00 |
| kirkland | to do this across separate hosts, you need to have persistent storage across both (like NFS) | 20:00 |
| kirkland | and it requires a slightly more complicated networking setup | 20:00 |
| kirkland | but it's quite doable | 20:00 |
| kirkland | okay, I think my time is up | 20:00 |
| kirkland | hope this was useful | 20:00 |
| kirkland | i'll be around in #ubuntu-server if you have more questions | 20:01 |
| kirkland | cheers | 20:01 |
| DktrKranz | Hello everybody, and thank you to be here at Python Application packaging hosted at Ubuntu Developer Week! | 20:05 |
| DktrKranz | My name is Luca Falavigna, I'm a MOTU and Debian Developer, and I'll be your SABPDFT (Self-Appointed Benevolent Python Dictator For Today) for the next minutes :) | 20:05 |
| DktrKranz | For those who weren't here last week, I already had a session about basic Python application packaging. You can find logs here: http://irclogs.ubuntu.com/2010/01/21/%23ubuntu-classroom.html | 20:06 |
| DktrKranz | It basically covered how to handle control files in debian/, and how to easily find dependencies for your Python applications. Even if you are good in Python packaging, there could be some interesting tips to improve your kung-fu :) | 20:06 |
| DktrKranz | Today we will look at some advanced tricks to improve overall Python packages quality and stability. | 20:07 |
| DktrKranz | But let's start with a really short introduction. | 20:07 |
| DktrKranz | I refer to a "Python application" as a piece of software composed by one or more Python scripts and eventually one of more modules (or "packages", distutils calls them that way). | 20:08 |
| DktrKranz | Other kinds of software are "Python modules", which contain modules potentially useful for other programs installed in the global namespace, and "Python extensions", which usually are C source files, compiled and liked for a given Python interpreter to extend its features. | 20:08 |
| DktrKranz | But how does a Python application look like? | 20:09 |
| DktrKranz | I've created a really dumb one for the occasion. Open your favourite terminal and launch dget -u http://people.debian.org/~dktrkranz/pythontest/pythontest_0.1-1/pythontest_0.1-1.dsc | 20:09 |
| DktrKranz | Once you've downloaded it, look at pythontest-0.1 directory, you will find pythontest script, PythonTest module (or "package") and setup.py, which is Python distutils' "makefile". | 20:10 |
| DktrKranz | It's probably the smallest Python application you'll ever seen, it just prints a message, and then exits :) | 20:11 |
| DktrKranz | Invest a couple of minutes to examine the few files available, then we will move to packaging. | 20:11 |
| DktrKranz | < Omar871> QUESTION: Will "setup.py install" work? | 20:12 |
| DktrKranz | Omar871: it should by launching "python setup.py install" | 20:12 |
| DktrKranz | I didn't test that way, but it should work as expected ;) | 20:13 |
| DktrKranz | < Omar871> QUESTION: How do I use the package after installing it? | 20:15 |
| DktrKranz | it installs a script under /usr/bin, named pythontest. you should be able to launch it, but we will see it in the next minutes | 20:16 |
| DktrKranz | < statik-lernid> QUESTION: when would we choose dh --with-python-central instead of the default? | 20:16 |
| DktrKranz | statik-lernid: it's a matter of tastes. I personally prefer python-support, I read the code and I understand a bit its logic. | 20:17 |
| DktrKranz | < lucio> QUESTION: what is pycentrak and pysupport and why do i care? | 20:18 |
| DktrKranz | lucio: these are helpers which manage to expand ${python:Depends} to the right values, and byte-compile Python files during package install | 20:19 |
| DktrKranz | they do more, but that's beyond the scope of the lesson. | 20:20 |
| DktrKranz | lucio: if you are interested, a more detailed description of packaging is held in the logs of the other session, see above :) | 20:20 |
| DktrKranz | ok, let's move to the packaging itself now | 20:21 |
| DktrKranz | Compile the package (dpkg-buildpackage -uc -us) and install the resulting pythontest_0.1-1_all.deb | 20:21 |
| DktrKranz | To test it, open another console and move to a completely different location than the build directory. Keep this in mind: never test a Python software installed system-wide by launching it in the build directory. This will save you from errors, and let you discover serious problems in your packaging! | 20:22 |
| DktrKranz | Now launch "pythontest". You should now see a message on your console saying package works correctly. | 20:23 |
| DktrKranz | < Omar871> QUESTION: Is debhelper needed for this? | 20:25 |
| DktrKranz | Omar871: for this examples, yes. debhelper is explicitly listed in Depends field in debian/control | 20:25 |
| DktrKranz | < statik-lernid> QUESTION: when I dpkg -L pythontest, I see files installed to /usr/share/pyshared/. when I python -c "import PythonTest;print | 20:27 |
| DktrKranz | PythonTest.__file__" I see files located at /usr/lib/pymodules/python2.6/PythonTest/__init__.pyc. why the different paths? | 20:27 |
| DktrKranz | statik-lernid: good question. This is how python-support worsk | 20:28 |
| DktrKranz | it byte-compiles files for every supported Python version at runtime, and stores files under /usr/lib/pymodules/python*/PythonTest | 20:28 |
| DktrKranz | while package only has them under pyshared, because they are common to multiple Python versions at the same time | 20:29 |
| DktrKranz | that's the magic behind python-support and python-central :) | 20:29 |
| DktrKranz | < POX> QUESTION: shouldn't Python applications use private dirs by default (i.e. when other packages do not use their modules)? What's --install-lib | 20:30 |
| DktrKranz | and --install-scripts? | 20:30 |
| DktrKranz | POX: yes. Stay tuned for that :) | 20:30 |
| DktrKranz | Even if package works, we installed PythonTest module in Python global namespace, which is completely useless for other packages in this case. Nobody will ever want to access our PythonTest module, and we should also want to avoid collisions with other modules as well. | 20:31 |
| DktrKranz | To avoid global namespace pollution, we should install our PythonTest module in a "private directory", that is one not listed in the output shown by python -c "import sys; print sys.path" command. | 20:31 |
| DktrKranz | /usr/share/packagename is a very good choice in almost all cases, so we'll adjust our package to install modules in /usr/share/pythontest. | 20:32 |
| DktrKranz | I prepared a second package, you can grab it with dget -u http://people.debian.org/~dktrkranz/pythontest/pythontest_0.1-2/pythontest_0.1-2.dsc, and see changes I've made here: http://people.debian.org/~dktrkranz/pythontest/0.1-1_0.1-2.patch | 20:33 |
| DktrKranz | Basically, the only change I did is passing --install-lib=/usr/share/pythontest to distutils to instruct it to install PythonTest in our private directory (see the three rules files to see several methods to do that). | 20:34 |
| DktrKranz | Again, compile, install the resulting pythontest_0.1-2_all.deb, and test it by launching pythontest in the other console. | 20:34 |
| DktrKranz | You should now see something unexpected... | 20:34 |
| DktrKranz | I got "Uh-oh! There is something wrong with the package, I can not import PythonTest module!", so we probably did something bad here :) | 20:35 |
| DktrKranz | Before explaining that, let's answer to some questions. | 20:37 |
| DktrKranz | < mhall119|work> QUESTION: so the dh $@ in debian/rules is processing the pythontest.* files? | 20:37 |
| DktrKranz | mhall119|work: it's dh_pysupport (or dh_pycentral), called by dh sequencer which parsed "dh $@" | 20:38 |
| DktrKranz | you can see dh_pysupport has been called in your terminal output | 20:39 |
| DktrKranz | < Omar871> QUESTION: What about virtual env? | 20:40 |
| DktrKranz | Omar871: we don't usually use virtualenv to avoid shipping multiple files in the same package | 20:40 |
| DktrKranz | < mhall119|work> QUESTION: so I understand, dh->dh_pysupport->pythontest.postinst.debhelper | 20:41 |
| DktrKranz | mhall119|work: exactly. dh calls dh_pysupport at the given time, while dh_pysupport also generates *postinst files to handle byte-compilation of Python objects. | 20:42 |
| DktrKranz | < johnsgruber> QUESTION: In my case for the first example source package the built deb says to put the library in /usr/share/python-support/pythontest | 20:43 |
| DktrKranz | johnsgruber: do you have a python-support version << 0.90.0 ? | 20:43 |
| DktrKranz | e.g, you use an older Ubuntu/Debian release than Intrepid/Karmic, or testing/sid? | 20:44 |
| DktrKranz | Back to our problem then. | 20:46 |
| DktrKranz | Why did we get that? We moved PythonTest to a private directory, and this is good, but our pythontest script is no longer able to access it, and this is bad! | 20:46 |
| DktrKranz | This is why PythonTest is not in global namespace anymore, but pythontest script obviously thinks it is there. | 20:46 |
| DktrKranz | We have two options now: using sys.path.append() from inside of pythontest script (the one installed in /usr/bin), or moving it to the same directory where PythonTest is (/usr/share/pythontest). | 20:47 |
| DktrKranz | My preferred way to achieve this goal is the latter, I don't like playing with sys.path :) | 20:48 |
| DktrKranz | Let's download today's third package with dget -u http://people.debian.org/~dktrkranz/pythontest/pythontest_0.1-3/pythontest_0.1-3.dsc and look at the diff at http://people.debian.org/~dktrkranz/pythontest/0.1-2_0.1-3.patch | 20:48 |
| DktrKranz | This time I passed a new option to distutils: --install-script=/usr/share/pythontest, to instruct it to install our scripts in /usr/share/pythontest too. I also created a /usr/bin/pythontest symlink pointing to /usr/share/pythontest/pythontest, this way we still have pythontest in our $PATH :) | 20:49 |
| DktrKranz | Again, compile, install the resulting pythontest_0.1-3_all.deb and test it by launching pythontest in the other console. | 20:49 |
| DktrKranz | Now you should see something familiar, and more friendly :) | 20:50 |
| DktrKranz | We have PythonTest module in a private directory, and pythontest is able to import it without problem, let's celebrate! \o/ | 20:51 |
| DktrKranz | It's a little tricky at times, but you just need some practice, and things will be clearer. | 20:51 |
| DktrKranz | < mhall119|work> QUESTION: applications should always be put in a private space? | 20:51 |
| DktrKranz | mhall119|work: it's not always possible, sometimes modules are required by other packages, and having it in global namespace helps, but normally the private directory should be used | 20:52 |
| DktrKranz | in my memory, > 95% Python applications can be installed using private directory method | 20:53 |
| DktrKranz | < Omar871> QUESTION: Can you please explain to me what exactly is a "private space"? | 20:54 |
| DktrKranz | Omar871: a private directory is a directory not accessible by Python global namespace. | 20:55 |
| DktrKranz | Omar871: for example, if you launch python console and import a module (f.e. gtk, but feel free to pick another one), python interpreter will look in some directories looing for gtk module | 20:56 |
| DktrKranz | these are the ones listed by "python -c "import sys; print sys.path" command | 20:56 |
| DktrKranz | other ones are "private", given that python interpreter won't look in those searching for modules | 20:57 |
| DktrKranz | < POX> QUESTION: what if script and module name is the same? What's --install-scripts? | 20:57 |
| DktrKranz | POX: in that case, one of the quickest solutions is to manually install scripts having the same name as modules/packages (e.g. foo -> foo.py) | 20:58 |
| DktrKranz | POX: --install-scripts is another distutils option which instructs it to install files listed under scripts=[] (seen it in setup.py) to a given directory. | 20:59 |
| DktrKranz | < rmunn|lernid> QUESTION: Won't DktrKranz's approach fail once Python 2.7 is released and absolute imports (http://www.python.org/dev/peps/pep-0328/) | 21:00 |
| DktrKranz | become the default? Isn't it better to fiddle with sys.path? | 21:00 |
| DktrKranz | rmunn|lernid: nice question. I haven't had the occasion to look at relative and absolute imports yet applied to Debian packaging, but I guess something have to be adjusted for that, and sys.path mangling could become handy. | 21:02 |
| DktrKranz | < mhall119|work> QUESTION: is rules.cdbs being used in what we're currently doing? I see that your patches updated it as well | 21:03 |
| DktrKranz | mhall119|work: not directly. I included it in my example for those familiar with CDBS over dh7. I patched it to show how to handle things with different helpers. | 21:03 |
| DktrKranz | (debian/rules.olddh follows the same rule) | 21:04 |
| DktrKranz | < mhall119|work> QUESTION: is setup.py being called by anything in this setup? | 21:06 |
| DktrKranz | mhall119|work: it's called by dh_auto_install | 21:06 |
| DktrKranz | which in turn is called by dh sequencer, yay for tiny rules :) | 21:07 |
| DktrKranz | < Omar871> QUESTION: Which channel can we find you at? | 21:08 |
| DktrKranz | Omar871: several maintainer and developers interested in Python packaging are on #debian-python, OFTC network | 21:09 |
| DktrKranz | feel free to join, ask questions and look for sponsorship :) | 21:09 |
| DktrKranz | < Emilien> QUESTION: elaboration: Well, I've heard that with Quickly you can package your app, which is written in Python too... Wouldn't it be easier | 21:12 |
| DktrKranz | to use that instead of having to set everything manually up? | 21:12 |
| DktrKranz | Emilien: I've heard of Quicly, I know is active developed by some cool guys here, I didn't have the occasion to test it yet, I still prefer using my favourite editor, just because things to cut are huge. | 21:13 |
| DktrKranz | so, I can't answer to that question, but I guess if it manages things as described here today, it should be fine (but please don't completely trust helpers) | 21:14 |
| DktrKranz | < Emilien> dktrkranz: what do you mean by "things to cut are huge"? | 21:15 |
| DktrKranz | Emilien: usually, templates are prepared to fit everyone's needs, while I prefer to tailor a package to fit its needs, removing every unneeded bit. | 21:16 |
| DktrKranz | OK, we're a bit late, if you still have questions, we can move to our usual channels. Thanks everyone for coming, I had a great time! See you :) | 21:19 |
| === lucio is now known as Guest6501 | ||
Generated by irclog2html.py 2.7 by Marius Gedminas - find it at mg.pov.lt!
