[16:02] <cr3> hi folks! time for a Checkbox tutorial
[16:03] <cr3> first, a poll: who here has used Checkbox (previously known as hwtest) in the menu: Administration -> Hardware Testing?
[16:04] <ara> o/ ;-)
[16:04] <schwuk> .o/
[16:04] <cr3> ara and swhcuk: woohoo! thanks for contributing hardware and test information to Launchpad!
[16:04] <danbh_intrepid> I have, even posted once on the launchpad page?
[16:05] <cr3> danbh_intrepid: thanks, much appreciated!
[16:05] <gueafrt5> I have, on hardy :p
[16:06] <cr3> Checkbox is a testing framework which makes it possible to run tests and then submit the results somewhere, such as Launchpad for example.
[16:06] <cr3> The tests can be automated or manual, they can also integrate third party suites such as autotest from the Linux kernel, LTP from the Linux Testing Project, LSB from the Linux Standard Base, etc.
[16:07] <cr3> The tests can be defined very simply in a flat text file. Lets look at one example:
[16:07] <cr3> name: audio
[16:07] <cr3> plugin: manual
[16:07] <cr3> categories: laptop, desktop
[16:07] <cr3> requires: alsa.type == 'control'
[16:07] <cr3> command: audio_playback $data_path/audio_playback.wav
[16:07] <cr3> _description: Testing detected sound card: . $(audio_playback --view) . Did you hear a sound?
[16:08] <cr3> The file which contains this test constitutes a suite, within which the name: field above must be unique. it's simply an identifer, no need to worry about it too much.
[16:09] <cr3> The plugin: field refers to plugins within the Checkbox framework which provides "manual" and "auto" by default.
[16:09] <cr3> The manual plugin will, as its name implies, prompt the user for a question.
[16:09] <cr3> The auto plugin will run a test automatically from the command: field described in a moment.
[16:10] <cr3> the categories: field limits testing to specific categories of systems which can be: laptop, desktop or server. at some point, maybe thin client will be added
[16:11] <cr3> The requires: field is a tricky field which defines whether a test should be run or not. For example, testing Firefox would only really make sense if the package was actually installed. Furthermore, it would not be appropriate behavior for Checkbox to autmoatically install a package just because one of the many tests might need it.
[16:12] <ara> cr3: and in this case, the alsa.type?
[16:12] <cr3> So, the requires field makes it possible to limit the running of a test based on what is called a "registry" which will be described later. Suffice it to say for now, there are registries for packages and devices, so you can limit based on those.
[16:13] <cr3> ara: alsa.type is provided by the device registry which maps lshal information. so, try running this: lshal | grep alsa.type. if one of the lines output matches 'control', as defined in the test above, then the audio test will be run
[16:14] <cr3> For the Firefox example given above, the requires: field would look like: package.name == 'firefox'
[16:15] <cr3> For the advanced user, the requires: field could also provide boolean expressions and lists: package.name == 'firefox' && system.formfactor == 'notebook'
[16:15] <cr3> err, make that 'laptop' rather than 'notebook'
[16:16] <cr3> So, moving on to the command: field which describes the actual command to run. This field is not absolutely necessary for a manual test however, in this particular case, it happens to be useful in order to play a sound.
[16:17] <cr3> As some might notice, the command: field contains a $data_path environment variable defined by the manual and auto plugins which have a data path, which is simply a directory where data files are stored for testing purposes such as audio files in this particular case.
[16:18] <cr3> If more environment variables might be needed, these can be defined in another plugin which could be defined in the plugin: field.
[16:19] <cr3> Finally, the description: field provides a simple description for the test. In the case of manual tests, this is the actual question asked to the enduser. In the case of automated tests, this is strictly informational.
[16:19] <cr3> As some might notice again, the description: field contains variables which might be interpolated before being presented to the enduser.
[16:20] <cr3> For those familiar with shell programming, the description is actually interpreted as a here document so more elaborate scripting can be done within the description for ultimate flexibility.
[16:22] <cr3> Now that we understand most of the relevant fields for defining a test, these can be defined in a suites test file. For hwtest, these are available under the directory: /usr/share/hwtest/suites
[16:22] <cr3> All the files under that directory are automatically parsed for tests, so you can simply drop new tests in that directory
[16:23] <cr3> Here is what a minimal automated test would look like:
[16:23] <cr3> name: cr3
[16:23] <cr3> plugin: manual
[16:23] <cr3> description: Is cr3 available in #ubuntu-classroom?
[16:23] <ara> :-)
[16:23] <cr3> That's it, three lines!
[16:24] <cr3> However small that might be, three lines could be painful to write for the hundreds if not thousands of tests available in other third party suites
[16:25] <cr3> This leads to the concept of plugins which can totally modify the behavior of Checkbox and how it manages tests.
[16:25] <cr3> By the way, a plugin is also how Checkbox can be modified to submit test results to another location than Launchpad such as your filesystem for example.
[16:26] <cr3> A plugin is simply a Python module which registers for events. A minimal plugin would look like:
[16:28] <cr3> class Cr3Plugin(Plugin):
[16:28] <cr3> def register(self, manager):
[16:28] <cr3> super(Cr3Plugin, self).register(manager)
[16:28] <cr3> self._manager.reactor.call_on("report", self.report)
[16:28] <cr3> def report(self):
[16:28] <cr3> self._manager.reactor.fire("report-cr3", "cr3 is in #ubuntu-classroom")
[16:29] <cr3> factory = Cr3Plugin
[16:29] <cr3> ok, that output didn't look too good in IRC but it should be simple enough to grock with some explanations
[16:29] <cr3> First, the new plugin should inherit from the Plugin class.
[16:30] <cr3> Second, it should register to be called on particular events such as the "report" event in this particular example.
[16:30] <cr3> The registration is done with this call: self._manager.reactor.call_on("report", self.report)
[16:30] <cr3> So, upon the "report" event, the self.report method will be called.
[16:31] <cr3> Third, in the report method is defined to perform some action. In this particular case, the event handler is dispatching a new event called "report-cr3"
[16:32] <cr3> So, anyone who has registered for this particular event will be called.
[16:33] <cr3> A neat trick is that registering for events can be done with regular expressions too! So, the above call could be rewritten to register for all events starting with "report":
[16:33] <cr3> self._manager.reactor.call_on("report.*", self.report)
[16:34] <cr3> Finally, the plugin should provide a global factory variable which will be the hook for instantiating your plugin.
[16:35] <cr3> Then, in a similar way as test suites, you can simply drop your plugin into the corresponding directory and it will be automatically parsed by the Checkbox framework.
[16:35] <cr3> The default directory is: /usr/share/hwtest/plugins
[16:35] <cr3> Some of the interesting features provided by plugins include:
[16:37] <cr3> - Integration with configuration files where some variables can be formalized as required or optional using the required_attributes and optional_attributes class variables.
[16:37] <cr3> - Persistence where some information can be preserved seemlessly across invocations of the Checkbox application.
[16:38] <cr3> - Access to the registry, which will be described shortly, in order to query information about the system.
[16:39] <cr3> That's about it, very few paradigms are meant to be provided  by Plugins in order to lower the barrier to entry as much as possible.
[16:40] <cr3> err, that's about it regarding plugins :)
[16:41] <ara> now registries, i guess :-)
[16:41] <cr3> The registry, which has been mentionned in a couple occasions already, provides a consistent way of accessing hierarchical information in the system.
[16:41] <cr3> ara: you read my mind
[16:42] <cr3> It is meant to expose a single rooted data structure which can be accessed consistently regardless of where the information might be stored
[16:42] <cr3> For example, cpuinfo.flags would access /proc/cpuinfo
[16:43] <cr3> Another example, packages.firefox.version would access dpkg output
[16:44] <cr3> One of the motivations for provding this kind of data structure is to be able to define the requires: field in tests in a consistent way
[16:44] <cr3> Another is to provide plugins with a convenient way to obtain and query information about the system.
[16:44] <ara> I guess registries require, in general, sudo permissions, don't they?
[16:45] <cr3> In fact, a very cool way of querying the registry is by using registry_eval_recursive! This makes it possible to eval an expression across the whole registry
[16:46] <cr3> ara: the concept of registries simply accesses information. so, if that information requires sudo privileges, then yes.
[16:47] <cr3> ara: however, for the two examples given above using /proc/cpuinfo and dpkg, then simple user privileges are fine
[16:47] <ara> cr3: thanks for the clarification :-)
[16:48] <cr3> A new registry is defined in a similar way as plugins in terms of:
[16:49] <cr3> 1. Deriving a class but, instead of Plugin, the Registry class in this case.
[16:49] <cr3> 2. Creating a factory global variable as a hook.
[16:49] <cr3> 3. Copying the new registry as a Python module under a directory, /usr/share/hwtest/registries in this case.
[16:50] <cr3> The main difference is how the Registry class is derived. So, lets look at yet another example:
[16:50] <cr3> class Cr3Registry(Registry):
[16:50] <cr3>   def items(self):
[16:50] <cr3>     return (("foo": 1), ("bar": 2))
[16:50] <cr3> factory = Cr3Registry
[16:51] <cr3> Then, if I copy this code to a Python module called cr3.py under the registries directory, I could then query the cr3 namespace.
[16:51] <cr3> For example, cr3.foo would be 1
[16:52] <cr3> So, the same concept could be extended to lshal, /proc/cpuinfo, /proc/meminfo, dpkg, lsb_release, etc.
[16:53] <cr3> In other words, the new Registry simply needs to define an items method and then the RegistryManager handles the rest.
[16:54] <cr3> Another similarity with Plugins is that a repository benefits from integration with configuration files, which will be the final building block to understanding Checkbox.
[16:55] <cr3> The configuration file(s) in Checkbox are mostly responsible for defining the behavior of the application.
[16:55] <cr3> Checkbox tries to hard code as little policy as possible and delegates this responsibility to the configuration file(s)
[16:56] <cr3> These files are in an .ini format and stored in the directory /etc/hwtest.d
[16:56] <cr3> The default section of the configuration file, which is typically at the top, defines just a handful of variables:
[16:57] <cr3> version: the version of the configuration file. this is stored in the configuration rather than the application because, as mentionned earlier, the configuration is mostly responsible for the behavior of the application
[16:58] <cr3> includes: other configuration files to include. for example, the GTK and CLI configuration files which define the interface to use also include the base configuration
[16:58] <cr3> plugins: name of the sections in the configuration file(s) which define plugin information, such as the directory where they are stored
[16:58] <cr3> registries: similar as plugins, but defines registry information
[16:59] <cr3> That's it for the default section. The rest strictly depends on what has been defined above.
[17:00] <cr3> For example, if I defined: plugins = cr3/plugins. Then, I would define a section of the same name with plugin related attributes. For example:
[17:00] <cr3> [cr3/plugins]
[17:00] <cr3> directories = /tmp/plugins
[17:00] <cr3> persist_filename = /tmp/plugins.persist
[17:00] <cr3> blacklist = cr3
[17:01] <cr3> So, all the plugins under /tmp/plugins will be parsed exacept for the cr3.py plugin because it has been blacklisted.
[17:02] <cr3> I'm running out of time so that's about it for the configuration file, the rest could be easily understood from the existing files distributed with the Checkbox (previously known as hwtest) project.
[17:02] <cr3> For more information, please visit the project available on Launchpad: https://launchpad.net/checkbox
[17:02] <cr3> Enjoy!
[17:03] <ara> thanks for the session!
[17:13] <cody-somerville> \o/
[17:53] <wasikevin> cr3: when is the class?
[17:53] <cr3> wasikevin: class ended an hour ago, sorry dude
[17:54] <cr3> wasikevin: the logs should be available as detailed in the topic: https://wiki.ubuntu.com/UbuntuDeveloperWeek
[17:54] <wasikevin> cr3: thx
[17:54] <wasikevin> cr3: sorry, I just noticed it.
[17:59] <cr3> wasikevin: if you have any questions, please ask in #ubuntu-testing in case there's another class going on in here