Testing CSS in mozilla: where to go from here

Version 1.1

Contents

Introduction

Aims

The primary aim of Quality Assurance (QA) is to make the product better by reporting failures in sufficient detail for the engineering team to then fix these failures.

In order to report failures, they first have to be found. Finding failures forms the bulk of the work done by QA. There are several ways of finding failures, and these are discussed in detail in the next section. Testing the product directly is the most obvious way of finding failures, but there are several other techniques, for instance reading user feedback (e.g. from beta programmes).

CSS

Cascading Style Sheets (CSS) is a simple technology for styling web pages. It is designed to allow an easy separation of the stylistic aspects of a document (e.g. "green bold text") from the structural and semantical aspects of a document (e.g. "section header").

CSS is based on two fundamental concepts.

The first concept is that CSS is a tree decoration language. CSS defines a list of properties, for example 'color' or 'font-size'. Applying CSS style sheets to a tree causes each node in the tree to have a specified value for each of these properties.

The second concept is the CSS rendering model. This describes how blocks, tables, text and other layout elements are displayed, how fonts are selected, and so on.

The mapping of the decorated tree into the rendering model is what forms the majority of the CSS specification.

CSS was originally invented in 1995 by Hakon Lie and Bert Bos, and became a World Wide Web (W3C) Recommendation in late 1996. In 1998 a second version was released [2], and since then much progress has been made on a third version, which will be split into many modules for both political and technical reasons, and on publishing extensive errata for the published versions based on implementation experience.

The CSS Working Group, which is responsible for this work, consists of representatives from various different implementors of CSS user agents, users of the technology, and other interested parties.

Further reading: http://www.w3.org/Style/CSS/

Mozilla

Mozilla is a free software internet application suite. It forms the basis of products such as the Netscape browser, the "Instant AOL" consumer device, and the ActiveState Komodo IDE.

Further reading: http://www.mozilla.org/

Testing methodology

Types of coverage ("when")

There are several ways of finding failures. These form a broad spectrum of test coverage, ranging from the unsuspecting public to the simplest of targeted test cases. Let us examine each in turn.

Types of test cases ("where")

If the various techniques for finding failures gives a list of when bugs are typically found, then the various different kinds of test cases gives a list of where the failures are found.

Finding bugs

The following flowchart is a summary of this section.

                                  BETA FEEDBACK
                                        |
 EXTREMELY                             \|/
COMPLICATED        USER FEEDBACK --> WEB SITE <-- DOGFOOD
 DEMO PAGE                              |
     |                                  |
     |                                  |
    \|/                                \|/
  LIST OF -----> COMPLICATED ------> REDUCED
   BUGS   <-----    TESTS             TEST
    /|\            |                    |
     |            \|/                  \|/
     AUTOMATED TESTS                BUG FILED

CSS: what has been done so far

Existing coverage (as of September 2001)

CSS1 is thoroughly covered and methodical testing at this stage would not give a tests-to-bugs ratio that is worth the time investment. The only exception would be the list related properties.

CSS2 is less thoroughly covered. Positioning, tables, generated content and the font matching algorithm have had little testing.

Selectors, the cascade, syntax, the block box model, the inline box model, floats, colour and background related properties, the text properties, and the font properties are all well covered.

Current tests are spread across many test suites, including:

New tests

The majority of new tests should be in the areas listed as lacking tests in the previous section. These are the areas that have the least support in Mozilla.

With the recent advent of LAS, the automation system for layout tests, it would be a good idea to work on automating the many manual tests already in existence. Having done this, linking the automation with the tinderbox tests would give a good advance warning of regressions.

Writing manual QA tests for CSS

How manual QA tests are used

Tests are viewed one after the other in quick succession, usually in groups of several hundred to a thousand. As such, it is vital that:

A badly written test can lead to the tester not noticing a regression, as well as breaking the tester's concentration.

Ideal tests

Well designed CSS tests typically fall into the following categories, named after the features that the test will have when correctly rendered by a user agent (UA).

Note: The terms "the test has passed" and "the test has failed" refer to whether the user agent has passed or failed a particular test -- a test can pass in one web browser and fail in another. In general, the language "the test has passed" is used when it is clear from context that a particular user agent is being tested, and the term "this-or-that-user-agent has passed the test" is used when multiple user agents are being compared.

Ideal tests, as well as having well defined characteristics when they pass, should have some clear signs when they fail. It can sometimes be hard to make a test do something only when the test fails, because it is very hard to predict how user agents will fail! Furthermore, in a rather ironic twist, the best tests are those that catch the most unpredictable failures!

Having said that, here are the best ways to indicate failures:

These are in addition to those inherent to the various test types, e.g., differences in the two halves of a two identical renderings test obviously also shows a bug.

Tests to avoid

Techniques

In addition to the techniques mentioned in the previous sections, there are some techniques that are important to consider or to underscore.

Glossary

There are many terms which will be encountered when writing or using tests for CSS. This list is by no means complete, but should give the reader a head start.

References

  1. Cascading Style Sheets: http://www.w3.org/TR/REC-CSS2/
  2. Dogfood in the Jargon File: http://www.tuxedo.org/~esr/jargon/html/entry/dogfood.html
  3. The Mozilla Tinderbox: http://tinderbox.mozilla.org/showbuilds.cgi?tree=SeaMonkey
  4. Bugzilla: http://bugzilla.mozilla.org

Contributors

Ian Hickson <ian@hixie.ch>