Conformance Test Suites for mobile web technologies

The Mobile Web Test Suites Working Group has analyzed the following conformance tests suites. See also some of these tests suites integrated in a mobile-friendly test harness.

Spec	Test Suite	Status	Coverage	Test assertions	Documentation	Test harness	Public Availability	Maintenance process	License	Comments	Usability on a mobile device
XHTML Basic 1.1 / XHTML MP 1.2 [pdf]	XHTML TS 1(20061020), XHTML TS 2(20060404), XHTML TS3(20060707), Enabler Test Specification (Conformance) for XHTML Mobile Profile	1.1: Approved OMA specification (OMA Browsing 2.2) 1.2: Candidate Version (not yet approved)	~250 tests, but no indication of actual coverage;	referenced in enabler and tests suites, but not publicly available apparently	Each test case describes its expected output that the tester is supposed to compare to what she gets on her device/user agent; there seems to be a more complete accompanying documentation, but it is not publicly available	Limited navigation from one test to another	Made public in January 2007	Problems can be reported to the OMA Problem Reporting system, but that system is reserved to OMA members - although anyone is supposedly allowed to report a problem (@@@); the development of the test suite follows the established OMA process	OMA Test Code License Agreement allows for derivative work
SVG Basic and Tiny 1.1	W3C Scalable Vector Graphics (SVG) 1.1 Test Suite	Stable and maintained (last update Dec 13 2006)	Number of tests (Basic/Tiny): 256/152; assertion coverage is undefined	Tests links to the chapter of the specification under test.	Test cases include the documentation that has: - test name - link to the chapter of the specification (Basic and Tiny tests refer to full version of the spec) - SVG and PNG images that need to be compared by sight - Short description explains the conditions - Detailed description that explains what you can see on the pictures and why.	Two test harnesses are available - frame-based harness, with the reference PNG and the test SVG presented side by side, together with instructions (for browser implementations and plug-ins) (harnesses for Basic and Tiny) - an SVG-based harness which uses the SVG image element to include the actual test, and provides navigation links.	Publicly available through http. Suite may be downloaded also in zip archive, but it does not contain harness.	Feedback on the tests and suggestions for new tests may be sent to svg-testsuite- [email protected].	W3C	Test suite is well organized. But it contain the interactive tests only.
SVG Tiny 1.2	Test suite (Beta), Sep 7 2007	Beta, still being worked on	Unknown, but being worked on
CSS Mobile Profile	Test cases for CSS Mobile Profile 1.0	Fairly stable, but should integrate more tests as the specification progresses (again) toward Candidate Recomendation, and converges with the OMA WCSS.	Unknown, estimated 80%. Both HTML (220) and XHTML (230) tests available.	The test suite link above points to a result summary template, which in turn points to various CSS tests. The tests don't point to specific parts of the specification.	Not found	Your browser would work as harness, clicking links and navigating back again. No automation process, the test suite is tested manually.	Publicly available	Not defined	Not defined		Some of the tests rely on font effects that are not available on many phones. Other identified issues: till, dom
WCSS 1.1	WCSS Test Suite	Part of OMA approved Browsing 2.2 specification	There are 358 tests. The test suite seems to be (close to) complete. No major missing items found.	?	MS Word documentation	The test suite consists of a listing of categories and subcategories. By following those, one can reach each test, implying a lot of next/back navigation. This can be solved more neatly by <link rel="next">. At the end of each test, there is a link to post results to a potential tracker script that doesn't exist at this point.	The test suite is available for OMA members and for nonmembers upon a license agreement.	OMA IOP process (anyone can raise a problem report)	OMA Test Code License Agreement allows for derivative work	The tests appear to hold high quality. Each test has a clear pass condition. The description, however, should be made much more concise.	- Some tests rely on font and color effects for those UAs which support them. - Some tests rely on font (e.g. italic font) and color effects where this is not necessarily needed. - Almost all tests are suiteable for mobile screens. Only a few table tests which should be re-engineered for testing on small screens were found
DOM Level 1	DOM Conformance Test Suites	Stable	Coverage looks quite good. There seems to be 543 tests for the DOM Core level 1.	Found no assertions, but each test has a pointer to the relevant specification's section.	Fairly complete	This test suite uses the browser as the harness (EcmaScript). Looks easy enough to run.	The test suite is publicly available	There is a mailing list and a number of documents explaining how to participate on the process.	W3C License (?)	Test Suite is well organized. There is a Java version and an ECMA Script version for each of the DOM levels. Test Suite even as stable as it may be is a number of years old. Relevance to Mobile not apparent at this time.	Nearly impossible to use on a mobile device (dom) → mobile-friendly version
WICD Mobile Profile	Testcases for WICD Mobile 1.0	Spec in CR as of September 2007	~128 tests. Specifications coverage not documented, however it looks adequate.	Each test maps to a test assertion in the specifications.	Internal documentation is quite good and pass/fail criteria is well explained.	Fairly easy navigation from one test to another.	The test suite is publicly available on a W3C server. No indication was found on how to access the test cases files for downloading.	Unknown	W3C		Not tested on a mobile device, but it looks as it should be easy enough

Dominique Hazaël-Massieux

The running of this working group is funded by the European Union through the MobiWeb 2.0 Project, and its set up was funded throug the 3GWeb European Project.