CSE 423 Lab #8: k0 Test Coverage

Due: Sunday 3/30 11:59pm Turnin: your test case contributions on Canvas in a .zip file.

Your grade will be based on whether the instructor thinks your submitted work represents at least two hours of effort.

The purpose of this week's lab is to produce a semantics-oriented test suite (henceforth named tests/ and includable in your future HW submissions) for the k0 language. The test suite is a directory hierarchy containing source .kt files that you can use as test input, and any data files that they run on. Although I say semantics-oriented, you can't get to semantics testing unless you also have lexical and syntax functionality. So by this lab, you have three kinds of tests (lexical, syntax, and semantic); each test should have a prefix of either lex, syn, or sem. You also have three categories of expected diagnostics, placed in subdirectories of tests/:

for things that are legal, put them in tests/k0
for things that are Kotlin but not in k0, put them in tests/kotlin
for things that are illegal, put them in tests/errors

Example 423 project directory structure after this lab:

   your_423_project/
      ... your makefile and all your compiler source files, plus subdirectory:
      tests/
         ...your testrunner.sh (or .kt etc.), plus three subdirectories:
         errors/
	    {lex,syn,sem}*.kt; test-cases that should be errors
	 k0/
	    {lex,syn,sem}*.kt; test-cases that should pass
	 kotlin/
	    {lex,syn,sem}*.kt; test-cases for Kotlin-not-k0 errors

For this lab you are assembling a suite of test cases, and your compiler is expected to print either "No errors" or print a lexical, syntax, or semantic error message (with filename, and line number, and offending character or token).

1. Download one or more starting points

Please start by (re)organizing any tests you've already written
Before you go writing 1,000 new test cases, or programs that generate test cases, I recommend that you look for what you can find available for free on the internet.
For Kotlin there should be a lot of tests out there. For k0, there will be none. But if you look on some Kotlin tutorial websites, many of their toy examples might be good k0 tests, possibly with slight tweaking.

2. Establish Lexical Coverage

Check whether your tests/k0 directory has tests that use every terminal symbol that is legal under k0. If not, construct new tests lex1.kt through lexN.kt until you have a test that uses each terminal symbol. Produce as many lexical error tests as you can think of. At the very least, for each regex that is bounded by beginning and ending characters, you should have tests where the ending characters are missing.

3. Establish Syntactic Coverage

Check whether your tests/k0 directory has tests that use every rule in your grammar that is legal under k0. If not construct new tests syn1.kt through synN.kt until you have a test that uses each grammar rule. Provide as many error tests as you can think of. For each production rule that includes terminal symbols, can you make a test case that would use that production rule, except it is missing an expected terminal symbol?

4. Establish Semantic Coverage

Develop tests for all possible data types on each operator. Develop tests that check for bad function calls, including wrong number of arguments, incorrect parameter types, or incorrect return type.

5. Write a testrunner script

Write a script (in any language that will run on login.cs.nmt.edu and/or your own test machine(s); bash or python are examples) that runs your compiler on all your tests. Run it and redirect your output to a file. Examine that output to look for problems.

It is usual to save/record the expected output for each test that you add, and to re-test your compiler on all tests whenever you make changes. You can diff these regression tests against expected output and tell when a test output has become broken by a change to your compiler.