Extract preprocessing from tokenizer

README.md

@@ -24,8 +24,7 @@ afterwards can be a bit cryptic as to where it originated.
 - `[operation][number type]`, e.g. `divi` for divide (div) integer
 - `%[register]` for addressing registers
 - `$[value]` for using immediate (literal) integer values
-- `'a'` for using immediate character values (currently only supports non
-escaped characters)
+- `'a'` for using immediate character values
 - `;` for end of statement (mandatory)
 - `[label]:` for labels
 - `#[text]` for comments: any text is ignored till a newline (`\n`) is found
@@ -92,13 +91,33 @@ There is currently no strict checking, so be careful.
 ### Preprocessor
 
 All preprocessor directives are prefixed by a `#`. Ill formed preprocessor
-directives do not halt compilation, they are merely reported and then ignored.
+directives do not halt compilation, they are merely ignored. All preprocessing
+is done in a single pass. Recursion or definition of a directive by another
+directive is not supported therefore.
 
-- `DEFINE` replaces any occurrence of the first argument by the second argument.
-	The second argument may be empty, effectively deleting occurences of argument
-	one. Quotes are currently not supported and arguments are separated by
-	whitespace. If multiple defines exist for the same substitution the first
-	declared is used.
+- `DEFINE <x> [y]` replaces any occurrence of the first argument (`x`) by the
+	second optional argument (`y`). The second argument can be empty, effectively
+	deleting all occurrences of `x`. Quotes are currently not supported and
+	arguments are separated by whitespace. If multiple defines exist the later
+	declarations will overwrite the previous.
+
+### Registers
+
+All registers are 32 bits wide. The following 4 registers currently exist:
+
+- A
+- B
+- C
+- D
+
+### Immediates
+
+An immediate integer value for 42 is for examle `$42`. Negative values are
+allowed, for example `$-42`. Notation must be in decimal, hexadecimal and octals
+are **not supported**.
+
+The immediate character value for the letter g is `'g'`. Character values must
+be a single character, escaped or multi byte characters are **not supported**.
 
 ### Operands