Prokee Module: bws

This module provides the second part of a two part scanning approach.

1. BasicBlockScanner
2. BasicWesternScanner (bws)

Tokens

The following types of tokens are recognized:

• Literals
• Operators
• Keywords
• Other words (separated by operators or change of alphabets)
• Level-2 blocks (as provided by BasicBlockScanner)

Alphabets

Example

[alphabet_hexdigit]
->first="0";
->last="9";
->extra="ABCDEFGH";
->extra="abcdefgh";

[alphabet_alphanumeric]
->first="0";
->last="9";
->first="A";
->last="Z";
->first="a";
->last="z";
->extra="_";

[alphabet_Japanese]
->include="Hiragana";//including an other alphabet
->include="Katakana";//including an other alphabet
->include="Kanji";   //including an other alphabet

Global Settings

Example

[settings]
->useAlphabets="true";
->useDictionary="false";
->dictionary="main_german";
->dictionary="main_english";
->cutat="Operator";
->cutat="Alphabet";
->cutat="Symbol";