This file implements the scanner for the application. The scanner is responsible for reading data from a file and creating tokens out of the data that are easily recognizable by a parser. The parsers that are used in the application are all very simple. Hence the token is also very simple; it's represented by a single struct. The scanner is stateful and may only be used with one parser at a time. In other words only one file can be scanned at a time. There are only two parsers that utilize this scanner: the elevReader that reads .elev files and the trlReader that reads .trl files. Since the scanner is stateful, parsers that use it must make sure to call initTokenizer when starting to scan a new file and to call finalizeTokenizer when scanning of the file is complete. The latter is more of a concern as it frees up the memory that the scanner is consuming. It turns out that the scanner consumes quite a large amount of memory as it first reads the entire file into memory and then picks off tokens as the application progresses.