String bracket- and quotation-aware string tokenizer. More...

#include <tokenizer.h>

Public Member Functions
	Tokenizer ()
	Constructor. Creates tokenizer with given character as separator. More...

void	tokenizeLine (const std::string &line)
	Tokenizes given record (string). More...

int	giveNumberOfTokens ()
	returns the number of tokens. More...

const char *	giveToken (int i)
	Returns pointer to i-th token. More...

Protected Member Functions
std::string	readSimpleToken (std::size_t &pos, const std::string &line)
	Reads next simple token (stops when whitespace character is reached) More...

std::string	readToken (std::size_t &pos, const std::string &line, char sep)
	Reads next token (stops when separator is reached) More...

std::string	readStructToken (std::size_t &pos, const std::string &line)
	Reads next structured token (bounded by '{' '}' pairs, possibly nested). More...

std::string	readStringToken (std::size_t &pos, const std::string &line)
	Reads next string token (quoted). More...

std::string	readSimpleExpressionToken (std::size_t &pos, const std::string &line)
	Reads next simple expression token (section identified by starting with '$' and finishing with '$'). More...

Private Attributes
std::vector< std::string >	tokens
	Array of tokens. More...

Detailed Description

String bracket- and quotation-aware string tokenizer.

This class splits given record (represented as string) to particular tokens, which are separated by white spaces. Tokenizer recognizes "quoted strings" and structured tokens that are bounded by '{}, $$' pairs, can be nested and represent single token.

Definition at line 50 of file tokenizer.h.

Constructor & Destructor Documentation

oofem::Tokenizer::Tokenizer ( )

Constructor. Creates tokenizer with given character as separator.

Definition at line 43 of file tokenizer.C.

Member Function Documentation

int oofem::Tokenizer::giveNumberOfTokens ( )

returns the number of tokens.

Definition at line 140 of file tokenizer.C.

References tokens.

Referenced by oofem::OOFEMTXTInputRecord::finish(), oofem::OOFEMTXTInputRecord::giveKeywordIndx(), oofem::OOFEMTXTInputRecord::giveRecordKeywordField(), oofem::OOFEMTXTInputRecord::OOFEMTXTInputRecord(), oofem::OOFEMTXTInputRecord::operator=(), and oofem::OOFEMTXTInputRecord::setRecordString().

const char * oofem::Tokenizer::giveToken ( int i )

Returns pointer to i-th token.

Definition at line 146 of file tokenizer.C.

References tokens.

Referenced by oofem::OOFEMTXTInputRecord::finish(), oofem::OOFEMTXTInputRecord::giveField(), oofem::OOFEMTXTInputRecord::giveKeywordIndx(), and oofem::OOFEMTXTInputRecord::giveRecordKeywordField().

std::string oofem::Tokenizer::readSimpleExpressionToken	(	std::size_t &	pos,
		const std::string &	line
	)

protected

Reads next simple expression token (section identified by starting with '$' and finishing with '$').

Parameters

pos	Position (index) in token buffer.
line	Record from which token is parsed.

Definition at line 75 of file tokenizer.C.

References OOFEM_WARNING, and readToken().

Referenced by tokenizeLine().

std::string oofem::Tokenizer::readSimpleToken	(	std::size_t &	pos,
		const std::string &	line
	)

protected

Reads next simple token (stops when whitespace character is reached)

Parameters

pos	Starting position.
line	Record from which token is parsed.

Definition at line 89 of file tokenizer.C.

Referenced by tokenizeLine().

std::string oofem::Tokenizer::readStringToken	(	std::size_t &	pos,
		const std::string &	line
	)

protected

Reads next string token (quoted).

Parameters

pos	Position (index) in token buffer.
line	Record from which token is parsed.

Definition at line 49 of file tokenizer.C.

References OOFEM_WARNING, and readToken().

Referenced by tokenizeLine().

std::string oofem::Tokenizer::readStructToken	(	std::size_t &	pos,
		const std::string &	line
	)

protected

Reads next structured token (bounded by '{' '}' pairs, possibly nested).

Parameters

pos	Starting position (should point to a '{').
line	Record from which token is parsed.

Definition at line 63 of file tokenizer.C.

References OOFEM_WARNING, and readToken().

Referenced by tokenizeLine().

std::string oofem::Tokenizer::readToken	(	std::size_t &	pos,
		const std::string &	line,
		char	sep
	)

protected

Reads next token (stops when separator is reached)

Parameters

pos	Starting position.
line	Record from which token is parsed.
sep	Separator.

Definition at line 100 of file tokenizer.C.

Referenced by readSimpleExpressionToken(), readStringToken(), and readStructToken().

void oofem::Tokenizer::tokenizeLine ( const std::string & line )

Tokenizes given record (string).

Definition at line 110 of file tokenizer.C.

References readSimpleExpressionToken(), readSimpleToken(), readStringToken(), readStructToken(), and tokens.

Referenced by oofem::OOFEMTXTInputRecord::OOFEMTXTInputRecord(), oofem::OOFEMTXTInputRecord::operator=(), and oofem::OOFEMTXTInputRecord::setRecordString().

Member Data Documentation

std :: vector< std :: string > oofem::Tokenizer::tokens

private

Array of tokens.

Definition at line 54 of file tokenizer.h.

Referenced by giveNumberOfTokens(), giveToken(), and tokenizeLine().

The documentation for this class was generated from the following files:

Public Member Functions

Protected Member Functions

Private Attributes

Detailed Description

Constructor & Destructor Documentation

Member Function Documentation

Member Data Documentation