TokenList Class Reference

#include <tokenlist.h>

Public Member Functions
	TokenList (const Settings *settings)
	~TokenList ()
	TokenList (const TokenList &)=delete
TokenList &	operator= (const TokenList &)=delete
const std::string &	getSourceFilePath () const
bool	isC () const
bool	isCPP () const
void	setLang (Standards::Language lang)
void	addtoken (const std::string &str, const nonneg int lineno, const nonneg int column, const nonneg int fileno, bool split=false)
void	addtoken (const std::string &str, const Token *locationTok)
void	addtoken (const Token *tok, const nonneg int lineno, const nonneg int column, const nonneg int fileno)
void	addtoken (const Token *tok, const Token *locationTok)
void	addtoken (const Token *tok)
bool	createTokens (std::istream &code, const std::string &file0)
	Create tokens from code. More...
bool	createTokens (std::istream &code, Standards::Language lang)
void	createTokens (simplecpp::TokenList &&tokenList)
void	deallocateTokens ()
	Deallocate list. More...
int	appendFileIfNew (std::string fileName)
	append file name if seen the first time; return its index in any case More...
const Token *	front () const
	get first token of list More...
Token *	front ()
const Token *	back () const
	get last token of list More...
Token *	back ()
const std::vector< std::string > &	getFiles () const
	Get filenames (the sourcefile + the files it include). More...
std::string	getOrigFile (const Token *tok) const
const std::string &	file (const Token *tok) const
	get filename for given token More...
std::string	fileLine (const Token *tok) const
	Get file:line for a given token. More...
std::size_t	calculateHash () const
	Calculates a hash of the token list used to compare multiple token lists with each other as quickly as possible. More...
void	createAst () const
	Create abstract syntax tree. More...
void	validateAst (bool print) const
	Check abstract syntax tree. More...
bool	validateToken (const Token *tok) const
	Verify that the given token is an element of the tokenlist. More...
void	simplifyPlatformTypes ()
	Convert platform dependent types to standard types. More...
void	simplifyStdType ()
	Collapse compound standard types into a single token. More...
void	clangSetOrigFiles ()
bool	isKeyword (const std::string &str) const

Static Public Member Functions
static void	deleteTokens (Token *tok)
	Delete all tokens in given token list. More...
static void	insertTokens (Token *dest, const Token *src, nonneg int n)
static Token *	copyTokens (Token *dest, const Token *first, const Token *last, bool one_line=true)
	Copy tokens. More...

Private Member Functions
void	determineCppC ()
bool	createTokensInternal (std::istream &code, const std::string &file0)

Private Attributes
TokensFrontBack	mTokensFrontBack
	Token list. More...
std::vector< std::string >	mFiles
	filenames for the tokenized source code (source + included) More...
std::vector< std::string >	mOrigFiles
	Original filenames for the tokenized source code (source + included) More...
const Settings *const	mSettings {}
	settings More...
Standards::Language	mLang {Standards::Language::None}
	File is known to be C/C++ code. More...

Detailed Description

Definition at line 53 of file tokenlist.h.

Constructor & Destructor Documentation

TokenList() [1/2]

TokenList::TokenList ( const Settings *  settings )

explicit

Definition at line 59 of file tokenlist.cpp.

References Settings::enforcedLang, mLang, and mSettings.

~TokenList()

TokenList::~TokenList

(

)

Definition at line 68 of file tokenlist.cpp.

References deallocateTokens().

TokenList() [2/2]

TokenList::TokenList ( const TokenList &  )

delete

Member Function Documentation

addtoken() [1/5]

void TokenList::addtoken	(	const std::string &	str,
		const nonneg int	lineno,
		const nonneg int	column,
		const nonneg int	fileno,
		bool	split = false
	)

Definition at line 145 of file tokenlist.cpp.

References TokensFrontBack::back, Token::column(), Token::fileIndex(), TokensFrontBack::front, Token::insertToken(), Token::linenr(), mTokensFrontBack, split(), and Token::str().

Referenced by TemplateSimplifier::addNamespace(), clangimport::AstNode::addtoken(), TemplateSimplifier::expandTemplate(), parsedecl(), and SymbolDatabase::setArrayDimensionsUsingValueFlow().

addtoken() [2/5]

void TokenList::addtoken	(	const std::string &	str,
		const Token *	locationTok
	)

Definition at line 178 of file tokenlist.cpp.

References TokensFrontBack::back, Token::column(), Token::fileIndex(), TokensFrontBack::front, Token::insertToken(), Token::linenr(), mTokensFrontBack, and Token::str().

addtoken() [3/5]

void TokenList::addtoken

(

const Token *

tok

)

Definition at line 238 of file tokenlist.cpp.

References TokensFrontBack::back, Token::column(), Token::fileIndex(), Token::flags(), TokensFrontBack::front, Token::getMacroName(), Token::insertToken(), Token::linenr(), mTokensFrontBack, Token::originalName(), Token::setMacroName(), and Token::str().

addtoken() [4/5]

void TokenList::addtoken	(	const Token *	tok,
		const nonneg int	lineno,
		const nonneg int	column,
		const nonneg int	fileno
	)

Definition at line 196 of file tokenlist.cpp.

References TokensFrontBack::back, Token::column(), Token::fileIndex(), Token::flags(), TokensFrontBack::front, Token::insertToken(), Token::linenr(), mTokensFrontBack, Token::originalName(), and Token::str().

addtoken() [5/5]

void TokenList::addtoken	(	const Token *	tok,
		const Token *	locationTok
	)

Definition at line 217 of file tokenlist.cpp.

References TokensFrontBack::back, Token::column(), Token::fileIndex(), Token::flags(), TokensFrontBack::front, Token::insertToken(), Token::linenr(), mTokensFrontBack, Token::originalName(), and Token::str().

appendFileIfNew()

int TokenList::appendFileIfNew

(

std::string

fileName

)

append file name if seen the first time; return its index in any case

Definition at line 110 of file tokenlist.cpp.

References determineCppC(), mFiles, and Path::sameFileName().

Referenced by CppCheck::checkClang(), createTokens(), and clangimport::AstNode::setLocations().

back() [1/2]

Token* TokenList::back ( )

inline

Definition at line 132 of file tokenlist.h.

back() [2/2]

const Token* TokenList::back ( ) const

inline

get last token of list

Definition at line 128 of file tokenlist.h.

Referenced by clangimport::AstNode::addFullScopeNameTokens(), TemplateSimplifier::addNamespace(), clangimport::AstNode::addtoken(), clangimport::AstNode::addTypeTokens(), clangimport::AstNode::createScope(), SymbolDatabase::createSymbolDatabaseEnums(), SymbolDatabase::createSymbolDatabaseExprIds(), SymbolDatabase::createSymbolDatabaseIncompleteVars(), SymbolDatabase::createSymbolDatabaseSetFunctionPointers(), SymbolDatabase::createSymbolDatabaseSetScopePointers(), SymbolDatabase::createSymbolDatabaseSetTypePointers(), SymbolDatabase::createSymbolDatabaseSetVariablePointers(), clangimport::AstNode::createTokens(), clangimport::AstNode::createTokens1(), clangimport::AstNode::createTokensForCXXRecord(), clangimport::AstNode::createTokensFunctionDecl(), clangimport::AstNode::createTokensVarDecl(), SymbolDatabase::debugSymbolDatabase(), TemplateSimplifier::expandTemplate(), clangimport::AstNode::getNestedInScope(), isNotEqual(), Tokenizer::validate(), and valueFlowFunctionReturn().

calculateHash()

std::size_t TokenList::calculateHash

(

)

const

Calculates a hash of the token list used to compare multiple token lists with each other as quickly as possible.

Definition at line 424 of file tokenlist.cpp.

References front(), and Token::next().

Referenced by CppCheck::checkFile().

clangSetOrigFiles()

void TokenList::clangSetOrigFiles

(

)

Definition at line 127 of file tokenlist.cpp.

References mFiles, and mOrigFiles.

Referenced by clangimport::parseClangAstDump().

copyTokens()

Token * TokenList::copyTokens ( Token *  dest, const Token *  first, const Token *  last, bool  one_line = true  )

static

Copy tokens.

Parameters

dest	destination token where copied tokens will be inserted after
first	first token to copy
last	last token to copy
one_line	true=>copy all tokens to the same line as dest. false=>copy all tokens to dest while keeping the 'line breaks'

Returns

new location of last token copied

Definition at line 264 of file tokenlist.cpp.

References Token::fileIndex(), Token::flags(), Token::insertToken(), Token::linenr(), Token::link(), Token::Match(), Token::next(), Token::setTokenDebug(), Token::tokType(), and Token::varId().

Referenced by TemplateSimplifier::fixForwardDeclaredDefaultArgumentValues(), TemplateSimplifier::simplifyTemplateAliases(), Tokenizer::simplifyTypedefCpp(), and Tokenizer::simplifyUsing().

createAst()

void TokenList::createAst

(

)

const

Create abstract syntax tree.

Referenced by Tokenizer::simplifyTokens1().

createTokens() [1/3]

void TokenList::createTokens

(

simplecpp::TokenList &&

tokenList

)

Definition at line 374 of file tokenlist.cpp.

References Token::assignProgressValues(), TokensFrontBack::back, Settings::basePaths, Token::column(), determineCppC(), Token::fileIndex(), TokensFrontBack::front, Path::getRelativePath(), Token::insertToken(), Token::linenr(), mFiles, mOrigFiles, mSettings, mTokensFrontBack, Settings::relativePaths, Token::setMacroName(), and Token::str().

createTokens() [2/3]

bool TokenList::createTokens	(	std::istream &	code,
		const std::string &	file0
	)

Create tokens from code.

The code must be preprocessed first:

• multiline strings are not handled.
• UTF in the code are not handled.
• comments are not handled.

  Parameters

  code	input stream for code
  file0	source file name

Definition at line 336 of file tokenlist.cpp.

References appendFileIfNew(), ASSERT_LANG, and createTokensInternal().

Referenced by CppCheck::checkFile(), createTokensInternal(), getMinMaxValues(), gettokenlistfromvalid(), isNotEqual(), and SymbolDatabase::setValueTypeInTokenList().

createTokens() [3/3]

bool TokenList::createTokens	(	std::istream &	code,
		Standards::Language	lang
	)

Definition at line 347 of file tokenlist.cpp.

References ASSERT_LANG, createTokensInternal(), and mLang.

createTokensInternal()

bool TokenList::createTokensInternal ( std::istream &  code, const std::string &  file0  )

private

Definition at line 361 of file tokenlist.cpp.

References createTokens(), and mFiles.

Referenced by createTokens().

deallocateTokens()

void TokenList::deallocateTokens

(

)

Deallocate list.

Definition at line 86 of file tokenlist.cpp.

References TokensFrontBack::back, deleteTokens(), TokensFrontBack::front, mFiles, and mTokensFrontBack.

Referenced by ~TokenList().

deleteTokens()

void TokenList::deleteTokens ( Token *  tok )

static

Delete all tokens in given token list.

Parameters

tok	token list to delete

Definition at line 132 of file tokenlist.cpp.

References Token::next().

Referenced by deallocateTokens().

determineCppC()

void TokenList::determineCppC ( )

private

Definition at line 94 of file tokenlist.cpp.

References ASSERT_LANG, getSourceFilePath(), Path::identify(), and mLang.

Referenced by appendFileIfNew(), and createTokens().

file()

const std::string& TokenList::file

(

const Token *

tok

)

const

get filename for given token

Parameters

tok	The given token

Returns

filename for the given token

Referenced by Tokenizer::dump(), CTU::getFileInfo(), CheckClass::getFileInfo(), CTU::getFunctionId(), SuppressionList::markUnmatchedInlineSuppressionsAsChecked(), CheckUnusedFunctions::parseTokens(), Tokenizer::simplifyTypedef(), and Tokenizer::simplifyTypedefCpp().

fileLine()

std::string TokenList::fileLine

(

const Token *

tok

)

const

Get file:line for a given token.

Parameters

tok	given token

Returns

location for given token

Referenced by SymbolDatabase::printOut(), TemplateSimplifier::printOut(), SymbolDatabase::printVariable(), scopeToString(), and tokenToString().

front() [1/2]

Token* TokenList::front ( )

inline

Definition at line 123 of file tokenlist.h.

front() [2/2]

const Token* TokenList::front ( ) const

inline

get first token of list

Definition at line 119 of file tokenlist.h.

Referenced by Tokenizer::addSemicolonAfterUnknownMacro(), Tokenizer::arraySize(), CheckAssert::assertWithSideEffects(), calculateHash(), Tokenizer::calculateScopes(), TemplateSimplifier::checkComplicatedSyntaxErrorsInTemplates(), Tokenizer::checkForEnumsWithTypedef(), Tokenizer::combineOperators(), Tokenizer::combineStringAndCharLiterals(), Tokenizer::concatenateNegativeNumberAndAnyPositive(), Tokenizer::createLinks(), Tokenizer::createLinks2(), SymbolDatabase::createSymbolDatabaseEnums(), SymbolDatabase::createSymbolDatabaseExprIds(), SymbolDatabase::createSymbolDatabaseIncompleteVars(), SymbolDatabase::createSymbolDatabaseSetFunctionPointers(), SymbolDatabase::createSymbolDatabaseSetScopePointers(), SymbolDatabase::createSymbolDatabaseSetTypePointers(), SymbolDatabase::createSymbolDatabaseSetVariablePointers(), clangimport::AstNode::createTokensFunctionDecl(), clangimport::AstNode::createTokensVarDecl(), SymbolDatabase::debugSymbolDatabase(), Tokenizer::deleteInvalidTypedef(), Tokenizer::dump(), Tokenizer::elseif(), TemplateSimplifier::expandTemplate(), getMinMaxValues(), TemplateSimplifier::getTemplateDeclarations(), TemplateSimplifier::getTemplateInstantiations(), gettokenlistfromvalid(), Library::isFloatArgValid(), Library::isIntArgValid(), isNotEqual(), Tokenizer::markCppCasts(), clangimport::parseClangAstDump(), parsedecl(), Tokenizer::printDebugOutput(), Tokenizer::removeExtraTemplateKeywords(), Tokenizer::removeMacroInClassDef(), Tokenizer::removeMacrosInGlobalScope(), Tokenizer::removePragma(), Tokenizer::removeRedundantSemicolons(), TemplateSimplifier::replaceTemplateUsage(), SymbolDatabase::setArrayDimensionsUsingValueFlow(), setTypes(), ValueFlow::setValues(), clangimport::AstNode::setValueType(), SymbolDatabase::setValueTypeInTokenList(), Tokenizer::setVarId(), Tokenizer::setVarIdPass1(), Tokenizer::setVarIdPass2(), Tokenizer::simplifyAddBraces(), Tokenizer::simplifyArrayAccessSyntax(), TemplateSimplifier::simplifyCalculations(), Tokenizer::simplifyCAlternativeTokens(), Tokenizer::simplifyCaseRange(), Tokenizer::simplifyDoublePlusAndDoubleMinus(), Tokenizer::simplifyEmptyNamespaces(), Tokenizer::simplifyExternC(), Tokenizer::simplifyFunctionParameters(), Tokenizer::simplifyFunctionPointers(), Tokenizer::simplifyHeadersAndUnusedTemplates(), Tokenizer::simplifyIfSwitchForInit(), Tokenizer::simplifyInitVar(), Tokenizer::simplifyLabelsCaseDefault(), Tokenizer::simplifyParameterVoid(), Tokenizer::simplifyPointerToStandardType(), Tokenizer::simplifyRedundantConsecutiveBraces(), Tokenizer::simplifyRedundantParentheses(), Tokenizer::simplifyRoundCurlyParentheses(), Tokenizer::simplifySQL(), Tokenizer::simplifyStaticConst(), TemplateSimplifier::simplifyTemplateAliases(), TemplateSimplifier::simplifyTemplateArgs(), TemplateSimplifier::simplifyTemplates(), Tokenizer::simplifyTokenList1(), Tokenizer::simplifyTypedef(), Tokenizer::simplifyTypedefCpp(), Tokenizer::simplifyTypedefLHS(), Tokenizer::simplifyTypeIntrinsics(), Tokenizer::simplifyUsing(), Tokenizer::simplifyUsingToTypedef(), Tokenizer::simplifyVarDecl(), Tokenizer::simplifyVariableMultipleAssign(), Tokenizer::sizeofAddParentheses(), Tokenizer::splitTemplateRightAngleBrackets(), Tokenizer::tokens(), valueFlowArray(), valueFlowArrayBool(), valueFlowArrayElement(), valueFlowBitAnd(), valueFlowDebug(), valueFlowGlobalConstVar(), valueFlowGlobalStaticVar(), valueFlowImpossibleValues(), valueFlowInferCondition(), valueFlowIteratorInfer(), valueFlowIterators(), valueFlowLifetime(), valueFlowNumber(), valueFlowPointerAlias(), valueFlowRightShift(), valueFlowSameExpressions(), valueFlowSmartPointer(), valueFlowString(), valueFlowUninit(), valueFlowUnknownFunctionReturn(), and SymbolDatabase::~SymbolDatabase().

getFiles()

const std::vector<std::string>& TokenList::getFiles ( ) const

inline

Get filenames (the sourcefile + the files it include).

The first filename is the filename for the sourcefile

Returns

vector with filenames

Definition at line 141 of file tokenlist.h.

Referenced by CppCheck::checkNormalTokens(), ErrorMessage::ErrorMessage(), getSourceFilePath(), CheckUnusedFunctions::parseTokens(), Tokenizer::printDebugOutput(), Token::printValueFlow(), TemplateSimplifier::simplifyTemplateInstantiations(), TemplateSimplifier::simplifyTemplates(), Tokenizer::simplifyTokens1(), Tokenizer::simplifyTypedefCpp(), and Tokenizer::simplifyUsing().

getOrigFile()

std::string TokenList::getOrigFile

(

const Token *

tok

)

const

getSourceFilePath()

const std::string & TokenList::getSourceFilePath

(

)

const

Returns

the source file path. e.g. "file.cpp"

Definition at line 75 of file tokenlist.cpp.

References emptyString, and getFiles().

Referenced by bailoutInternal(), Summaries::create(), SymbolDatabase::createSymbolDatabaseFindAllScopes(), clangimport::AstNode::createTokens1(), determineCppC(), ErrorMessage::fromInternalError(), and valueFlowConditionExpressions().

insertTokens()

void TokenList::insertTokens ( Token *  dest, const Token *  src, nonneg int  n  )

static

Definition at line 304 of file tokenlist.cpp.

References Token::column(), Token::createMutualLinks(), Token::fileIndex(), Token::flags(), Token::getMacroName(), Token::insertToken(), Token::linenr(), Token::Match(), Token::next(), Token::originalName(), Token::setMacroName(), Token::str(), Token::tokType(), and Token::varId().

Referenced by Tokenizer::simplifyInitVar(), and Tokenizer::simplifyVarDecl().

isC()

bool TokenList::isC

(

)

const

Returns

true if the code is C

Referenced by Token::isC().

isCPP()

bool TokenList::isCPP

(

)

const

Returns

true if the code is C++

Referenced by Token::isCpp(), ValueFlowAnalyzer::isCPP(), valueFlowAfterMove(), valueFlowForwardAssign(), valueFlowFunctionDefaultParameter(), valueFlowImpossibleValues(), and valueFlowNumber().

isKeyword()

bool TokenList::isKeyword

(

const std::string &

str

)

const

Referenced by Token::update_property_info().

operator=()

TokenList& TokenList::operator= ( const TokenList &  )

delete

setLang()

void TokenList::setLang

(

Standards::Language

lang

)

Referenced by CppCheck::checkFile(), clangimport::AstNode::createTokens1(), and clangimport::AstNode::setValueType().

simplifyPlatformTypes()

void TokenList::simplifyPlatformTypes

(

)

Convert platform dependent types to standard types.

32 bits: size_t -> unsigned long 64 bits: size_t -> unsigned long long

Referenced by getMinMaxValues(), SymbolDatabase::setValueTypeInTokenList(), and Tokenizer::simplifyTokenList1().

simplifyStdType()

void TokenList::simplifyStdType

(

)

Collapse compound standard types into a single token.

unsigned long long int => long _isUnsigned=true,_isLong=true

Referenced by getMinMaxValues(), SymbolDatabase::setValueTypeInTokenList(), and Tokenizer::simplifyTokenList1().

validateAst()

void TokenList::validateAst

(

bool

print

)

const

Check abstract syntax tree.

Throws InternalError on failure

Referenced by Tokenizer::simplifyTokens1().

validateToken()

bool TokenList::validateToken

(

const Token *

tok

)

const

Verify that the given token is an element of the tokenlist.

That method is implemented for debugging purposes.

Parameters

[in]

tok

token to be checked

Returns

true if token was found in tokenlist, false else. In case of nullptr true is returned.

Referenced by TemplateSimplifier::simplifyTemplateInstantiations().

Member Data Documentation

mFiles

std::vector<std::string> TokenList::mFiles

private

filenames for the tokenized source code (source + included)

Definition at line 212 of file tokenlist.h.

Referenced by appendFileIfNew(), clangSetOrigFiles(), createTokens(), createTokensInternal(), and deallocateTokens().

mLang

Standards::Language TokenList::mLang {Standards::Language::None}

private

File is known to be C/C++ code.

Definition at line 221 of file tokenlist.h.

Referenced by createTokens(), determineCppC(), and TokenList().

mOrigFiles

std::vector<std::string> TokenList::mOrigFiles

private

Original filenames for the tokenized source code (source + included)

Definition at line 215 of file tokenlist.h.

Referenced by clangSetOrigFiles(), and createTokens().

mSettings

const Settings* const TokenList::mSettings {}

private

settings

Definition at line 218 of file tokenlist.h.

Referenced by createTokens(), and TokenList().

mTokensFrontBack

TokensFrontBack TokenList::mTokensFrontBack

private

Token list.

Definition at line 209 of file tokenlist.h.

Referenced by addtoken(), createTokens(), and deallocateTokens().

---

The documentation for this class was generated from the following files:

• tokenlist.h
• tokenlist.cpp