本文整理汇总了C#中Lexer.ParseToken方法的典型用法代码示例。如果您正苦于以下问题：C# Lexer.ParseToken方法的具体用法？C# Lexer.ParseToken怎么用？C# Lexer.ParseToken使用的例子？那么恭喜您, 这里精选的方法代码示例或许可以为您提供帮助。您也可以进一步了解该方法所在类Lexer的用法示例。
在下文中一共展示了Lexer.ParseToken方法的1个代码示例，这些例子默认根据受欢迎程度排序。您可以为喜欢或者感觉有用的代码点赞，您的评价将有助于系统推荐出更棒的C#代码示例。
示例1: ParseTokens
private static void ParseTokens(string sourceString, Lexer.Lexer lexer)
{
// we are treating comments as preprocessor commands so we strip them here
sourceString = lexer.StripComments(sourceString);
// we are now ready to parse the tokens from the text.
var lineCount = 1; // used to track the line number, 1 indexed
// Iterate through all lines of the code and extract tokens from them.
foreach (var line in sourceString.Split('\n').Select(line => line.Trim()))
{
var subLine = line;
while (subLine.Length != 0)
{
// this call pops off a token and returns the remaining string
// token that was removed is added to the lexer's token selection
subLine = lexer.ParseToken(subLine, lineCount);
}
// a line has been parsed, so remember we are working with the next line
lineCount ++;
}
// after tokens are extracted, print the list of tokens
foreach (var token in lexer.GetTokens())
{
Console.Write($"[{token.StringRepresentation}], ");
}
}