本文整理汇总了Java中org.antlr.v4.runtime.CommonTokenStream.getTokens方法的典型用法代码示例。如果您正苦于以下问题：Java CommonTokenStream.getTokens方法的具体用法？Java CommonTokenStream.getTokens怎么用？Java CommonTokenStream.getTokens使用的例子？那么恭喜您, 这里精选的方法代码示例或许可以为您提供帮助。您也可以进一步了解该方法所在类org.antlr.v4.runtime.CommonTokenStream的用法示例。
在下文中一共展示了CommonTokenStream.getTokens方法的1个代码示例，这些例子默认根据受欢迎程度排序。您可以为喜欢或者感觉有用的代码点赞，您的评价将有助于系统推荐出更棒的Java代码示例。
示例1: getTokenList
import org.antlr.v4.runtime.CommonTokenStream; //导入方法依赖的package包/类
/**
* Extract the token list from the Cypher input. Uses ANTLR classes to perform this. Some tokens
* are excluded, such as EOF and semi colons.
*
* @param cyp Cypher input as text.
* @param DEBUG_PRINT Print out debug statements or not.
* @return A list of tokens as deciphered by the ANTLR classes, based on the openCypher grammar.
*/
public static ArrayList<String> getTokenList(String cyp, boolean DEBUG_PRINT) {
CypherLexer lexer = new CypherLexer(new ANTLRInputStream(cyp));
CommonTokenStream tokens = new CommonTokenStream(lexer);
tokens.fill();
CypherParser parser = new CypherParser(tokens);
// dangerous - comment out if something is going wrong.
parser.removeErrorListeners();
ParseTree tree = parser.cypher();
ParseTreeWalker walker = new ParseTreeWalker();
cypherWalker = null;
cypherWalker = new CypherWalker();
walker.walk(cypherWalker, tree);
if (DEBUG_PRINT) cypherWalker.printInformation();
ArrayList<String> tokenList = new ArrayList<>();
for (Object t : tokens.getTokens()) {
CommonToken tok = (CommonToken) t;
String s = tok.getText().toLowerCase();
// exclude some tokens from the list of tokens. This includes the EOF pointer,
// semi-colons, and alias artifacts.
if (!" ".equals(s) && !"<eof>".equals(s) && !";".equals(s) && !"as".equals(s) &&
!cypherWalker.getAlias().contains(s)) {
tokenList.add(s);
}
}
return tokenList;
}