本文整理汇总了Python中scanner.Scanner.tokenize方法的典型用法代码示例。如果您正苦于以下问题：Python Scanner.tokenize方法的具体用法？Python Scanner.tokenize怎么用？Python Scanner.tokenize使用的例子？那么恭喜您, 这里精选的方法代码示例或许可以为您提供帮助。您也可以进一步了解该方法所在类scanner.Scanner的用法示例。
在下文中一共展示了Scanner.tokenize方法的1个代码示例，这些例子默认根据受欢迎程度排序。您可以为喜欢或者感觉有用的代码点赞，您的评价将有助于系统推荐出更棒的Python代码示例。
示例1: run_parser
# 需要导入模块: from scanner import Scanner [as 别名]
# 或者: from scanner.Scanner import tokenize [as 别名]
def run_parser(self, file_, lib_classes = None):
"""parses a jml file and returns a list of abstract syntax trees, or
parses a string containing JaML code.
"""
scanner = Scanner()
# Try and parse a file, if this fails, parse as a string
try:
# This is to see if it's a file or not
open(file_)
# Stores the directory of the main class
dir_ = os.path.dirname(file_)
# Add the name of the main class to begin with
file_name = os.path.basename(file_)
class_name = file_name.replace('.jml', '')
# this stores names of all already seen class references
seen = [class_name]
# Stores classes to parse
to_parse = [class_name]
# Stores the ASTs of parsed classes
parsed = nodes.ProgramASTs()
while to_parse:
# Append file information to the class names
file_name = os.path.join(dir_, to_parse[0] + '.jml')
# Get the raw input
raw = open(file_name)
input_ = raw.read()
# Scan and parse the file
tokens = scanner.tokenize(input_)
ast = self.parse(tokens)
# Check the class and file are named the same
if to_parse[0] != ast.children[0].value:
msg = 'Class name and file name do not match!'
raise NameError(msg)
to_parse.pop(0)
parsed.append(ast)
# Fined classes reference from the one just parsed
if lib_classes == None:
lib_classes = {}
refed_classes = self._find_refed_classes
refed_classes = refed_classes(ast, seen, [], lib_classes, dir_)
seen += refed_classes
to_parse += refed_classes
return parsed
except IOError:
# Simply parse the program held in the string
tokens = scanner.tokenize(file_)
ast = self.parse(tokens)
ast_wrapper = nodes.ProgramASTs()
ast_wrapper.append(ast)
return ast_wrapper