本文整理汇总了Python中scanner.Scanner.eos方法的典型用法代码示例。如果您正苦于以下问题：Python Scanner.eos方法的具体用法？Python Scanner.eos怎么用？Python Scanner.eos使用的例子？那么恭喜您, 这里精选的方法代码示例或许可以为您提供帮助。您也可以进一步了解该方法所在类scanner.Scanner的用法示例。
在下文中一共展示了Scanner.eos方法的1个代码示例，这些例子默认根据受欢迎程度排序。您可以为喜欢或者感觉有用的代码点赞，您的评价将有助于系统推荐出更棒的Python代码示例。
示例1: consume_text_worker
# 需要导入模块: from scanner import Scanner [as 别名]
# 或者: from scanner.Scanner import eos [as 别名]
def consume_text_worker( params ):
then = datetime.datetime.now()
offset = 0
if OFFSET_KEY in params:
offset = int( params[ OFFSET_KEY ] )
text = datastore.read_current_blob( offset )
scanner = Scanner( text )
current_word = None
num_tasks = 0
pattern = re.compile( r"[\w']+|[.,!?;]" )
timeout = False
elapsed = None
while not timeout and not scanner.eos():
token = scanner.scan( pattern )
if current_word is not None:
add_forward_link( current_word, token )
num_tasks = num_tasks + 1
current_word = token
scanner.get()
elapsed = datetime.datetime.now() - then
if elapsed >= TASK_TIME:
offset = scanner.pos
add_task( "/tasks/consume_text", { OFFSET_KEY : offset } )
logging.debug( "consume_text_worker hit timelimit, will resume at offset %d" % offset )
timeout = True
logging.debug( "scanned %d tokens in %d seconds" % (num_tasks,elapsed.total_seconds() ) )
if scanner.eos():
finalise_links()