本文整理汇总了Java中edu.stanford.nlp.process.WhitespaceTokenizer类的典型用法代码示例。如果您正苦于以下问题：Java WhitespaceTokenizer类的具体用法？Java WhitespaceTokenizer怎么用？Java WhitespaceTokenizer使用的例子？那么恭喜您, 这里精选的类代码示例或许可以为您提供帮助。
WhitespaceTokenizer类属于edu.stanford.nlp.process包，在下文中一共展示了WhitespaceTokenizer类的3个代码示例，这些例子默认根据受欢迎程度排序。您可以为喜欢或者感觉有用的代码点赞，您的评价将有助于系统推荐出更棒的Java代码示例。
示例1: WhitespaceTokenizerAnnotator
import edu.stanford.nlp.process.WhitespaceTokenizer; //导入依赖的package包/类
public WhitespaceTokenizerAnnotator(Properties props) {
super(false);
boolean eolIsSignificant =
Boolean.valueOf(props.getProperty(EOL_PROPERTY, "false"));
eolIsSignificant =
(eolIsSignificant ||
Boolean.valueOf(props.getProperty
(StanfordCoreNLP.NEWLINE_SPLITTER_PROPERTY, "false")));
factory = new WhitespaceTokenizer.WhitespaceTokenizerFactory<CoreLabel>
(new CoreLabelTokenFactory(), eolIsSignificant);
}
示例2: getTokenizerFactory
import edu.stanford.nlp.process.WhitespaceTokenizer; //导入依赖的package包/类
/**
* Return a tokenizer which might be suitable for tokenizing text that
* will be used with this Treebank/Language pair, without tokenizing carriage returns (i.e., treating them as white space). The implementation in AbstractTreebankLanguagePack
* returns a factory for {@link WhitespaceTokenizer}.
*
* @return A tokenizer
*/
public TokenizerFactory<? extends HasWord> getTokenizerFactory() {
return WhitespaceTokenizer.factory(false);
}
示例3: getTokenizerFactory
import edu.stanford.nlp.process.WhitespaceTokenizer; //导入依赖的package包/类
/**
* Return a tokenizer which might be suitable for tokenizing text that
* will be used with this Treebank/Language pair, without tokenizing carriage returns (i.e., treating them as white space). The implementation in AbstractTreebankLanguagePack
* returns a factory for {@link WhitespaceTokenizer}.
*
* @return A tokenizer
*/
@Override
public TokenizerFactory<? extends HasWord> getTokenizerFactory() {
return WhitespaceTokenizer.factory(false);
}