本文整理匯總了Java中opennlp.tools.tokenize.TokenizerME.tokenize方法的典型用法代碼示例。如果您正苦於以下問題：Java TokenizerME.tokenize方法的具體用法？Java TokenizerME.tokenize怎麽用？Java TokenizerME.tokenize使用的例子？那麽, 這裏精選的方法代碼示例或許可以為您提供幫助。您也可以進一步了解該方法所在類opennlp.tools.tokenize.TokenizerME的用法示例。
在下文中一共展示了TokenizerME.tokenize方法的3個代碼示例，這些例子默認根據受歡迎程度排序。您可以為喜歡或者感覺有用的代碼點讚，您的評價將有助於係統推薦出更棒的Java代碼示例。
示例1: segmentWords
import opennlp.tools.tokenize.TokenizerME; //導入方法依賴的package包/類
public List<String> segmentWords(String text) {
List<String> wordsList = new ArrayList<String>();
try {
InputStream modelIn = getClass().getResourceAsStream(wordBin);;
TokenizerModel model = new TokenizerModel(modelIn);
TokenizerME tokenizer = new TokenizerME(model);
String[] words = tokenizer.tokenize(text);
for(String word : words)
if (!punctuation.contains(word))
wordsList.add(word);
modelIn.close();
} catch (IOException e) {
e.printStackTrace();
}
return wordsList;
}
示例2: shouldNLP
import opennlp.tools.tokenize.TokenizerME; //導入方法依賴的package包/類
@Test
public void shouldNLP()
{
try
{
TokenizerME tokenizerME = new TokenizerME(new TokenizerModel(new ClassPathResource("opennlp-models/en/en-token.bin").getInputStream()));
String[] tokens001 = tokenizerME.tokenize("I love Vincent Van Gogh");
Assert.assertEquals("I" , tokens001[0]);
Assert.assertEquals("love" , tokens001[1]);
Assert.assertEquals("Vincent", tokens001[2]);
Assert.assertEquals("Van" , tokens001[3]);
Assert.assertEquals("Gogh" , tokens001[4]);
String[] tokens002 = tokenizerME.tokenize("He's so amazing");
Assert.assertEquals("He" , tokens002[0]);
Assert.assertEquals("'s" , tokens002[1]);
Assert.assertEquals("so" , tokens002[2]);
Assert.assertEquals("amazing", tokens002[3]);
}
catch (Exception e)
{
e.printStackTrace();
Assert.fail(e.getMessage());
}
}
示例3: tokenize
import opennlp.tools.tokenize.TokenizerME; //導入方法依賴的package包/類
String[] tokenize(String sentence) {
TokenizerME tokenizer = new TokenizerME(tokenModel);
String cleanString = sentence.replace('’', '\''); // this is the type of apostrophe that OpenNLP expects
return tokenizer.tokenize(cleanString);
}