当前位置: 首页>>代码示例>>Java>>正文


Java Tokenizer类代码示例

本文整理汇总了Java中com.atilika.kuromoji.ipadic.Tokenizer的典型用法代码示例。如果您正苦于以下问题:Java Tokenizer类的具体用法?Java Tokenizer怎么用?Java Tokenizer使用的例子?那么恭喜您, 这里精选的类代码示例或许可以为您提供帮助。


Tokenizer类属于com.atilika.kuromoji.ipadic包,在下文中一共展示了Tokenizer类的8个代码示例,这些例子默认根据受欢迎程度排序。您可以为喜欢或者感觉有用的代码点赞,您的评价将有助于系统推荐出更棒的Java代码示例。

示例1: translate

import com.atilika.kuromoji.ipadic.Tokenizer; //导入依赖的package包/类
/** Attempts to translate a text using the word list. */
public String translate(Tokenizer tok, String text)
{
	StringBuilder res = new StringBuilder();
	
	for(Token t : tok.tokenize(text)) {
		if(Transliterator.hasAsianChar(t.getSurface())) {
			// attempt (partial) translation
			String s = t.getSurface();
			String r = StringUtils.replaceEach(s, srcEverywhere, trlEverywhere);
			r = replaceAnyOfAtStart(r, srcPrefix, trlPrefix);
			r = replaceAnyOfAtEnd(r, srcSuffix, trlSuffix);
			if(r != s && verbose > 1)
				System.out.println("translation list: translated " + s + " to " + r);
			res.append(r);				
		}
		else // tokens without asian chars can be ignored here and will be copied directly
			res.append(t.getSurface());
	}
	return res.toString();
}
 
开发者ID:rrobek,项目名称:JapanMapTranslate,代码行数:22,代码来源:TranslationList.java

示例2: Transliterator

import com.atilika.kuromoji.ipadic.Tokenizer; //导入依赖的package包/类
public Transliterator()
{
   	tokKanji = new Tokenizer(); 
   	
   	trHiragana = new Kakasi();
   	trHiragana.setupHiraganaConverter(Kakasi.ASCII);
   	trHiragana.setRomajiCapitalizeMode(true);
   	
   	trKatakana = new Kakasi();
   	trKatakana.setupKatakanaConverter(Kakasi.ASCII);
   	trKatakana.setRomajiCapitalizeMode(true);		
}
 
开发者ID:rrobek,项目名称:JapanMapTranslate,代码行数:13,代码来源:Transliterator.java

示例3: getTokenizer

import com.atilika.kuromoji.ipadic.Tokenizer; //导入依赖的package包/类
private static synchronized Tokenizer getTokenizer() {
    if (token == null) {
        final Tokenizer.Builder builder = new Tokenizer.Builder();
        builder.mode(Tokenizer.Mode.SEARCH);
        token = builder.build();
    }
    return token;
}
 
开发者ID:chory-amam,项目名称:slack-capybara,代码行数:9,代码来源:WordAnalyzer.java

示例4: analyze

import com.atilika.kuromoji.ipadic.Tokenizer; //导入依赖的package包/类
public static List<String> analyze(final String sentence) {
    Tokenizer search = getTokenizer();
    final List<Token> tokens = search.tokenize(sentence);
    final List<String> analiezed = Lists.newArrayList();
    for (final Token token : tokens) {
        final String word = token.getSurface();
        analiezed.add(word);
    }
    return analiezed;
}
 
开发者ID:chory-amam,项目名称:slack-capybara,代码行数:11,代码来源:WordAnalyzer.java

示例5: getTokenizer

import com.atilika.kuromoji.ipadic.Tokenizer; //导入依赖的package包/类
Tokenizer getTokenizer() {
	return tokKanji;
}
 
开发者ID:rrobek,项目名称:JapanMapTranslate,代码行数:4,代码来源:Transliterator.java

示例6: Tokenizer

import com.atilika.kuromoji.ipadic.Tokenizer; //导入依赖的package包/类
/**
 * Construct a default tokenizer
 */
public Tokenizer() {
    this(new Builder());
}
 
开发者ID:deeplearning4j,项目名称:deeplearning4j,代码行数:7,代码来源:Tokenizer.java

示例7: JapaneseTokenizer

import com.atilika.kuromoji.ipadic.Tokenizer; //导入依赖的package包/类
/**
 * Tokenize the string with Kuromoji, optionally using baseForms.
 *
 * Note: It is safe to create new instances from multiple threads.
 * @param kuromoji The kuromoji instance.
 * @param toTokenize The string to tokenize.
 * @param useBaseForm normalize conjugations "走った" -> "走る" instead of "走っ"
 */
public JapaneseTokenizer(Tokenizer kuromoji, String toTokenize, boolean useBaseForm) {
    this.useBaseForm = useBaseForm;
    this.tokens = kuromoji.tokenize(toTokenize);
    this.tokenCount = this.tokens.size();
    this.currentToken = 0;
}
 
开发者ID:deeplearning4j,项目名称:deeplearning4j,代码行数:15,代码来源:JapaneseTokenizer.java

示例8: build

import com.atilika.kuromoji.ipadic.Tokenizer; //导入依赖的package包/类
/**
 * Creates the custom tokenizer instance
 *
 * @return tokenizer instance, not null
 */
@Override
public Tokenizer build() {
    return new Tokenizer(this);
}
 
开发者ID:deeplearning4j,项目名称:deeplearning4j,代码行数:10,代码来源:Tokenizer.java


注:本文中的com.atilika.kuromoji.ipadic.Tokenizer类示例由纯净天空整理自Github/MSDocs等开源代码及文档管理平台,相关代码片段筛选自各路编程大神贡献的开源项目,源码版权归原作者所有,传播和使用请参考对应项目的License;未经允许,请勿转载。