本文整理汇总了C#中ITokenizer.HasNext方法的典型用法代码示例。如果您正苦于以下问题：C# ITokenizer.HasNext方法的具体用法？C# ITokenizer.HasNext怎么用？C# ITokenizer.HasNext使用的例子？那么恭喜您, 这里精选的方法代码示例或许可以为您提供帮助。您也可以进一步了解该方法所在类ITokenizer的用法示例。
在下文中一共展示了ITokenizer.HasNext方法的1个代码示例，这些例子默认根据受欢迎程度排序。您可以为喜欢或者感觉有用的代码点赞，您的评价将有助于系统推荐出更棒的C#代码示例。
示例1: PennTreeReader
/// <summary>
/// Read parse trees from a Reader.
/// </summary>
/// <param name="input">The Reader</param>
/// <param name="tf">TreeFactory -- factory to create some kind of Tree</param>
/// <param name="tn">the method of normalizing trees</param>
/// <param name="st">Tokenizer that divides up Reader</param>
public PennTreeReader(TextReader input, ITreeFactory tf, TreeNormalizer tn, ITokenizer<string> st)
{
reader = input;
treeFactory = tf;
treeNormalizer = tn;
tokenizer = st;
// check for whacked out headers still present in Brown corpus in Treebank 3
string first = (st.HasNext() ? st.Peek() : null);
if (first != null && first.StartsWith("*x*x*x"))
{
int foundCount = 0;
while (foundCount < 4 && st.HasNext())
{
first = st.Next();
if (first != null && first.StartsWith("*x*x*x"))
{
foundCount++;
}
}
}
}