当前位置: 首页>>代码示例>>TypeScript>>正文


TypeScript WordTokenizer.tokenize方法代码示例

本文整理汇总了TypeScript中natural.WordTokenizer.tokenize方法的典型用法代码示例。如果您正苦于以下问题:TypeScript WordTokenizer.tokenize方法的具体用法?TypeScript WordTokenizer.tokenize怎么用?TypeScript WordTokenizer.tokenize使用的例子?那么, 这里精选的方法代码示例或许可以为您提供帮助。您也可以进一步了解该方法所在natural.WordTokenizer的用法示例。


在下文中一共展示了WordTokenizer.tokenize方法的3个代码示例,这些例子默认根据受欢迎程度排序。您可以为喜欢或者感觉有用的代码点赞,您的评价将有助于系统推荐出更棒的TypeScript代码示例。

示例1: createTokens

function createTokens(text: string): Array<string> {
  const tokenized: Array<string> = tokenizer.tokenize(text);
  const biGrams = NGrams.ngrams(tokenized, 2).map((words) => words.join(' '));
  const triGrams = NGrams.ngrams(tokenized, 3).map((words) => words.join(' ')) as Array<string>;
  const ngrams: Array<string> = _.flatten(tokenized.concat(biGrams, triGrams)) as Array<string>;
  const allPhrases = ngrams.map(phrase => phrase.toLowerCase());
  return allPhrases;
}
开发者ID:fyndme,项目名称:bot-framework,代码行数:8,代码来源:helpers.ts

示例2: getLocationConfidence

export function getLocationConfidence(text: string, searchLocation: string): number {
  let matchingCities = _.map(locations, (subLocations, key) => {
    // console.log(subLocations);
    return _.map(subLocations, (cities) => {
      // console.log(cities, searchLocation);
      const normalizedCities: Array<string> = cities.map(city => city.toLowerCase());
      return _.includes(normalizedCities, searchLocation.toLowerCase()) ? normalizedCities : null;
    });
  });

  const cityList = _.compact(_.flattenDeep<string>(matchingCities) as Array<string>);
  const allPhrases = createTokens(text);
  const matchingPhrase = _.intersection(allPhrases, cityList);

  const textTokenized: Array<string> = tokenizer.tokenize(text);
  const locationTokenized = tokenizer.tokenize(matchingPhrase[0]);

  return locationTokenized.length / textTokenized.length;
}
开发者ID:fyndme,项目名称:bot-framework,代码行数:19,代码来源:helpers.ts

示例3: grabTopics

export function grabTopics(text: string): Promise<Intent> {
  const datesCompacted = runThroughClassifiers(text, dateClassifiers);
  const datesGrouped = _.groupBy(datesCompacted, 'topic');

  const specials = _.compact(tokenizer.tokenize(text).filter(token => !isNaN(parseInt(token, 10))));

  const intent: Intent = {
    action: null,
    details: {
      dates:  _.mapValues(datesGrouped, (classifications: Array<Classification>) => classifications.map(classification => _.startCase(classification.label))),
      specialWords: specials,
      locations: locatonExtractor(text),
    },
    topic: 'details',
  };

  // if (this && this.debugOn) { console.log('details intent', util.inspect(intent, { depth: null })); };

  return Promise.resolve(intent);
}
开发者ID:fyndme,项目名称:bot-framework,代码行数:20,代码来源:helpers.ts


注:本文中的natural.WordTokenizer.tokenize方法示例由纯净天空整理自Github/MSDocs等开源代码及文档管理平台,相关代码片段筛选自各路编程大神贡献的开源项目,源码版权归原作者所有,传播和使用请参考对应项目的License;未经允许,请勿转载。