当前位置: 首页>>代码示例>>Java>>正文


Java KeyValue.heapSize方法代码示例

本文整理汇总了Java中org.apache.hadoop.hbase.KeyValue.heapSize方法的典型用法代码示例。如果您正苦于以下问题:Java KeyValue.heapSize方法的具体用法?Java KeyValue.heapSize怎么用?Java KeyValue.heapSize使用的例子?那么, 这里精选的方法代码示例或许可以为您提供帮助。您也可以进一步了解该方法所在org.apache.hadoop.hbase.KeyValue的用法示例。


在下文中一共展示了KeyValue.heapSize方法的1个代码示例,这些例子默认根据受欢迎程度排序。您可以为喜欢或者感觉有用的代码点赞,您的评价将有助于系统推荐出更棒的Java代码示例。

示例1: reduce

import org.apache.hadoop.hbase.KeyValue; //导入方法依赖的package包/类
@Override
protected void reduce(K row, Iterable<Put> vals, Context context)
    throws IOException, InterruptedException {
  // Using HeapSize to create an upper bound on the memory size of
  // the puts and flush some portion of the content while looping. This
  // flush could result in multiple Puts for a single rowkey. That is
  // acceptable because Combiner is run as an optimization and it's not
  // critical that all Puts are grouped perfectly.
  long threshold = context.getConfiguration().getLong(
      "putcombiner.row.threshold", 1L * (1<<30));
  int cnt = 0;
  long curSize = 0;
  Put put = null;
  Map<byte[], List<Cell>> familyMap = null;
  for (Put p : vals) {
    cnt++;
    if (put == null) {
      put = p;
      familyMap = put.getFamilyCellMap();
    } else {
      for (Entry<byte[], List<Cell>> entry : p.getFamilyCellMap()
          .entrySet()) {
        List<Cell> cells = familyMap.get(entry.getKey());
        List<Cell> kvs = (cells != null) ? (List<Cell>) cells : null;
        for (Cell cell : entry.getValue()) {
          KeyValue kv = KeyValueUtil.ensureKeyValueTypeForMR(cell);
          curSize += kv.heapSize();
          if (kvs != null) {
            kvs.add(kv);
          }
        }
        if (cells == null) {
          familyMap.put(entry.getKey(), entry.getValue());
        }
      }
      if (cnt % 10 == 0) context.setStatus("Combine " + cnt);
      if (curSize > threshold) {
        if (LOG.isDebugEnabled()) {
          LOG.debug(String.format("Combined %d Put(s) into %d.", cnt, 1));
        }
        context.write(row, put);
        put = null;
        curSize = 0;
        cnt = 0;
      }
    }
  }
  if (put != null) {
    if (LOG.isDebugEnabled()) {
      LOG.debug(String.format("Combined %d Put(s) into %d.", cnt, 1));
    }
    context.write(row, put);
  }
}
 
开发者ID:fengchen8086,项目名称:ditb,代码行数:55,代码来源:PutCombiner.java


注:本文中的org.apache.hadoop.hbase.KeyValue.heapSize方法示例由纯净天空整理自Github/MSDocs等开源代码及文档管理平台,相关代码片段筛选自各路编程大神贡献的开源项目,源码版权归原作者所有,传播和使用请参考对应项目的License;未经允许,请勿转载。