当前位置: 首页>>代码示例>>Java>>正文

Java TableTunnel类代码示例

本文整理汇总了Java中com.aliyun.odps.tunnel.TableTunnel类的典型用法代码示例。如果您正苦于以下问题：Java TableTunnel类的具体用法？Java TableTunnel怎么用？Java TableTunnel使用的例子？那么, 这里精选的类代码示例或许可以为您提供帮助。


TableTunnel类属于com.aliyun.odps.tunnel包，在下文中一共展示了TableTunnel类的11个代码示例，这些例子默认根据受欢迎程度排序。您可以为喜欢或者感觉有用的代码点赞，您的评价将有助于系统推荐出更棒的Java代码示例。

示例1: createOdpsTableWithRecords

import com.aliyun.odps.tunnel.TableTunnel; //导入依赖的package包/类
protected void createOdpsTableWithRecords(String tableName,
    String[] partitionCols, int numRecords, ColumnGenerator... extraCols)
    throws OdpsException, IOException, ParseException {
  createOdpsTable(tableName, partitionCols, extraCols);
  TableTunnel tunnel = new TableTunnel(odps);
  PartitionSpec partitionSpec = getSimplePartitionSpec(partitionCols);
  TableTunnel.UploadSession uploadSession = partitionSpec == null
      ? tunnel.createUploadSession(PROJECT, tableName)
      : tunnel.createUploadSession(PROJECT, tableName, partitionSpec);
  RecordWriter recordWriter = uploadSession.openRecordWriter(0);
  for (int i = 0; i < numRecords; i++) {
    Record record = getOdpsRecord(uploadSession, i, extraCols);
    recordWriter.write(record);
  }
  recordWriter.close();
  uploadSession.commit(new Long[] {0L});
}
开发者ID:aliyun，项目名称:aliyun-maxcompute-data-collectors，代码行数:18，代码来源:OdpsExportTest.java

示例2: MaxComputeDataTransferUDTF

import com.aliyun.odps.tunnel.TableTunnel; //导入依赖的package包/类
public MaxComputeDataTransferUDTF() throws Exception {
    odpsConfig.init("odps.conf");

    Account account = new AliyunAccount(odpsConfig.getAccessId(), odpsConfig.getAccessKey());
    odps = new Odps(account);
    odps.setEndpoint(odpsConfig.getOdpsEndPoint());
    odps.setDefaultProject(odpsConfig.getProjectName());
    tunnel = new TableTunnel(odps);
    tunnel.setEndpoint(odpsConfig.getTunnelEndPoint());
}
开发者ID:aliyun，项目名称:aliyun-maxcompute-data-collectors，代码行数:11，代码来源:MaxComputeDataTransferUDTF.java

示例3: MaxComputeDataTransferUDTFMultiPart

import com.aliyun.odps.tunnel.TableTunnel; //导入依赖的package包/类
public MaxComputeDataTransferUDTFMultiPart() throws Exception {
    odpsConfig.init("odps.conf");

    Account account = new AliyunAccount(odpsConfig.getAccessId(), odpsConfig.getAccessKey());
    odps = new Odps(account);
    odps.setEndpoint(odpsConfig.getOdpsEndPoint());
    odps.setDefaultProject(odpsConfig.getProjectName());
    tunnel = new TableTunnel(odps);
    tunnel.setEndpoint(odpsConfig.getTunnelEndPoint());
}
开发者ID:aliyun，项目名称:aliyun-maxcompute-data-collectors，代码行数:11，代码来源:MaxComputeDataTransferUDTFMultiPart.java

示例4: OdpsTunnelWriter

import com.aliyun.odps.tunnel.TableTunnel; //导入依赖的package包/类
public OdpsTunnelWriter(TableTunnel tunnel, String project,
                        String tableName, int retryCount, String sessionId, boolean useCompress) {
  this.tunnel = tunnel;
  this.project = project;
  this.tableName = tableName;
  this.retryCount = retryCount;
  this.sharedSessionId = sessionId;
  this.useCompress = useCompress;
}
开发者ID:aliyun，项目名称:aliyun-maxcompute-data-collectors，代码行数:10，代码来源:OdpsTunnelWriter.java

示例5: buildTunnelWriter

import com.aliyun.odps.tunnel.TableTunnel; //导入依赖的package包/类
private OdpsWriter buildTunnelWriter(String project, String tableName,
                                     String tunnelEndPoint, int retryCount, String sessionId) {
  TableTunnel tunnel = new TableTunnel(odps);
  if (StringUtils.isNotEmpty(tunnelEndPoint)) {
    tunnel.setEndpoint(tunnelEndPoint);
  }
  return new OdpsTunnelWriter(tunnel, project, tableName, retryCount, sessionId, useCompress);
}
开发者ID:aliyun，项目名称:aliyun-maxcompute-data-collectors，代码行数:9，代码来源:OdpsUploadProcessor.java

示例6: getOdpsRecord

import com.aliyun.odps.tunnel.TableTunnel; //导入依赖的package包/类
private Record getOdpsRecord(TableTunnel.UploadSession uploadSession,
    int idx, ColumnGenerator... extraCols) throws ParseException {
  Record record = uploadSession.newRecord();
  record.setBigint("id", (long) idx);
  record.setString("msg", getMsgPrefix() + idx);
  int colNum = 0;
  for (ColumnGenerator generator: extraCols) {
    String field = forIdx(colNum++);
    String fieldValue = generator.getExportText(idx);
    switch (generator.getOdpsType()) {
      case STRING:
        record.setString(field, fieldValue);
        break;
      case BIGINT:
        record.setBigint(field, Long.parseLong(fieldValue));
        break;
      case DATETIME:
        String dateFormat = generator.getDateFormat();
          record.setDatetime(field,
              new SimpleDateFormat(dateFormat).parse(fieldValue));
        break;
      case DOUBLE:
        record.setDouble(field, Double.parseDouble(fieldValue));
        break;
      case DECIMAL:
        record.setDecimal(field, new BigDecimal(fieldValue));
        break;
      default:
        throw new RuntimeException("Unknown column type: "
            + generator.getOdpsType());
    }
  }
  return record;
}
开发者ID:aliyun，项目名称:aliyun-maxcompute-data-collectors，代码行数:35，代码来源:OdpsExportTest.java

示例7: verifyOdpsDataset

import com.aliyun.odps.tunnel.TableTunnel; //导入依赖的package包/类
protected void verifyOdpsDataset(String tableName, String partitionSpec,
                                 Object[][] valsArray)
    throws OdpsException, IOException {
  TableTunnel tunnel = new TableTunnel(odps);
  TableTunnel.DownloadSession downloadSession;
  if (partitionSpec != null) {
    downloadSession = tunnel.createDownloadSession(PROJECT, tableName,
        new PartitionSpec(partitionSpec));
  } else {
    downloadSession = tunnel.createDownloadSession(PROJECT, tableName);
  }
  long count = downloadSession.getRecordCount();
  RecordReader recordReader = downloadSession.openRecordReader(0, count);
  try {
    List<String> expectations = new ArrayList<String>();
    if (valsArray != null) {
      for (Object[] vals: valsArray) {
        expectations.add(Arrays.toString(vals));
      }
    }
    Record record;
    while ((record = recordReader.read()) != null
        && expectations.size() > 0) {
      String actual = Arrays.toString(convertOdpsRecordToArray(record));
      assertTrue("Expect record: " + actual, expectations.remove(actual));
    }
    assertFalse(record != null);
    assertEquals(0, expectations.size());
  } finally {
    recordReader.close();
  }
}
开发者ID:aliyun，项目名称:aliyun-maxcompute-data-collectors，代码行数:33，代码来源:OdpsTestCase.java

示例8: createTableTunnel

import com.aliyun.odps.tunnel.TableTunnel; //导入依赖的package包/类
private TableTunnel createTableTunnel(){
	  Account account = new AliyunAccount(accessId, accessKey);
      Odps odps = new Odps(account);
      odps.setEndpoint(odpsUrl);
      odps.setDefaultProject(project);	
      return new TableTunnel(odps);
}
开发者ID:DTStack，项目名称:jlogstash-output-plugin，代码行数:8，代码来源:OutOdps.java

示例9: init

import com.aliyun.odps.tunnel.TableTunnel; //导入依赖的package包/类
public boolean init(StepMetaInterface smi, StepDataInterface sdi) {
    if (super.init(smi, sdi)) {
        meta = (OdpsInputMeta) smi;
        data = (OdpsInputData) sdi;

        Account account = new AliyunAccount(environmentSubstitute(meta.getAccessId()),
            environmentSubstitute(meta.getAccessKey()));
        Odps odps = new Odps(account);
        odps.setEndpoint(environmentSubstitute(meta.getEndpoint()));
        odps.setDefaultProject(environmentSubstitute(meta.getProjectName()));
        odps.setUserAgent("Maxcompute-Kettle-Plugin-2.0.0");

        TableTunnel tableTunnel = new TableTunnel(odps);
        String tunnelEndpoint = environmentSubstitute(meta.getTunnelEndpoint());
        if (!StringUtils.isEmpty(tunnelEndpoint)) {
            tableTunnel.setEndpoint(tunnelEndpoint);
        }
        DownloadSession downloadSession = null;
        try {
            if (meta.getPartition() != null && !meta.getPartition().trim().equals("")) {
                PartitionSpec partitionSpec =
                    new PartitionSpec(environmentSubstitute(meta.getPartition()));
                downloadSession = tableTunnel
                    .createDownloadSession(environmentSubstitute(meta.getProjectName()),
                        environmentSubstitute(meta.getTableName()), partitionSpec);
            } else {
                downloadSession = tableTunnel
                    .createDownloadSession(environmentSubstitute(meta.getProjectName()),
                        environmentSubstitute(meta.getTableName()));
            }

            schema = downloadSession.getSchema();
            initOdpsFieldPosMap(schema);

            long count = downloadSession.getRecordCount();
            logBasic("count is: " + count);
            data.tunnelRecordReader = downloadSession.openRecordReader(0L, count);
            return true;
        } catch (TunnelException e) {
            logError(e.getMessage(), e);
        } catch (Exception ex) {
            logError(ex.getMessage(), ex);
        }
    }
    return false;
}
开发者ID:aliyun，项目名称:aliyun-maxcompute-data-collectors，代码行数:47，代码来源:OdpsInput.java

示例10: init

import com.aliyun.odps.tunnel.TableTunnel; //导入依赖的package包/类
public boolean init(StepMetaInterface smi, StepDataInterface sdi) {
    if (super.init(smi, sdi)) {
        meta = (OdpsOutputMeta) smi;
        data = (OdpsOutputData) sdi;

        Account account = new AliyunAccount(environmentSubstitute(meta.getAccessId()),
            environmentSubstitute(meta.getAccessKey()));
        Odps odps = new Odps(account);
        odps.setEndpoint(environmentSubstitute(meta.getEndpoint()));
        odps.setDefaultProject(environmentSubstitute(meta.getProjectName()));
        odps.setUserAgent("Maxcompute-Kettle-Plugin-2.0.0");

        TableTunnel tableTunnel = new TableTunnel(odps);
        String tunnelEndpoint = environmentSubstitute(meta.getTunnelEndpoint());
        if (!StringUtils.isEmpty(tunnelEndpoint)) {
            tableTunnel.setEndpoint(tunnelEndpoint);
        }
        try {
            MaxcomputeUtil.dealTruncate(odps,
                odps.tables().get(environmentSubstitute(meta.getTableName())),
                environmentSubstitute(meta.getPartition()), meta.isTruncate());

            if (meta.getPartition() != null && !meta.getPartition().trim().equals("")) {
                PartitionSpec partitionSpec =
                    new PartitionSpec(environmentSubstitute(meta.getPartition()));
                data.uploadSession = tableTunnel
                    .createUploadSession(environmentSubstitute(meta.getProjectName()),
                        environmentSubstitute(meta.getTableName()), partitionSpec);
            } else {
                data.uploadSession = tableTunnel
                    .createUploadSession(environmentSubstitute(meta.getProjectName()),
                        environmentSubstitute(meta.getTableName()));
            }

            schema = data.uploadSession.getSchema();
            initOdpsFieldPosMap(schema);
            initOdpsColumn2StreamFieldMap();

            data.recordWriter = data.uploadSession.openBufferedWriter();

            return true;
        } catch (TunnelException e) {
            logError(e.getMessage(), e);
        } catch (Exception ex) {
            logError(ex.getMessage(), ex);
        }
    }
    return false;
}
开发者ID:aliyun，项目名称:aliyun-maxcompute-data-collectors，代码行数:50，代码来源:OdpsOutput.java

示例11: setConf

import com.aliyun.odps.tunnel.TableTunnel; //导入依赖的package包/类
@Override
public void setConf(Configuration configuration) {
  this.conf = configuration;
  rowDOList = new LinkedList<OdpsRowDO>();

  inputDateFormat = conf.get(OdpsConstants.DATE_FORMAT);
  retryCount = conf.getInt(OdpsConstants.RETRY_COUNT,
          OdpsConstants.DEFAULT_RETRY_COUNT);
  batchSize = conf.getInt(OdpsConstants.BATCH_SIZE,
          OdpsConstants.DEFAULT_BATCH_SIZE);
  useCompress = conf.getBoolean(OdpsConstants.USE_COMPRESS_IN_UPLOAD, false);

  String project = conf.get(OdpsConstants.PROJECT);
  String endpoint = conf.get(OdpsConstants.ENDPOINT);
  String tableName = conf.get(OdpsConstants.TABLE_NAME);
  String tunnelEndPoint = OdpsUtil.getTunnelEndPoint(conf.get(OdpsConstants.TUNNEL_ENDPOINT));

  odps = new Odps(new AliyunAccount(conf.get(OdpsConstants.ACCESS_ID),
          conf.get(OdpsConstants.ACCESS_KEY)));
  odps.setUserAgent(OdpsUtil.getUserAgent());
  odpsTable = buildOdpsTable(odps, project, endpoint, tableName);

  partitionKeys = strToArray(conf.get(OdpsConstants.PARTITION_KEY));
  partitionValues = strToArray(conf.get(OdpsConstants.PARTITION_VALUE));
  if (partitionKeys != null) {
    partitionMap = buildPartitionMap();
  }

  List<String> inputColumnNames = Arrays.asList(
          conf.getStrings(OdpsConstants.INPUT_COL_NAMES));
  odpsRecordBuilder = new OdpsRecordBuilder(odpsTable,
          inputDateFormat, inputColumnNames);
  try {
    if (conf.getBoolean(OdpsConstants.ODPS_DISABLE_DYNAMIC_PARTITIONS, false)) {
      String partition = getPartitionSpec(partitionKeys, partitionValues, Maps.newHashMap());
      TableTunnel.UploadSession uploadSession = null;
      TableTunnel tunnel = new TableTunnel(odps);
      if (StringUtils.isNotEmpty(tunnelEndPoint)) {
        tunnel.setEndpoint(tunnelEndPoint);
      }
      if (partition == null) {
        uploadSession = tunnel.createUploadSession(project, tableName);
      } else {
        uploadSession =
            tunnel.createUploadSession(project, tableName, new PartitionSpec(partition));
      }
      odpsWriter =
          buildTunnelWriter(project, tableName, tunnelEndPoint, retryCount, uploadSession);
    } else {
      odpsWriter =
          buildTunnelWriter(project, tableName, tunnelEndPoint, retryCount, new String(""));
    }
  } catch (Exception e) {
    throw new RuntimeException(e);
  }
}
开发者ID:aliyun，项目名称:aliyun-maxcompute-data-collectors，代码行数:57，代码来源:OdpsUploadProcessor.java


注：本文中的com.aliyun.odps.tunnel.TableTunnel类示例由纯净天空整理自Github/MSDocs等开源代码及文档管理平台，相关代码片段筛选自各路编程大神贡献的开源项目，源码版权归原作者所有，传播和使用请参考对应项目的License；未经允许，请勿转载。