當前位置: 首頁>>代碼示例>>Java>>正文


Java ApplicationSubmissionContext.getApplicationId方法代碼示例

本文整理匯總了Java中org.apache.hadoop.yarn.api.records.ApplicationSubmissionContext.getApplicationId方法的典型用法代碼示例。如果您正苦於以下問題:Java ApplicationSubmissionContext.getApplicationId方法的具體用法?Java ApplicationSubmissionContext.getApplicationId怎麽用?Java ApplicationSubmissionContext.getApplicationId使用的例子?那麽, 這裏精選的方法代碼示例或許可以為您提供幫助。您也可以進一步了解該方法所在org.apache.hadoop.yarn.api.records.ApplicationSubmissionContext的用法示例。


在下文中一共展示了ApplicationSubmissionContext.getApplicationId方法的15個代碼示例,這些例子默認根據受歡迎程度排序。您可以為喜歡或者感覺有用的代碼點讚,您的評價將有助於係統推薦出更棒的Java代碼示例。

示例1: submitApplication

import org.apache.hadoop.yarn.api.records.ApplicationSubmissionContext; //導入方法依賴的package包/類
@Override
protected void submitApplication(
    ApplicationSubmissionContext submissionContext, long submitTime,
    String user) throws YarnException {
  //Do nothing, just add the application to RMContext
  RMAppImpl application =
      new RMAppImpl(submissionContext.getApplicationId(), this.rmContext,
          this.conf, submissionContext.getApplicationName(), user,
          submissionContext.getQueue(), submissionContext,
          this.rmContext.getScheduler(),
          this.rmContext.getApplicationMasterService(),
          submitTime, submissionContext.getApplicationType(),
          submissionContext.getApplicationTags(), null);
  this.rmContext.getRMApps().put(submissionContext.getApplicationId(),
      application);
  //Do not send RMAppEventType.START event
  //so the state of Application will not reach to NEW_SAVING state.
}
 
開發者ID:naver,項目名稱:hadoop,代碼行數:19,代碼來源:RMHATestBase.java

示例2: submitApplication

import org.apache.hadoop.yarn.api.records.ApplicationSubmissionContext; //導入方法依賴的package包/類
@SuppressWarnings("deprecation")
@Override
public void submitApplication(ApplicationSubmissionContext context)
    throws IOException {
  int waitingTime = conf.getInt(
      HPCConfiguration.YARN_APPLICATION_HPC_CLIENT_RS_MAX_WAIT_MS,
      HPCConfiguration.DEFAULT_YARN_APPLICATION_HPC_CLIENT_RS_MAX_WAIT_MS);
  int noOfTimes = conf.getInt(
      HPCConfiguration.YARN_APPLICATION_HPC_CLIENT_RS_RETRIES_MAX,
      HPCConfiguration.DEFAULT_YARN_APPLICATION_HPC_CLIENT_RS_RETRIES_MAX);
  ApplicationId applicationId = context.getApplicationId();
  String nodeName = checkAndWaitForResourcesToBeAvailable(
      applicationId, waitingTime, noOfTimes);
      
  HPCCommandExecutor.launchContainer(
  		context.getAMContainerSpec(),
      ContainerId.newInstance(
          ApplicationAttemptId.newInstance(applicationId, 1), 1).toString(),
      context.getApplicationName(), conf, applicationId.getId(), nodeName);
  HPCCommandExecutor.setJobState(applicationId.getId(), "running::0", conf);
}
 
開發者ID:intel-hpdd,項目名稱:scheduling-connector-for-hadoop,代碼行數:22,代碼來源:SlurmApplicationClient.java

示例3: recoverApplication

import org.apache.hadoop.yarn.api.records.ApplicationSubmissionContext; //導入方法依賴的package包/類
protected void recoverApplication(ApplicationStateData appState,
    RMState rmState) throws Exception {
  ApplicationSubmissionContext appContext =
      appState.getApplicationSubmissionContext();
  ApplicationId appId = appContext.getApplicationId();

  // create and recover app.
  RMAppImpl application =
      createAndPopulateNewRMApp(appContext, appState.getSubmitTime(),
          appState.getUser(), true);

  application.handle(new RMAppRecoverEvent(appId, rmState));
}
 
開發者ID:naver,項目名稱:hadoop,代碼行數:14,代碼來源:RMAppManager.java

示例4: createAndPopulateNewRMApp

import org.apache.hadoop.yarn.api.records.ApplicationSubmissionContext; //導入方法依賴的package包/類
private RMAppImpl createAndPopulateNewRMApp(
    ApplicationSubmissionContext submissionContext, long submitTime,
    String user, boolean isRecovery) throws YarnException {
  ApplicationId applicationId = submissionContext.getApplicationId();
  ResourceRequest amReq =
      validateAndCreateResourceRequest(submissionContext, isRecovery);

  // Create RMApp
  RMAppImpl application =
      new RMAppImpl(applicationId, rmContext, this.conf,
          submissionContext.getApplicationName(), user,
          submissionContext.getQueue(),
          submissionContext, this.scheduler, this.masterService,
          submitTime, submissionContext.getApplicationType(),
          submissionContext.getApplicationTags(), amReq);

  // Concurrent app submissions with same applicationId will fail here
  // Concurrent app submissions with different applicationIds will not
  // influence each other
  if (rmContext.getRMApps().putIfAbsent(applicationId, application) !=
      null) {
    String message = "Application with id " + applicationId
        + " is already present! Cannot add a duplicate!";
    LOG.warn(message);
    throw new YarnException(message);
  }
  // Inform the ACLs Manager
  this.applicationACLsManager.addApplication(applicationId,
      submissionContext.getAMContainerSpec().getApplicationACLs());
  String appViewACLs = submissionContext.getAMContainerSpec()
      .getApplicationACLs().get(ApplicationAccessType.VIEW_APP);
  rmContext.getSystemMetricsPublisher().appACLsUpdated(
      application, appViewACLs, System.currentTimeMillis());
  return application;
}
 
開發者ID:naver,項目名稱:hadoop,代碼行數:36,代碼來源:RMAppManager.java

示例5: submitApplication

import org.apache.hadoop.yarn.api.records.ApplicationSubmissionContext; //導入方法依賴的package包/類
@Override
public void submitApplication(ApplicationSubmissionContext context)
    throws IOException {
  int waitingTime = conf.getInt(
      HPCConfiguration.YARN_APPLICATION_HPC_CLIENT_RS_MAX_WAIT_MS,
      HPCConfiguration.DEFAULT_YARN_APPLICATION_HPC_CLIENT_RS_MAX_WAIT_MS);
  int noOfTimes = conf.getInt(
      HPCConfiguration.YARN_APPLICATION_HPC_CLIENT_RS_RETRIES_MAX,
      HPCConfiguration.DEFAULT_YARN_APPLICATION_HPC_CLIENT_RS_RETRIES_MAX);
  ApplicationId applicationId = context.getApplicationId();

  String applicationName = context.getApplicationName();
  SocketWrapper socket = SocketCache.getSocket(applicationId.getId());
  if (socket.waitForReady(waitingTime * noOfTimes)) {
    PBSCommandExecutor.launchContainer(
        context.getAMContainerSpec(),
        ContainerId.newContainerId(
            ApplicationAttemptId.newInstance(applicationId, 1), 1l)
            .toString(), applicationName, conf, applicationId.getId(), true,
        socket.getContainerHostName());
  }

  // Set the Job Name
  int jobid = applicationId.getId();
  String pbsJobName = applicationName.replaceAll("\\s", "");
  if (pbsJobName.length() > 13) {
    pbsJobName = pbsJobName.substring(0, 12);
  }

  String qalterCmd = conf.get(
      HPCConfiguration.YARN_APPLICATION_HPC_COMMAND_PBS_QALTER,
      HPCConfiguration.DEFAULT_YARN_APPLICATION_HPC_COMMAND_PBS_QALTER);
  Shell
      .execCommand(qalterCmd, String.valueOf(jobid), "-N", "Y#" + pbsJobName);
}
 
開發者ID:intel-hpdd,項目名稱:scheduling-connector-for-hadoop,代碼行數:36,代碼來源:PBSApplicationClient.java

示例6: placeApplication

import org.apache.hadoop.yarn.api.records.ApplicationSubmissionContext; //導入方法依賴的package包/類
public void placeApplication(ApplicationSubmissionContext asc, String user)
    throws YarnException {
  try {
    readLock.lock();
    if (null == rules || rules.isEmpty()) {
      return;
    }
    
    String newQueueName = null;
    for (PlacementRule rule : rules) {
      newQueueName = rule.getQueueForApp(asc, user);
      if (newQueueName != null) {
        break;
      }
    }
    
    // Failed to get where to place application
    if (null == newQueueName && null == asc.getQueue()) {
      String msg = "Failed to get where to place application="
          + asc.getApplicationId();
      LOG.error(msg);
      throw new YarnException(msg);
    }
    
    // Set it to ApplicationSubmissionContext
    if (!StringUtils.equals(asc.getQueue(), newQueueName)) {
      LOG.info("Placed application=" + asc.getApplicationId() + " to queue="
          + newQueueName + ", original queue=" + asc.getQueue());
      asc.setQueue(newQueueName);
    }
  } finally {
    readLock.unlock();
  }
}
 
開發者ID:aliyun-beta,項目名稱:aliyun-oss-hadoop-fs,代碼行數:35,代碼來源:PlacementManager.java

示例7: getQueueForApp

import org.apache.hadoop.yarn.api.records.ApplicationSubmissionContext; //導入方法依賴的package包/類
@Override
public String getQueueForApp(ApplicationSubmissionContext asc, String user)
    throws YarnException {
  String queueName = asc.getQueue();
  ApplicationId applicationId = asc.getApplicationId();
  if (mappings != null && mappings.size() > 0) {
    try {
      String mappedQueue = getMappedQueue(user);
      if (mappedQueue != null) {
        // We have a mapping, should we use it?
        if (queueName.equals(YarnConfiguration.DEFAULT_QUEUE_NAME)
            || overrideWithQueueMappings) {
          LOG.info("Application " + applicationId + " user " + user
              + " mapping [" + queueName + "] to [" + mappedQueue
              + "] override " + overrideWithQueueMappings);
          return mappedQueue;
        }
      }
    } catch (IOException ioex) {
      String message = "Failed to submit application " + applicationId +
          " submitted by user " + user + " reason: " + ioex.getMessage();
      throw new YarnException(message);
    }
  }
  
  return queueName;
}
 
開發者ID:aliyun-beta,項目名稱:aliyun-oss-hadoop-fs,代碼行數:28,代碼來源:UserGroupMappingPlacementRule.java

示例8: run

import org.apache.hadoop.yarn.api.records.ApplicationSubmissionContext; //導入方法依賴的package包/類
public ApplicationId run() {
    client.start();

    YarnClientApplication app = client.newApp();
    ApplicationSubmissionContext am = setupAM(app);
    ApplicationId appId = client.submitApp(am);
    return am.getApplicationId();
}
 
開發者ID:xushjie1987,項目名稱:es-hadoop-v2.2.0,代碼行數:9,代碼來源:YarnLauncher.java

示例9: startAppMaster

import org.apache.hadoop.yarn.api.records.ApplicationSubmissionContext; //導入方法依賴的package包/類
private ApplicationReport startAppMaster(ApplicationSubmissionContext appContext) throws Exception {
  appContext.setMaxAppAttempts(MAX_ATTEMPT);

  Map<String, LocalResource> localResources = new HashMap<>();
  Set<Path> shippedPaths = new HashSet<>();
  collectLocalResources(localResources, shippedPaths);
  final ContainerLaunchContext amContainer = setupApplicationMasterContainer(false, true, false);

  amContainer.setLocalResources(localResources);

  final String classPath = localResources.keySet().stream().collect(Collectors.joining(File.pathSeparator));
  final String shippedFiles = shippedPaths.stream().map(Path::toString)
      .collect(Collectors.joining(","));

  // Setup CLASSPATH and environment variables for ApplicationMaster
  ApplicationId appId = appContext.getApplicationId();
  final Map<String, String> appMasterEnv = setUpAmEnvironment(
      appId,
      classPath,
      shippedFiles,
      getDynamicPropertiesEncoded()
  );

  amContainer.setEnvironment(appMasterEnv);

  // Set up resource type requirements for ApplicationMaster
  Resource capability = Records.newRecord(Resource.class);
  capability.setMemory(getFlinkConfiguration()
      .getInteger(JobManagerOptions.JOB_MANAGER_HEAP_MEMORY));
  capability.setVirtualCores(1);

  appContext.setApplicationName(job.name());
  appContext.setApplicationType(ATHENAX_APPLICATION_TYPE);
  appContext.setAMContainerSpec(amContainer);
  appContext.setResource(capability);
  appContext.setApplicationTags(Collections.singleton(job.metadata().serialize()));
  if (job.queue() != null) {
    appContext.setQueue(job.queue());
  }

  LOG.info("Submitting application master {}", appId);
  yarnClient.submitApplication(appContext);

  PollDeploymentStatus poll = new PollDeploymentStatus(appId);
  YARN_POLL_EXECUTOR.submit(poll);
  try {
    return poll.result.get();
  } catch (ExecutionException e) {
    LOG.warn("Failed to deploy {}, cause: {}", appId.toString(), e.getCause());
    yarnClient.killApplication(appId);
    throw (Exception) e.getCause();
  }
}
 
開發者ID:uber,項目名稱:AthenaX,代碼行數:54,代碼來源:AthenaXYarnClusterDescriptor.java

示例10: submitApplication

import org.apache.hadoop.yarn.api.records.ApplicationSubmissionContext; //導入方法依賴的package包/類
@Override
public SubmitApplicationResponse submitApplication(
    SubmitApplicationRequest request) throws YarnException {
  ApplicationSubmissionContext submissionContext = request
      .getApplicationSubmissionContext();
  ApplicationId applicationId = submissionContext.getApplicationId();

  // ApplicationSubmissionContext needs to be validated for safety - only
  // those fields that are independent of the RM's configuration will be
  // checked here, those that are dependent on RM configuration are validated
  // in RMAppManager.

  String user = null;
  try {
    // Safety
    user = UserGroupInformation.getCurrentUser().getShortUserName();
  } catch (IOException ie) {
    LOG.warn("Unable to get the current user.", ie);
    RMAuditLogger.logFailure(user, AuditConstants.SUBMIT_APP_REQUEST,
        ie.getMessage(), "ClientRMService",
        "Exception in submitting application", applicationId);
    throw RPCUtil.getRemoteException(ie);
  }

  // Check whether app has already been put into rmContext,
  // If it is, simply return the response
  if (rmContext.getRMApps().get(applicationId) != null) {
    LOG.info("This is an earlier submitted application: " + applicationId);
    return SubmitApplicationResponse.newInstance();
  }

  if (submissionContext.getQueue() == null) {
    submissionContext.setQueue(YarnConfiguration.DEFAULT_QUEUE_NAME);
  }
  if (submissionContext.getApplicationName() == null) {
    submissionContext.setApplicationName(
        YarnConfiguration.DEFAULT_APPLICATION_NAME);
  }
  if (submissionContext.getApplicationType() == null) {
    submissionContext
      .setApplicationType(YarnConfiguration.DEFAULT_APPLICATION_TYPE);
  } else {
    if (submissionContext.getApplicationType().length() > YarnConfiguration.APPLICATION_TYPE_LENGTH) {
      submissionContext.setApplicationType(submissionContext
        .getApplicationType().substring(0,
          YarnConfiguration.APPLICATION_TYPE_LENGTH));
    }
  }

  try {
    // call RMAppManager to submit application directly
    rmAppManager.submitApplication(submissionContext,
        System.currentTimeMillis(), user);

    LOG.info("Application with id " + applicationId.getId() + 
        " submitted by user " + user);
    RMAuditLogger.logSuccess(user, AuditConstants.SUBMIT_APP_REQUEST,
        "ClientRMService", applicationId);
  } catch (YarnException e) {
    LOG.info("Exception in submitting application with id " +
        applicationId.getId(), e);
    RMAuditLogger.logFailure(user, AuditConstants.SUBMIT_APP_REQUEST,
        e.getMessage(), "ClientRMService",
        "Exception in submitting application", applicationId);
    throw e;
  }

  SubmitApplicationResponse response = recordFactory
      .newRecordInstance(SubmitApplicationResponse.class);
  return response;
}
 
開發者ID:naver,項目名稱:hadoop,代碼行數:72,代碼來源:ClientRMService.java

示例11: submitApplication

import org.apache.hadoop.yarn.api.records.ApplicationSubmissionContext; //導入方法依賴的package包/類
@Override
public ApplicationId
    submitApplication(ApplicationSubmissionContext appContext)
        throws YarnException, IOException {
  ApplicationId applicationId = appContext.getApplicationId();
  if (applicationId == null) {
    throw new ApplicationIdNotProvidedException(
        "ApplicationId is not provided in ApplicationSubmissionContext");
  }
  SubmitApplicationRequest request =
      Records.newRecord(SubmitApplicationRequest.class);
  request.setApplicationSubmissionContext(appContext);

  // Automatically add the timeline DT into the CLC
  // Only when the security and the timeline service are both enabled
  if (isSecurityEnabled() && timelineServiceEnabled) {
    addTimelineDelegationToken(appContext.getAMContainerSpec());
  }

  //TODO: YARN-1763:Handle RM failovers during the submitApplication call.
  rmClient.submitApplication(request);

  int pollCount = 0;
  long startTime = System.currentTimeMillis();
  EnumSet<YarnApplicationState> waitingStates = 
                               EnumSet.of(YarnApplicationState.NEW,
                               YarnApplicationState.NEW_SAVING,
                               YarnApplicationState.SUBMITTED);
  EnumSet<YarnApplicationState> failToSubmitStates = 
                                EnumSet.of(YarnApplicationState.FAILED,
                                YarnApplicationState.KILLED);		
  while (true) {
    try {
      ApplicationReport appReport = getApplicationReport(applicationId);
      YarnApplicationState state = appReport.getYarnApplicationState();
      if (!waitingStates.contains(state)) {
        if(failToSubmitStates.contains(state)) {
          throw new YarnException("Failed to submit " + applicationId + 
              " to YARN : " + appReport.getDiagnostics());
        }
        LOG.info("Submitted application " + applicationId);
        break;
      }

      long elapsedMillis = System.currentTimeMillis() - startTime;
      if (enforceAsyncAPITimeout() &&
          elapsedMillis >= asyncApiPollTimeoutMillis) {
        throw new YarnException("Timed out while waiting for application " +
            applicationId + " to be submitted successfully");
      }

      // Notify the client through the log every 10 poll, in case the client
      // is blocked here too long.
      if (++pollCount % 10 == 0) {
        LOG.info("Application submission is not finished, " +
            "submitted application " + applicationId +
            " is still in " + state);
      }
      try {
        Thread.sleep(submitPollIntervalMillis);
      } catch (InterruptedException ie) {
        LOG.error("Interrupted while waiting for application "
            + applicationId
            + " to be successfully submitted.");
      }
    } catch (ApplicationNotFoundException ex) {
      // FailOver or RM restart happens before RMStateStore saves
      // ApplicationState
      LOG.info("Re-submit application " + applicationId + "with the " +
          "same ApplicationSubmissionContext");
      rmClient.submitApplication(request);
    }
  }

  return applicationId;
}
 
開發者ID:naver,項目名稱:hadoop,代碼行數:77,代碼來源:YarnClientImpl.java

示例12: startApp

import org.apache.hadoop.yarn.api.records.ApplicationSubmissionContext; //導入方法依賴的package包/類
@Before
public void startApp() throws Exception {
  // submit new app
  ApplicationSubmissionContext appContext = 
      yarnClient.createApplication().getApplicationSubmissionContext();
  ApplicationId appId = appContext.getApplicationId();
  // set the application name
  appContext.setApplicationName("Test");
  // Set the priority for the application master
  Priority pri = Records.newRecord(Priority.class);
  pri.setPriority(0);
  appContext.setPriority(pri);
  // Set the queue to which this application is to be submitted in the RM
  appContext.setQueue("default");
  // Set up the container launch context for the application master
  ContainerLaunchContext amContainer =
      BuilderUtils.newContainerLaunchContext(
        Collections.<String, LocalResource> emptyMap(),
        new HashMap<String, String>(), Arrays.asList("sleep", "100"),
        new HashMap<String, ByteBuffer>(), null,
        new HashMap<ApplicationAccessType, String>());
  appContext.setAMContainerSpec(amContainer);
  appContext.setResource(Resource.newInstance(1024, 1, 1));
  // Create the request to send to the applications manager
  SubmitApplicationRequest appRequest = Records
      .newRecord(SubmitApplicationRequest.class);
  appRequest.setApplicationSubmissionContext(appContext);
  // Submit the application to the applications manager
  yarnClient.submitApplication(appContext);

  // wait for app to start
  RMAppAttempt appAttempt = null;
  while (true) {
    ApplicationReport appReport = yarnClient.getApplicationReport(appId);
    if (appReport.getYarnApplicationState() == YarnApplicationState.ACCEPTED) {
      attemptId = appReport.getCurrentApplicationAttemptId();
      appAttempt =
          yarnCluster.getResourceManager().getRMContext().getRMApps()
            .get(attemptId.getApplicationId()).getCurrentAppAttempt();
      while (true) {
        if (appAttempt.getAppAttemptState() == RMAppAttemptState.LAUNCHED) {
          break;
        }
      }
      break;
    }
  }
  // Just dig into the ResourceManager and get the AMRMToken just for the sake
  // of testing.
  UserGroupInformation.setLoginUser(UserGroupInformation
    .createRemoteUser(UserGroupInformation.getCurrentUser().getUserName()));

  // emulate RM setup of AMRM token in credentials by adding the token
  // *before* setting the token service
  UserGroupInformation.getCurrentUser().addToken(appAttempt.getAMRMToken());
  appAttempt.getAMRMToken().setService(ClientRMProxy.getAMRMTokenService(conf));
}
 
開發者ID:naver,項目名稱:hadoop,代碼行數:58,代碼來源:TestAMRMClient.java

示例13: run

import org.apache.hadoop.yarn.api.records.ApplicationSubmissionContext; //導入方法依賴的package包/類
public boolean run() throws IOException, YarnException {
  LOG.info("Starting Client");
  
  // Connect to ResourceManager
  rmClient.start();
  try {  
    // Create launch context for app master
    LOG.info("Setting up application submission context for ASM");
    ApplicationSubmissionContext appContext = rmClient.createApplication()
        .getApplicationSubmissionContext();
    ApplicationId appId = appContext.getApplicationId();

    // set the application name
    appContext.setApplicationName(appName);

    // Set the priority for the application master
    Priority pri = Records.newRecord(Priority.class);
    pri.setPriority(amPriority);
    appContext.setPriority(pri);

    // Set the queue to which this application is to be submitted in the RM
    appContext.setQueue(amQueue);

    // Set up the container launch context for the application master
    ContainerLaunchContext amContainer = Records
        .newRecord(ContainerLaunchContext.class);
    appContext.setAMContainerSpec(amContainer);

    // unmanaged AM
    appContext.setUnmanagedAM(true);
    LOG.info("Setting unmanaged AM");

    // Submit the application to the applications manager
    LOG.info("Submitting application to ASM");
    rmClient.submitApplication(appContext);

    ApplicationReport appReport =
        monitorApplication(appId, EnumSet.of(YarnApplicationState.ACCEPTED,
          YarnApplicationState.KILLED, YarnApplicationState.FAILED,
          YarnApplicationState.FINISHED));

    if (appReport.getYarnApplicationState() == YarnApplicationState.ACCEPTED) {
      // Monitor the application attempt to wait for launch state
      ApplicationAttemptReport attemptReport =
          monitorCurrentAppAttempt(appId,
            YarnApplicationAttemptState.LAUNCHED);
      ApplicationAttemptId attemptId =
          attemptReport.getApplicationAttemptId();
      LOG.info("Launching AM with application attempt id " + attemptId);
      // launch AM
      launchAM(attemptId);
      // Monitor the application for end state
      appReport =
          monitorApplication(appId, EnumSet.of(YarnApplicationState.KILLED,
            YarnApplicationState.FAILED, YarnApplicationState.FINISHED));
    }

    YarnApplicationState appState = appReport.getYarnApplicationState();
    FinalApplicationStatus appStatus = appReport.getFinalApplicationStatus();

    LOG.info("App ended with state: " + appReport.getYarnApplicationState()
        + " and status: " + appStatus);
    
    boolean success;
    if (YarnApplicationState.FINISHED == appState
        && FinalApplicationStatus.SUCCEEDED == appStatus) {
      LOG.info("Application has completed successfully.");
      success = true;
    } else {
      LOG.info("Application did finished unsuccessfully." + " YarnState="
          + appState.toString() + ", FinalStatus=" + appStatus.toString());
      success = false;
    }
    
    return success;
  } finally {
    rmClient.stop();
  }
}
 
開發者ID:naver,項目名稱:hadoop,代碼行數:80,代碼來源:UnmanagedAMLauncher.java

示例14: run

import org.apache.hadoop.yarn.api.records.ApplicationSubmissionContext; //導入方法依賴的package包/類
private void run(String[] args) throws Exception {
    if (args.length == 0) {
        System.out.println("Usage: [options] [commands..]");
        System.out.println("options: [-file filename] [-appcp appClasspath]");
        return;
    }
    this.initArgs(args);
    // Create yarnClient
    YarnClient yarnClient = YarnClient.createYarnClient();
    yarnClient.init(conf);
    yarnClient.start();

    // Create application via yarnClient
    YarnClientApplication app = yarnClient.createApplication();

    // Set up the container launch context for the application master
    ContainerLaunchContext amContainer = Records
            .newRecord(ContainerLaunchContext.class);
    ApplicationSubmissionContext appContext = app
            .getApplicationSubmissionContext();
    // Submit application
    ApplicationId appId = appContext.getApplicationId();

    //add ctrl+c signal handler
    CtrlCHandler handler = new CtrlCHandler(appId, yarnClient);
    Signal intSignal = new Signal("INT");
    Signal.handle(intSignal, handler);

    // setup security token
    amContainer.setTokens(this.setupTokens());
    // setup cache-files and environment variables
    amContainer.setLocalResources(this.setupCacheFiles(appId));
    amContainer.setEnvironment(this.getEnvironment());
    String cmd = Environment.JAVA_HOME.$$() + "/bin/java"
            + " -Xmx900m"
            + " org.apache.hadoop.yarn.dmlc.ApplicationMaster"
            + this.cacheFileArg + ' ' + this.appArgs + " 1>"
            + ApplicationConstants.LOG_DIR_EXPANSION_VAR + "/stdout"
            + " 2>" + ApplicationConstants.LOG_DIR_EXPANSION_VAR + "/stderr";

    LOG.debug(cmd);
    amContainer.setCommands(Collections.singletonList(cmd));

    // Set up resource type requirements for ApplicationMaster
    Resource capability = Records.newRecord(Resource.class);
    capability.setMemory(1024);
    capability.setVirtualCores(1);
    LOG.info("jobname=" + this.jobName + ",username=" + this.userName);

    appContext.setApplicationName(jobName + ":DMLC-YARN");
    appContext.setAMContainerSpec(amContainer);
    appContext.setResource(capability);
    appContext.setQueue(queue);
    //appContext.setUser(userName);
    LOG.info("Submitting application " + appId);
    yarnClient.submitApplication(appContext);

    ApplicationReport appReport = yarnClient.getApplicationReport(appId);
    YarnApplicationState appState = appReport.getYarnApplicationState();
    while (appState != YarnApplicationState.FINISHED
            && appState != YarnApplicationState.KILLED
            && appState != YarnApplicationState.FAILED) {
        Thread.sleep(100);
        appReport = yarnClient.getApplicationReport(appId);
        appState = appReport.getYarnApplicationState();
    }

    System.out.println("Application " + appId + " finished with"
            + " state " + appState + " at " + appReport.getFinishTime());
    if (!appReport.getFinalApplicationStatus().equals(
            FinalApplicationStatus.SUCCEEDED)) {
        System.err.println(appReport.getDiagnostics());
        System.out.println("Available queues:");
        for (QueueInfo q : yarnClient.getAllQueues()) {
          System.out.println(q.getQueueName());
        }

        yarnClient.killApplication(appId);
    }
}
 
開發者ID:Intel-bigdata,項目名稱:MXNetOnYARN,代碼行數:81,代碼來源:Client.java

示例15: submitApplication

import org.apache.hadoop.yarn.api.records.ApplicationSubmissionContext; //導入方法依賴的package包/類
@Override
public SubmitApplicationResponse submitApplication(
    SubmitApplicationRequest request) throws YarnException {
  ApplicationSubmissionContext submissionContext = request
      .getApplicationSubmissionContext();
  ApplicationId applicationId = submissionContext.getApplicationId();
  CallerContext callerContext = CallerContext.getCurrent();

  // ApplicationSubmissionContext needs to be validated for safety - only
  // those fields that are independent of the RM's configuration will be
  // checked here, those that are dependent on RM configuration are validated
  // in RMAppManager.

  String user = null;
  try {
    // Safety
    user = UserGroupInformation.getCurrentUser().getShortUserName();
  } catch (IOException ie) {
    LOG.warn("Unable to get the current user.", ie);
    RMAuditLogger.logFailure(user, AuditConstants.SUBMIT_APP_REQUEST,
        ie.getMessage(), "ClientRMService",
        "Exception in submitting application", applicationId, callerContext);
    throw RPCUtil.getRemoteException(ie);
  }

  // Check whether app has already been put into rmContext,
  // If it is, simply return the response
  if (rmContext.getRMApps().get(applicationId) != null) {
    LOG.info("This is an earlier submitted application: " + applicationId);
    return SubmitApplicationResponse.newInstance();
  }

  if (submissionContext.getQueue() == null) {
    submissionContext.setQueue(YarnConfiguration.DEFAULT_QUEUE_NAME);
  }
  if (submissionContext.getApplicationName() == null) {
    submissionContext.setApplicationName(
        YarnConfiguration.DEFAULT_APPLICATION_NAME);
  }
  if (submissionContext.getApplicationType() == null) {
    submissionContext
      .setApplicationType(YarnConfiguration.DEFAULT_APPLICATION_TYPE);
  } else {
    if (submissionContext.getApplicationType().length() > YarnConfiguration.APPLICATION_TYPE_LENGTH) {
      submissionContext.setApplicationType(submissionContext
        .getApplicationType().substring(0,
          YarnConfiguration.APPLICATION_TYPE_LENGTH));
    }
  }

  try {
    // call RMAppManager to submit application directly
    rmAppManager.submitApplication(submissionContext,
        System.currentTimeMillis(), user);

    LOG.info("Application with id " + applicationId.getId() + 
        " submitted by user " + user);
    RMAuditLogger.logSuccess(user, AuditConstants.SUBMIT_APP_REQUEST,
        "ClientRMService", applicationId, callerContext);
  } catch (YarnException e) {
    LOG.info("Exception in submitting application with id " +
        applicationId.getId(), e);
    RMAuditLogger.logFailure(user, AuditConstants.SUBMIT_APP_REQUEST,
        e.getMessage(), "ClientRMService",
        "Exception in submitting application", applicationId, callerContext);
    throw e;
  }

  SubmitApplicationResponse response = recordFactory
      .newRecordInstance(SubmitApplicationResponse.class);
  return response;
}
 
開發者ID:aliyun-beta,項目名稱:aliyun-oss-hadoop-fs,代碼行數:73,代碼來源:ClientRMService.java


注:本文中的org.apache.hadoop.yarn.api.records.ApplicationSubmissionContext.getApplicationId方法示例由純淨天空整理自Github/MSDocs等開源代碼及文檔管理平台,相關代碼片段篩選自各路編程大神貢獻的開源項目,源碼版權歸原作者所有,傳播和使用請參考對應項目的License;未經允許,請勿轉載。