當前位置: 首頁>>代碼示例>>Java>>正文

Java SpiderThreadLocal類代碼示例

本文整理匯總了Java中com.geccocrawler.gecco.spider.SpiderThreadLocal類的典型用法代碼示例。如果您正苦於以下問題：Java SpiderThreadLocal類的具體用法？Java SpiderThreadLocal怎麽用？Java SpiderThreadLocal使用的例子？那麽, 這裏精選的類代碼示例或許可以為您提供幫助。


SpiderThreadLocal類屬於com.geccocrawler.gecco.spider包，在下文中一共展示了SpiderThreadLocal類的14個代碼示例，這些例子默認根據受歡迎程度排序。您可以為喜歡或者感覺有用的代碼點讚，您的評價將有助於係統推薦出更棒的Java代碼示例。

示例1: log

import com.geccocrawler.gecco.spider.SpiderThreadLocal; //導入依賴的package包/類
public static void log(String message, Class<? extends SpiderBean> spiderBeanClass, Throwable cause) {
	boolean debug = SpiderThreadLocal.get().getEngine().isDebug();
	log.error(spiderBeanClass.getName() + " render error : " + message);
	if(debug && cause != null) {
		log.error(message, cause);
	}
}
開發者ID:xtuhcy，項目名稱:gecco，代碼行數:8，代碼來源:RenderException.java

示例2: log

import com.geccocrawler.gecco.spider.SpiderThreadLocal; //導入依賴的package包/類
public static void log(Field field, String message, Throwable cause) {
	boolean debug = SpiderThreadLocal.get().getEngine().isDebug();
	log.error(field.getName() + " render error : " + message);
	if(debug && cause != null) {
		log.error(message, cause);
	}
}
開發者ID:xtuhcy，項目名稱:gecco，代碼行數:8，代碼來源:FieldRenderException.java

示例3: 

import com.geccocrawler.gecco.spider.SpiderThreadLocal; //導入依賴的package包/類
public Elements $(String selector) {
	Elements elements = document.select(selector);
	if (SpiderThreadLocal.get().getEngine().isDebug()) {
		if (!selector.equalsIgnoreCase("script")) {
			// log.debug("["+selector+"]--->["+elements+"]");
			System.out.println("[" + selector + "]--->[" + elements + "]");
		}
	}
	return elements;
}
開發者ID:xtuhcy，項目名稱:gecco，代碼行數:11，代碼來源:HtmlParser.java

示例4: get

import com.geccocrawler.gecco.spider.SpiderThreadLocal; //導入依賴的package包/類
public static Proxys get() {
	Spider spider = SpiderThreadLocal.get();
	if(spider == null) {
		return null;
	}
	return spider.getEngine().getProxysLoader();
}
開發者ID:xtuhcy，項目名稱:gecco，代碼行數:8，代碼來源:ProxysContext.java

示例5: isEnableProxy

import com.geccocrawler.gecco.spider.SpiderThreadLocal; //導入依賴的package包/類
public static boolean isEnableProxy() {
	Spider spider = SpiderThreadLocal.get();
	if(spider == null) {
		return false;
	}
	return spider.getEngine().isProxy();
}
開發者ID:xtuhcy，項目名稱:gecco，代碼行數:8，代碼來源:ProxysContext.java

示例6: download

import com.geccocrawler.gecco.spider.SpiderThreadLocal; //導入依賴的package包/類
public HttpResponse download(HttpRequest request, int timeout) throws DownloadException {
	try {
		URL url = new URL(request.getUrl());
		WebRequest webRequest = new WebRequest(url);
		webRequest.setHttpMethod(HttpMethod.GET);
		if(request instanceof HttpPostRequest) {//post
			HttpPostRequest post = (HttpPostRequest)request;
			webRequest.setHttpMethod(HttpMethod.POST);
			List<NameValuePair> requestParameters = new ArrayList<NameValuePair>();
			for(Map.Entry<String, Object> entry : post.getFields().entrySet()) {
				NameValuePair nvp = new NameValuePair(entry.getKey(), entry.getValue().toString());
				requestParameters.add(nvp);
			}
			webRequest.setRequestParameters(requestParameters);	
		}
		//header
		boolean isMobile = SpiderThreadLocal.get().getEngine().isMobile();
		webRequest.setAdditionalHeader("User-Agent", UserAgent.getUserAgent(isMobile));
		webRequest.setAdditionalHeaders(request.getHeaders());
		//proxy
		HttpHost proxy = Proxys.getProxy();
		if(proxy != null) {
			webRequest.setProxyHost(proxy.getHostName());
			webRequest.setProxyPort(proxy.getPort());
		}
		//timeout
		this.webClient.getOptions().setTimeout(timeout);
		//request,response
		webClient.getPage(webRequest);
		HtmlPage page = webClient.getPage(request.getUrl());
		HttpResponse resp = new HttpResponse();
		WebResponse webResponse = page.getWebResponse();
		int status = webResponse.getStatusCode();
		resp.setStatus(status);
		if(status == 302 || status == 301) {
			String redirectUrl = webResponse.getResponseHeaderValue("Location");
			resp.setContent(UrlUtils.relative2Absolute(request.getUrl(), redirectUrl));
		} else if(status == 200) {
			String content = page.asXml();
			resp.setContent(content);
			resp.setRaw(webResponse.getContentAsStream());
			String contentType = webResponse.getContentType();
			resp.setContentType(contentType);
			String charset = getCharset(request.getCharset(), contentType);
			resp.setCharset(charset);
		} else {
			throw new DownloadException("ERROR : " + status);
		}
		return resp;
	} catch(Exception ex) {
		throw new DownloadException(ex);
	}
}
開發者ID:xtuhcy，項目名稱:gecco-htmlunit，代碼行數:54，代碼來源:HtmlUnitDownloder.java

示例7: getRender

import com.geccocrawler.gecco.spider.SpiderThreadLocal; //導入依賴的package包/類
public static Render getRender(RenderType type){
	return SpiderThreadLocal.get().getEngine().getSpiderBeanFactory().getRenderFactory().getRender(type);
}
開發者ID:xtuhcy，項目名稱:gecco，代碼行數:4，代碼來源:RenderContext.java

示例8: getRet

import com.geccocrawler.gecco.spider.SpiderThreadLocal; //導入依賴的package包/類
public static Object getRet(){
    return SpiderThreadLocal.get().getEngine().getRet();
}
開發者ID:xtuhcy，項目名稱:gecco，代碼行數:4，代碼來源:EngineRetUtil.java

示例9: setRet

import com.geccocrawler.gecco.spider.SpiderThreadLocal; //導入依賴的package包/類
public static void setRet(Object o){
    SpiderThreadLocal.get().getEngine().setRet(o);
}
開發者ID:xtuhcy，項目名稱:gecco，代碼行數:4，代碼來源:EngineRetUtil.java

示例10: HttpClientDownloader

import com.geccocrawler.gecco.spider.SpiderThreadLocal; //導入依賴的package包/類
public HttpClientDownloader() {
	
	cookieContext = HttpClientContext.create();
	cookieContext.setCookieStore(new BasicCookieStore());
	
	Registry<ConnectionSocketFactory> socketFactoryRegistry = null;
	try {
		//構造一個信任所有ssl證書的httpclient
		SSLContext sslContext = SSLContexts.custom().loadTrustMaterial(null, new TrustStrategy() {
			@Override
			public boolean isTrusted(X509Certificate[] chain, String authType) throws CertificateException {
				return true;
			}
		}).build();
		SSLConnectionSocketFactory sslsf = new SSLConnectionSocketFactory(sslContext);
		socketFactoryRegistry = RegistryBuilder.<ConnectionSocketFactory>create()
		           .register("http", PlainConnectionSocketFactory.getSocketFactory())  
		           .register("https", sslsf)  
		           .build();
	} catch(Exception ex) {
		socketFactoryRegistry = RegistryBuilder.<ConnectionSocketFactory>create()
           .register("http", PlainConnectionSocketFactory.getSocketFactory())
           .register("https", SSLConnectionSocketFactory.getSocketFactory())
           .build();
	}
	RequestConfig clientConfig = RequestConfig.custom().setRedirectsEnabled(false).build();
	PoolingHttpClientConnectionManager syncConnectionManager = new PoolingHttpClientConnectionManager(socketFactoryRegistry);
	syncConnectionManager.setMaxTotal(1000);
	syncConnectionManager.setDefaultMaxPerRoute(50);
	httpClient = HttpClientBuilder.create()
			.setDefaultRequestConfig(clientConfig)
			.setConnectionManager(syncConnectionManager)
			.setRetryHandler(new HttpRequestRetryHandler() {
				@Override
				public boolean retryRequest(IOException exception, int executionCount, HttpContext context) {
					int retryCount = SpiderThreadLocal.get().getEngine().getRetry();
					boolean retry = (executionCount <= retryCount);
					if(log.isDebugEnabled() && retry) {
						log.debug("retry : " + executionCount);
					}
					return retry;
				}
			}).build();
}
開發者ID:xtuhcy，項目名稱:gecco，代碼行數:45，代碼來源:HttpClientDownloader.java

示例11: download

import com.geccocrawler.gecco.spider.SpiderThreadLocal; //導入依賴的package包/類
public static HttpResponse download(HttpRequest request) throws DownloadException {
	SpiderBeanContext context = SpiderThreadLocal.get().getSpiderBeanContext();
	return context.getDownloader().download(request, context.getTimeout());
}
開發者ID:xtuhcy，項目名稱:gecco，代碼行數:5，代碼來源:DownloaderContext.java

示例12: defaultDownload

import com.geccocrawler.gecco.spider.SpiderThreadLocal; //導入依賴的package包/類
public static HttpResponse defaultDownload(HttpRequest request) throws DownloadException {
	SpiderBeanContext context = SpiderThreadLocal.get().getSpiderBeanContext();
	Downloader downloader = SpiderThreadLocal.get().getEngine().getSpiderBeanFactory().getDownloaderFactory().defaultDownloader();
	return downloader.download(request, context.getTimeout());
}
開發者ID:xtuhcy，項目名稱:gecco，代碼行數:6，代碼來源:DownloaderContext.java

示例13: into

import com.geccocrawler.gecco.spider.SpiderThreadLocal; //導入依賴的package包/類
public static void into(HttpRequest request) {
	SpiderThreadLocal.get().getSpiderScheduler().into(request);
}
開發者ID:xtuhcy，項目名稱:gecco，代碼行數:4，代碼來源:DeriveSchedulerContext.java

示例14: into

import com.geccocrawler.gecco.spider.SpiderThreadLocal; //導入依賴的package包/類
public static void into(HttpRequest request) {
	SpiderThreadLocal.get().getEngine().getScheduler().into(request);
}
開發者ID:xtuhcy，項目名稱:gecco，代碼行數:4，代碼來源:StartSchedulerContext.java


注：本文中的com.geccocrawler.gecco.spider.SpiderThreadLocal類示例由純淨天空整理自Github/MSDocs等開源代碼及文檔管理平台，相關代碼片段篩選自各路編程大神貢獻的開源項目，源碼版權歸原作者所有，傳播和使用請參考對應項目的License；未經允許，請勿轉載。