We Keep Coding

How do I exstract data from multiple websites using restful API and Spring? [closed]

Closed. This question needs to be more focused. It is not currently accepting answers.
Want to improve this question? Update the question so it focuses on one problem only by editing this post.
Closed 1 year ago.
Improve this question
I got a task at school in which I have to do the following:
Implement the RESTful endpoint API, which simultaneously makes calls
to the following websites:
https://pizzerijalimbo.si/meni/
https://pizzerijalimbo.si/kontakt/
https://pizzerijalimbo.si/my-account/
https://pizzerijalimbo.si/o-nas/
The input for the endpoint is ‘integer’, which represents the number
of simultaneous calls to the above web pages (min 1 represents all
consecutive calls, max 4 represents all simultaneous calls).
Extracts a short title text from each page and saves this text in a
common global structure (array, folder (). The program should also
count successful calls. Finally, the service should list the number of
successful calls, the number of failed calls and the saved address
texts from all web pages.
With some help I managed to do something, but I still need help with data exstraction using Jsoup or any other method.
Here is the code that I have:
import java.util.Arrays;
import java.util.List;
import java.io.IOException;
import java.net.URL;
import java.util.Scanner;
#RestController
public class APIcontroller {
#Autowired
private RestTemplate restTemplate;
List<String> websites = Arrays.asList("https://pizzerijalimbo.si/meni/",
"https://pizzerijalimbo.si/kontakt/",
"https://pizzerijalimbo.si/my-account/",
"https://pizzerijalimbo.si/o-nas/");
#GetMapping("/podatki")
public List<Object> getData(#RequestParam(required = true) int numberOfWebsites) {
List<String> websitesToScrape = websites.subList(0, numberOfWebsites);
for (String website : websitesToScrape) {
Document doc = Jsoup.connect("https://pizzerijalimbo.si/meni/").get();
log(doc.title());
Elements newsHeadlines = doc.select("#mp-itn b a");
for (Element headline : newsHeadlines) {
log("%s\n\t%s",
headline.attr("title"), headline.absUrl("href"));
}
}
}
}
I also need to do it parallel, so the calls to a secific website go on at the same time.
But the main problem now is with the log funcion which does not work properly.
What I have tried:
I tried to solve the problem using Jsoup library, but I dont seem to
undersand it well, so I got an error in the for loop which says that
the method log is undefined. I also need to do a try catch to count possible failed calls and count the calls that are successfull as you can see in the task description.

WebScrapperController.java
package com.stackovertwo.stackovertwo;
import java.io.IOException;
import java.io.StringReader;
import java.util.ArrayList;
import java.util.Arrays;
import java.util.List;
import java.util.concurrent.CompletableFuture;
import java.util.concurrent.ExecutionException;
import java.util.regex.Pattern;
import java.util.stream.Collectors;
import javax.xml.parsers.DocumentBuilderFactory;
import javax.xml.parsers.ParserConfigurationException;
import org.jsoup.Jsoup;
import org.jsoup.select.Elements;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.http.HttpEntity;
import org.springframework.http.HttpHeaders;
import org.springframework.http.HttpMethod;
import org.springframework.http.HttpStatus;
import org.springframework.http.ResponseEntity;
import org.springframework.web.bind.annotation.GetMapping;
import org.springframework.web.bind.annotation.RequestParam;
import org.springframework.web.bind.annotation.RestController;
import org.springframework.web.client.RestTemplate;
//import org.w3c.dom.Document;
//import org.w3c.dom.DocumentFragment;
import org.jsoup.nodes.Document;
import org.w3c.dom.Node;
import org.xml.sax.InputSource;
import org.xml.sax.SAXException;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
#RestController
public class WebScrapperController {
#GetMapping("/")
public String index() {
return "Greetings from Spring Boot!";
}
// #Autowired
// private RestTemplate restTemplate;
#Autowired
WebScrapperService webScrapperService;
List<String> websites = Arrays.asList("https://pizzerijalimbo.si/meni/",
"https://pizzerijalimbo.si/kontakt/",
"https://pizzerijalimbo.si/my-account/",
"https://pizzerijalimbo.si/o-nas/");
#GetMapping("/podatki")
public ResponseEntity<Object> getData(#RequestParam(required = true) int numberOfWebsites) throws InterruptedException, ExecutionException {
List<SiteResponse> webSitesToScrape = new ArrayList<>();
// List<String> websitesToScrape = websites.subList(0, numberOfWebsites);
List<SiteResponse> responseResults = new ArrayList<SiteResponse>();
CompletableFuture<SiteResponse> futureData1 = webScrapperService.getWebScrappedContent(websites.get(0));
CompletableFuture<SiteResponse> futureData2 = webScrapperService.getWebScrappedContent(websites.get(1));
//CompletableFuture.allOf(futureData1, futureData2).join();
webSitesToScrape.add(futureData1.get());
webSitesToScrape.add(futureData2.get());
List<SiteResponse> result = webSitesToScrape.stream().collect(Collectors.toList());
return ResponseEntity.ok().body(result);
}
}
WebScrapperService.java
package com.stackovertwo.stackovertwo;
import org.jsoup.Jsoup;
import org.jsoup.nodes.Document;
import org.jsoup.select.Elements;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.http.HttpEntity;
import org.springframework.http.HttpHeaders;
import org.springframework.http.HttpMethod;
import org.springframework.http.ResponseEntity;
import org.springframework.scheduling.annotation.Async;
import org.springframework.stereotype.Service;
import org.springframework.web.client.RestTemplate;
import java.util.concurrent.CompletableFuture;
#Service
public class WebScrapperService {
#Autowired
private RestTemplate restTemplate;
Logger logger = LoggerFactory.getLogger(WebScrapperService.class);
#Async
public CompletableFuture<SiteResponse> getWebScrappedContent(String webSiteURL)
//throws InterruptedException
{
logger.info("Starting: getWebScrappedContent for webSiteURL {} with thread {}", webSiteURL, Thread.currentThread().getName());
HttpEntity<String> response = restTemplate.exchange(webSiteURL,
HttpMethod.GET, null, String.class);
//Thread.sleep(1000);
SiteResponse webSiteSummary = null ;
String resultString = response.getBody();
HttpHeaders headers = response.getHeaders();
int statusCode = ((ResponseEntity<String>) response).getStatusCode().value();
System.out.println(statusCode);
System.out.println("HEADERS"+headers);
try
{
Document doc = (Document) Jsoup.parse(resultString);
Elements header = doc.select(".elementor-inner h2.elementor-heading-title.elementor-size-default");
System.out.println(header.get(0).html());
// Return the fragment.
webSiteSummary = new SiteResponse(statusCode, header.get(0).html());
}
catch(Exception e) {
System.out.println("Exception "+e.getMessage());
}
logger.info("Complete: getWebScrappedContent for webSiteURL {} with thread {}", webSiteURL, Thread.currentThread().getName());
return CompletableFuture.completedFuture(webSiteSummary);
}
}
SpringBootApp.java
package com.stackovertwo.stackovertwo;
import org.springframework.boot.SpringApplication;
import org.springframework.boot.autoconfigure.SpringBootApplication;
import org.springframework.context.annotation.Bean;
import org.springframework.http.client.HttpComponentsClientHttpRequestFactory;
import org.springframework.web.client.RestTemplate;
import java.security.KeyManagementException;
import java.security.KeyStoreException;
import java.security.NoSuchAlgorithmException;
import java.security.cert.X509Certificate;
//import javax.net.ssl.HostnameVerifier;
//import javax.net.ssl.HttpsURLConnection;
import javax.net.ssl.SSLContext;
//import javax.net.ssl.SSLSession;
//import javax.net.ssl.TrustManager;
//import javax.net.ssl.X509TrustManager;
//import javax.security.cert.X509Certificate;
import org.apache.http.conn.ssl.TrustStrategy;
import org.apache.http.impl.client.*;
import org.apache.http.conn.ssl.*;
#SpringBootApplication
public class SpringBootApp
{
public static void main(String[] args)
{
SpringApplication.run(SpringBootApp.class, args);
}
#Bean
public RestTemplate restTemplate() throws KeyManagementException, NoSuchAlgorithmException, KeyStoreException {
TrustStrategy acceptingTrustStrategy = (X509Certificate[] chain, String authType) -> true;
SSLContext sslContext = org.apache.http.ssl.SSLContexts.custom()
.loadTrustMaterial(null, acceptingTrustStrategy)
.build();
SSLConnectionSocketFactory csf = new SSLConnectionSocketFactory(sslContext);
CloseableHttpClient httpClient = HttpClients.custom()
.setSSLSocketFactory(csf)
.build();
HttpComponentsClientHttpRequestFactory requestFactory =
new HttpComponentsClientHttpRequestFactory();
requestFactory.setHttpClient(httpClient);
//return new RestTemplate();
RestTemplate restTemplate = new RestTemplate(requestFactory);
return restTemplate;
}
}
Note: I disabled SSL verification while calling the webulr in resttemplate, but its not recommendd inproduction (For assignment its ok). But you need to import the keys via java keystore in case production : https://myshittycode.com/2015/12/17/java-https-unable-to-find-valid-certification-path-to-requested-target-2/

Having trouble in verifying the PACT from provider side using java junit5 maven spring-boot

I have to do PACT verification in java + spring-boot + maven.
I am currently running this tests with junit5.
My pom.xml looks like
<dependency>
<groupId>au.com.dius</groupId>
<artifactId>pact-jvm-provider-junit5</artifactId>
<version>4.0.10</version>
<scope>test</scope>
<exclusions>
<exclusion>
<groupId>com.google.code.gson</groupId>
<artifactId>gson</artifactId>
</exclusion>
</exclusions>
</dependency>
<dependency>
<groupId>au.com.dius</groupId>
<artifactId>pact-jvm-consumer-junit5</artifactId>
<version>4.0.10</version>
</dependency>
<plugin>
<groupId>au.com.dius.pact.provider</groupId>
<artifactId>maven</artifactId>
<version>4.1.11</version>
<configuration>
<pactDirectory>${basedir}/target/pacts</pactDirectory>
<pactBrokerUrl><BROKERURL></pactBrokerUrl>
<projectVersion><PROJECTVERSION></projectVersion>
<trimSnapshot>true</trimSnapshot>
</configuration>
</plugin>
Consumer side code is runnning perfectly and published on pact broker too, here is the snippet of it.
package com.contract;
import static org.hamcrest.CoreMatchers.is;
import static org.hamcrest.CoreMatchers.notNullValue;
import static org.hamcrest.MatcherAssert.assertThat;
import static org.junit.Assert.assertEquals;
import java.io.IOException;
import java.util.HashMap;
import java.util.List;
import java.util.Map;
import org.apache.http.HttpResponse;
import org.apache.http.client.ClientProtocolException;
import org.apache.http.client.fluent.Request;
import org.apache.http.entity.StringEntity;
import org.junit.jupiter.api.BeforeEach;
import org.junit.jupiter.api.Test;
import org.junit.jupiter.api.extension.ExtendWith;
import org.testng.Assert;
import com.fasterxml.jackson.databind.DeserializationFeature;
import com.fasterxml.jackson.databind.ObjectMapper;
import au.com.dius.pact.consumer.MockServer;
import au.com.dius.pact.consumer.dsl.PactDslWithProvider;
import au.com.dius.pact.consumer.junit5.PactConsumerTestExt;
import au.com.dius.pact.consumer.junit5.PactTestFor;
import au.com.dius.pact.core.model.RequestResponsePact;
import au.com.dius.pact.core.model.annotations.Pact;
#ExtendWith(PactConsumerTestExt.class)
class ContractTest {
public static final String jwt_token = "Bearer eyJraWQiOiIyZjFiNzlmMS0xMDQ2LTQ2NGYtYjM5YS0xOGY4MDg5ZGMyMzIiLCJ0eXAiOiJhdCtqd3QiLCJhbGciOiJSUzI1NiJ9.eyJhdWQiOiJodHRwczovL3RlbmFudDE1LWFkZW1hdHJpeHN0YWNrNC10cmlhbC5kc21sYWIuYm1jLmNvbSIsImNsaWVudF9pZCI6IjVmNDU4ZjBlLWZmMzAtNDQ2ZC04ZTQwLWYzNjBlNjgyZjYyNSIsInRlbmFudCI6InN0YWNrNHRlbmFudDYiLCJqdGkiOiI4NDc3MDBhNC1hYWY0LTQ2NWMtYmNhYi05MDM0MzYxOTY0MDIiLCJzdWIiOiJEZW1vIiwiaXNzIjoiaHR0cHM6Ly9hZGVpdHNtMi1yc3NvLmRzbWxhYi5ibWMuY29tL3Jzc28iLCJpYXQiOjE2MjY3Njg0MjQsImV4cCI6MTYyNjc2OTMyNH0.EoXpqiyo5nvdPeYHfqq0sa15dQKYayD60UEboPVJojuYBKmvvj2yU03e61wy9Fkq2pZdoTNTifAeLDiG0dYKdlYOI5YTx_K6HGMav6ofYeOwIUj4OBVu7NyWxXVQz-3aXoIyu-MifUtvs6oQwf2YZmOEVtbPuCBxa9C9yA4i72g1TD31Rba1-e5cGG5ipiIE7UaunJ2K-mkt-BL2kzmu6OHdIP6vly7iTzxfOccdrjXEmedgX2hpnPcL_2os5wHCLwdHJwuYLPlgqDbSLHgpXdGL43Jg4ASBbFHg3h30y1yXYJazlgOwvVeBoOVcQYnXBh7wHTMik7zVMAo1VL8N_Q";
public static final String API_V1_VIEW_ALGORITHM = "/aif/api/v1.0/algorithm/b82d4617-c39f-448b-bdf8-3fd52e3250ba";
public static final String BODY_GET_VIEW_ALGO_V1 = "{\"tenantId\":\"6881408\",\"id\":\"5bc098db-7acb-4361-86a9-3c439b477142\",\"name\":\"New Job\",\"description\":\"New Job\",\"creationTime\":\"1625054173273\",\"template\":\"\",\"modifiedTime\":\"1625054173273\",\"owner\":\"ppan\",\"enable\":\"true\",\"executionMode\":\"onDemand\",\"definition\":\"\"}";
private final static Map<String, String> headers;
static {
headers = new HashMap<>();
headers.put("Content-Type", "application/json");
headers.put("Authorization", jwt_token);
headers.put("authtype", "rsso-jwt");
}
#BeforeEach
public void setUp(MockServer mockServer) {
assertThat(mockServer, is(notNullValue()));
}
/* ----------------------------------Pact methods ----------------------------------------------------*/
#Pact(provider = <provider name> , consumer = <consumer name>)
public RequestResponsePact createPactViewAlgoV1(PactDslWithProvider builder) {
return builder
.given("algorithm id")
.uponReceiving("a request with details of Algorithm")
.path(API_V1_VIEW_ALGORITHM)
.method("GET")
.headers(headers)
.willRespondWith()
.status(200)
.body(BODY_GET_VIEW_ALGO_V1)
.toPact();
}
/* ----------------------------------Test methods ----------------------------------------------------*/
#Test
#PactTestFor(pactMethod = "createPactViewAlgoV1")
public void testTokenRequest(MockServer mockServer) throws ClientProtocolException, IOException {
HttpResponse httpResponse = Request.Get(mockServer.getUrl() + API_V1_VIEW_ALGORITHM)
.addHeader("Content-Type", headers.get("Content-Type"))
.addHeader("Authorization",headers.get("Authorization"))
.addHeader("authtype", headers.get("authtype"))
.execute()
.returnResponse();
assertEquals(httpResponse.getStatusLine().getStatusCode(), 200);
ObjectMapper objectMapper = new ObjectMapper();
objectMapper.enable(DeserializationFeature.ACCEPT_SINGLE_VALUE_AS_ARRAY);
AIFAlgorithmModelNew actualResult = objectMapper.readValue(httpResponse.getEntity().getContent(), AIFAlgorithmModelNew.class);
assertEquals(actualResult.getTenantId().toString(), "6881408");
assertEquals(actualResult.getExecutionMode().toString(), "onDemand");
assertEquals(actualResult.getEnable().toString(), "true");
assertEquals(actualResult.getName().toString(), "New Job");
}
}
I have written my Pact provider test class with standard pact format.
I am currently trying to run this on port: 8091.
Tried with port 9362 as well.
looks like this
package com.aif.api.contract;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.beans.factory.annotation.Value;
import org.springframework.boot.test.autoconfigure.web.servlet.AutoConfigureMockMvc;
import org.springframework.boot.test.context.SpringBootTest;
import org.springframework.boot.test.mock.mockito.MockBean;
import org.springframework.boot.test.mock.mockito.SpyBean;
import org.springframework.context.ApplicationContext;
import org.springframework.test.context.TestPropertySource;
import org.springframework.test.context.junit.jupiter.SpringExtension;
<Code libraries>
import au.com.dius.pact.provider.junit.Provider;
import au.com.dius.pact.provider.junit.State;
import au.com.dius.pact.provider.junit.loader.PactBroker;
import au.com.dius.pact.provider.junit5.HttpTestTarget;
import au.com.dius.pact.provider.junit5.HttpsTestTarget;
import au.com.dius.pact.provider.junit5.PactVerificationContext;
import au.com.dius.pact.provider.junit5.PactVerificationInvocationContextProvider;
import io.grpc.ManagedChannel;
import io.grpc.ManagedChannelBuilder;
import io.restassured.RestAssured;
import io.restassured.http.Cookie;
import io.restassured.config.SSLConfig;
import static org.mockito.Mockito.when;
import java.nio.ByteBuffer;
import java.util.Arrays;
import java.util.Base64;
import java.util.UUID;
import javax.ws.rs.core.Response;
import org.apache.http.HttpRequest;
import org.junit.AfterClass;
import org.junit.Before;
import org.junit.BeforeClass;
import org.junit.Test;
import org.junit.jupiter.api.BeforeEach;
import org.junit.jupiter.api.TestTemplate;
import org.junit.jupiter.api.extension.ExtendWith;
import org.mockito.Mockito;
import org.mockito.MockitoAnnotations;
#ExtendWith(SpringExtension.class)
#SpringBootTest(webEnvironment = SpringBootTest.WebEnvironment.DEFINED_PORT, classes = main.class)
#AutoConfigureMockMvc
#TestPropertySource(locations = "classpath:application-contract-test.properties")
#Provider("Provider")
#PactBroker(
host="${PACTBROKER_HOST:PACTBroker URL}",
port="${PACTBROKER_PORT:9292}"
)
public class BasicTest {
static {
System.setProperty("javax.net.ssl.keyStore", "./conf/server_key.p12");
System.setProperty("javax.net.ssl.keyStorePassword", "Y2hhbmdlaXQ=");
}
#Autowired
private ApplicationContext context;
#MockBean
Algor
#MockBean
AlgorithmResource algorithmResource;
#Value("${server.port}")
private int serverPort;
#Value("${pact.provider.version}")
private String version;
#Value("${pact.verifier.publishResults}")
private String publishResults;
#BeforeEach
void setupTestTarget(PactVerificationContext context) {
System.out.println("Inside setupTestTarget");
MockitoAnnotations.initMocks(this);
context.setTarget(new HttpTestTarget("localhost", serverPort, "/"));
System.setProperty("pact.provider.version", version);
System.setProperty("pact.verifier.publishResults", publishResults);
}
#TestTemplate
#ExtendWith(PactVerificationInvocationContextProvider.class)
void pactVerificationTestTemplate(PactVerificationContext context, HttpRequest request) {
context.verifyInteraction();
}
/* consumer = <Consumer Name> */
#State("algorithm id")
public void getAlgorithmById() throws Exception {
System.out.println("Inside getAlgorithmById");
AlgorithmResponseTOFull responseTO = new AlgorithmResponseTOFull();
responseTO.setTenantId("6881408");
responseTO.setId("5bc098db-7acb-4361-86a9-3c439b477142");
responseTO.setName("New Job");
responseTO.setDefinition("");
responseTO.setDescription("New Job");
responseTO.setCreationTime("1625054173273");
responseTO.setModifiedTime("1625054173273");
responseTO.setOwner("ppans");
responseTO.setEnable("true");
responseTO.setExecutionMode("onDemand");
responseTO.setTemplate(getAlgorithmTemplateResponseTO());
Response response = Response.ok(Util.buildResponse(responseTO, Constant.STATUS_MSG_SUCCESS)).build();
when(algorithmResource.getById(Mockito.any())).thenReturn(response);
}
}
When I run this, I get error like
Verifying a pact between <consumer name> and <provider name>
[Using Pact Broker PACTBROKERURL:9292]
Given algorithm id
a request with details of Algorithm
Inside pactVerificationTestTemplate
returns a response which
has status code 200 (FAILED)
has a matching body (FAILED)
Failures:
0) Verifying a pact between <consumer name> and <provider name> - a request with details of Algorithm returns a response which has statusResult code 200
expected status of 200 but was 401
1) Verifying a pact between <consumer name> and <provider name> - a request with details of Algorithm returns a response which has a matching body
Expected a response type of 'application/json' but the actual type was 'text/html'
Please help me to resolve this, as I am getting above error and not able to resolve it

In the consumer where the test is written , it is expecting a response of 200 and response type of "application/json".
If possible, please share the consumer test also that you have written.
Please check that the pact which is getting generated at the pact broker url, and also what is the response and response type it is expecting.

A 401 indicates the request contained invalid or no credentials.
Given that your consumer test has a JWT in it, my guess is that it's expired by the time the provider test runs.
See https://docs.pact.io/provider/handling_auth/ for strategies in dealing with this and the workshops here to see how to out them into action: https://docs.pact.io/implementation_guides/workshops/

java.lang.NoClassDefFoundError: org/apache/http/ssl/TrustStrategy, though it is present in classpath

I am running an application on tomcat server. I am getting following error while callling a specific function :
java.lang.NoClassDefFoundError: org/apache/http/ssl/TrustStrategy
My class is :
import java.io.ByteArrayOutputStream;
import java.io.Serializable;
import java.util.Map;
import javax.net.ssl.SSLContext;
import org.apache.commons.lang.StringUtils;
import org.apache.http.HttpEntity;
import org.apache.http.HttpHost;
import org.apache.http.HttpResponse;
import org.apache.http.client.HttpClient;
import org.apache.http.client.config.RequestConfig;
import org.apache.http.client.methods.HttpPost;
/*import org.apache.http.client.config.AuthSchemes;
import org.apache.http.client.config.CookieSpecs;
import org.apache.http.client.config.RequestConfig;*/
//import org.apache.http.client.methods.HttpPost;
import org.apache.http.conn.ssl.NoopHostnameVerifier;
import org.apache.http.conn.ssl.SSLConnectionSocketFactory;
import org.apache.http.conn.ssl.SSLContexts;
import org.apache.http.conn.ssl.TrustSelfSignedStrategy;
import org.apache.http.entity.StringEntity;
import org.apache.http.impl.client.HttpClients;
import org.apache.http.message.BasicHeader;
import org.apache.http.protocol.HTTP;
import org.apache.log4j.Logger;
import org.performics.air.business.common.data.HttpParam;
public class SendAndReceiveUtil implements Serializable {
//FIXME: move to suitable package
/**
*
*/
private static final long serialVersionUID = 2649891233958197253L;
private static Logger LOG = Logger.getLogger(SendAndReceiveUtil.class);
public static String httpPostWithTLS(String request,String url,Map<String,String> headerParameterMap){
String responseStr = null;
try{
// FIXME: need to handle supplier timeout and gzip
String contentType="";
String soapAction="";
boolean zipForRequest=false;
boolean acceptEncoding = false;
boolean zipForResponse=false;
if (headerParameterMap!=null){
contentType=headerParameterMap.get(HttpParam.CONTENTTYPE.toString())!=null?headerParameterMap.get(HttpParam.CONTENTTYPE.toString()):"";
zipForRequest=(headerParameterMap.containsKey(HttpParam.ZIPFORREQUEST.toString()))? new Boolean(headerParameterMap.get(HttpParam.ZIPFORREQUEST.toString())):false;
acceptEncoding=(headerParameterMap.containsKey(HttpParam.ACCEPT_ENCODING.toString()))? new Boolean(headerParameterMap.get(HttpParam.ACCEPT_ENCODING.toString())):false;
zipForResponse=(headerParameterMap.containsKey(HttpParam.ZIPFORRESPONSE.toString()))? new Boolean(headerParameterMap.get(HttpParam.ZIPFORRESPONSE.toString())):false;
soapAction=headerParameterMap.get(HttpParam.SOAPACTION.toString())!=null?headerParameterMap.get(HttpParam.SOAPACTION.toString()):"";
}
SSLContext sslcontext = SSLContexts.custom().loadTrustMaterial(null,new TrustSelfSignedStrategy()).build();
// Allow TLSv1.2 protocol only, use NoopHostnameVerifier to trust self-singed cert
SSLConnectionSocketFactory sslsf = new SSLConnectionSocketFactory(sslcontext,new String[] { "TLSv1.2" }, null, new NoopHostnameVerifier());
//do not set connection manager
HttpClient httpclient = HttpClients.custom().setSSLSocketFactory(sslsf).build();
HttpPost httpPost = new HttpPost(url);
httpPost.setHeader("SOAPAction", soapAction);
StringEntity mEntity = new StringEntity(request, "UTF-8");
if(StringUtils.isNotBlank(contentType)){
mEntity.setContentType(contentType);
mEntity.setContentEncoding(new BasicHeader(HTTP.CONTENT_TYPE,contentType));
}else{
mEntity.setContentType("text/xml;charset=UTF-8");
mEntity.setContentEncoding(new BasicHeader(HTTP.CONTENT_ENCODING,"gzip"));
}
httpPost.addHeader("Content-Encoding", "gzip" );
httpPost.addHeader("Accept-Encoding", "gzip,deflate" );
if(null!=headerParameterMap.get("Cookie")){
httpPost.addHeader("Cookie", headerParameterMap.get("Cookie"));
}
httpPost.setEntity(mEntity);
HttpResponse response = httpclient.execute(httpPost);
HttpEntity et=response.getEntity();
ByteArrayOutputStream os = new ByteArrayOutputStream();
et.writeTo(os);
responseStr = new String(os.toByteArray());
}catch(Exception e){
LOG.error(e.getMessage(), e);
}
return responseStr;
}
}
Error comes on calling httpPostWithTLS() function. I searched about it on net and found that class was available at compile but is not available at run time, but i am unable to correct it.
I am using following http jars:
commons-httpclient-3.1.jar
httpclient-4.5.3.jar
httpcore-4.4.6.jar
httpclient-cache-4.5.3.jar
httpclient-win-4.5.3.jar
httpmime-4.5.3.jar

Try to add the httpcore-4.4.6.jar and httpclient-cache-4.5.3.jar to server library and check.
I do face the similar issue after adding these jar file, my problem is resolved.
Thanks,

How to execute rest api from java and capture the response?

Executing this url on my browser: http://localhost:3161/devices/simulator/stop
I don't need login for it. It returns this rest api xml:
<response>
<type>response</type>
<ts>1463749194000</ts>
<status>OK</status>
<msg-version>2.3.0</msg-version>
<op>stop</op>
<data/>
</response>
How can I execute this from JAVA and then capture the xml response?

As other mentioned in this post, it is generic thing, you would be able to find it online already..
I know there are clients to call the REST services from Java. Two of them are listed for your case.
case -1 : if you are using Jersey REST API. Here to capture XML , you can go with your own way,for example use JAXB and XML elements to Java Bean properties.
import java.io.IOException;
import javax.ws.rs.core.MediaType;
import javax.ws.rs.core.UriBuilder;
import org.apache.http.client.ClientProtocolException;
import com.sun.jersey.api.client.Client;
import com.sun.jersey.api.client.WebResource;
import com.sun.jersey.api.client.config.ClientConfig;
import com.sun.jersey.api.client.config.DefaultClientConfig;
public class Test {
public static void main(String[] args) throws ClientProtocolException, IOException {
ClientConfig config = new DefaultClientConfig();
Client client = Client.create(config);
WebResource service = client.resource(UriBuilder.fromUri('http://localhost:3161/devices/simulator/stop').build());
// getting XML data
System.out.println(service. path('restPath').path('resourcePath').accept(MediaType.APPLICATION_JSON).get(String.class));
// getting JSON data
System.out.println(service. path('restPath').path('resourcePath').accept(MediaType.APPLICATION_XML).get(String.class));
}
}
Case 2: using HTTP method, it is simple method but parse XML instead of printing it here
import java.io.BufferedReader;
import java.io.IOException;
import java.io.InputStreamReader;
import org.apache.http.HttpResponse;
import org.apache.http.client.ClientProtocolException;
import org.apache.http.client.HttpClient;
import org.apache.http.client.methods.HttpGet;
import org.apache.http.impl.client.DefaultHttpClient;
public class Test {
public static void main(String[] args) throws ClientProtocolException, IOException {
HttpClient client = new DefaultHttpClient();
HttpGet request = new HttpGet('http://localhost:3161/devices/simulator/stop');
HttpResponse response = client.execute(request);
BufferedReader rd = new BufferedReader (new InputStreamReader(response.getEntity().getContent()));
String line = '';
while ((line = rd.readLine()) != null) {
System.out.println(line);
}
}
}

AppEngine: Could not verify SSL certificate for URL

I am attempting to connect to Intuit Quickbooks with OAuth, using their JAVA API V3, hosted on AppEngine (SDK 1.8.4). To get started, I'm just running their sample code from their instructions here: http://ippdocs.intuit.com/0025_QuickBooksAPI/0055_DevKits/0201_IPP_Java_DevKit_3.0
It's working well on TomCat, but when I adapt the code to work on AppEngine, I get this error:
OauthHelper.java Extract, the exception is raised at the last line:
import java.io.BufferedReader;
import java.io.IOException;
import java.io.InputStream;
import java.io.InputStreamReader;
import java.io.UnsupportedEncodingException;
import java.net.HttpURLConnection;
import java.net.MalformedURLException;
import java.net.URL;
import java.net.URLEncoder;
import java.util.HashMap;
import java.util.Map;
import oauth.signpost.OAuthConsumer;
import oauth.signpost.OAuthProvider;
import oauth.signpost.basic.DefaultOAuthConsumer;
import oauth.signpost.basic.DefaultOAuthProvider;
import oauth.signpost.exception.OAuthCommunicationException;
import oauth.signpost.exception.OAuthExpectationFailedException;
import oauth.signpost.exception.OAuthMessageSignerException;
import oauth.signpost.exception.OAuthNotAuthorizedException;
import oauth.signpost.http.HttpParameters;
[...]
URL url;
url = new URL(signedRequestTokenUrl);
HttpURLConnection httpconnection = (HttpURLConnection) url
httpconnection.setRequestMethod("GET");
httpconnection.setRequestProperty("Content-type", "application/xml");
httpconnection.setRequestProperty("Content-Length", "0");
if (httpconnection != null) {
BufferedReader rd = new BufferedReader(new InputStreamReader(
httpconnection.getInputStream()));
[...]
Causing this Error:
javax.net.ssl.SSLHandshakeException: Could not verify SSL certificate for URL: https://oauth.intuit.com/oauth/v1/get_request_token?oauth_signature=VzStL8UcIoDrgKdcU7jJAWaux5Y%3D&oauth_callback=http%3A%2F%2Flocalhost%3A8888%2Faccesstoken.htm&oauth_consumer_key=qyprdulFn7zfTw5ewpZhkPxSo4q27X&oauth_version=1.0&oauth_signature_method=HMAC-SHA1&oauth_timestamp=1379761817&oauth_nonce=5706619579888946790
at com.google.appengine.api.urlfetch.URLFetchServiceImpl.convertApplicationException(URLFetchServiceImpl.java:144)
at com.google.appengine.api.urlfetch.URLFetchServiceImpl.fetch(URLFetchServiceImpl.java:43)
at com.google.apphosting.utils.security.urlfetch.URLFetchServiceStreamHandler$Connection.fetchResponse(URLFetchServiceStreamHandler.java:417)
at com.google.apphosting.utils.security.urlfetch.URLFetchServiceStreamHandler$Connection.getInputStream(URLFetchServiceStreamHandler.java:296)
at com.intuit.utils.OauthHelper.getRequestTokenSignPost(OauthHelper.java:167)
The url is as follows:
https://oauth.intuit.com/oauth/v1/get_request_token?oauth_signature=xxxx&oauth_callback=http%3A%2F%2Flocalhost%3A8888%2Faccesstoken.htm&oauth_consumer_key=xxxx&oauth_version=1.0&oauth_signature_method=HMAC-SHA1&oauth_timestamp=1379768671&oauth_nonce=-78989043
The exact same piece of code works perfectly on TomCat 7.
Signpost
I also tried to replace DefaultOAuthConsumer & DefaultOAuthProvider by the following:
oauth.signpost.commonshttp.CommonsHttpOAuthProvider;
oauth.signpost.commonshttp.CommonsHttpOAuthConsumer;
, but it generates exactly the same exception.
Any ideas?

And right after I say that I tried something that to me makes little sense, but seems to work, try this:
URLFetchService fetcher = URLFetchServiceFactory.getURLFetchService();
FetchOptions lFetchOptions = FetchOptions.Builder.validateCertificate();
HTTPRequest request = new HTTPRequest(url, HTTPMethod.GET, lFetchOptions);
HTTPResponse response = fetcher.fetch(request);
For me it was the FetchOptions.Builder and explicitly setting "validateCertificate()"; See if this solve it for you.