We Keep Coding

How do I exstract data from multiple websites using restful API and Spring? [closed]

Closed. This question needs to be more focused. It is not currently accepting answers.
Want to improve this question? Update the question so it focuses on one problem only by editing this post.
Closed 1 year ago.
Improve this question
I got a task at school in which I have to do the following:
Implement the RESTful endpoint API, which simultaneously makes calls
to the following websites:
https://pizzerijalimbo.si/meni/
https://pizzerijalimbo.si/kontakt/
https://pizzerijalimbo.si/my-account/
https://pizzerijalimbo.si/o-nas/
The input for the endpoint is ‘integer’, which represents the number
of simultaneous calls to the above web pages (min 1 represents all
consecutive calls, max 4 represents all simultaneous calls).
Extracts a short title text from each page and saves this text in a
common global structure (array, folder (). The program should also
count successful calls. Finally, the service should list the number of
successful calls, the number of failed calls and the saved address
texts from all web pages.
With some help I managed to do something, but I still need help with data exstraction using Jsoup or any other method.
Here is the code that I have:
import java.util.Arrays;
import java.util.List;
import java.io.IOException;
import java.net.URL;
import java.util.Scanner;
#RestController
public class APIcontroller {
#Autowired
private RestTemplate restTemplate;
List<String> websites = Arrays.asList("https://pizzerijalimbo.si/meni/",
"https://pizzerijalimbo.si/kontakt/",
"https://pizzerijalimbo.si/my-account/",
"https://pizzerijalimbo.si/o-nas/");
#GetMapping("/podatki")
public List<Object> getData(#RequestParam(required = true) int numberOfWebsites) {
List<String> websitesToScrape = websites.subList(0, numberOfWebsites);
for (String website : websitesToScrape) {
Document doc = Jsoup.connect("https://pizzerijalimbo.si/meni/").get();
log(doc.title());
Elements newsHeadlines = doc.select("#mp-itn b a");
for (Element headline : newsHeadlines) {
log("%s\n\t%s",
headline.attr("title"), headline.absUrl("href"));
}
}
}
}
I also need to do it parallel, so the calls to a secific website go on at the same time.
But the main problem now is with the log funcion which does not work properly.
What I have tried:
I tried to solve the problem using Jsoup library, but I dont seem to
undersand it well, so I got an error in the for loop which says that
the method log is undefined. I also need to do a try catch to count possible failed calls and count the calls that are successfull as you can see in the task description.

WebScrapperController.java
package com.stackovertwo.stackovertwo;
import java.io.IOException;
import java.io.StringReader;
import java.util.ArrayList;
import java.util.Arrays;
import java.util.List;
import java.util.concurrent.CompletableFuture;
import java.util.concurrent.ExecutionException;
import java.util.regex.Pattern;
import java.util.stream.Collectors;
import javax.xml.parsers.DocumentBuilderFactory;
import javax.xml.parsers.ParserConfigurationException;
import org.jsoup.Jsoup;
import org.jsoup.select.Elements;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.http.HttpEntity;
import org.springframework.http.HttpHeaders;
import org.springframework.http.HttpMethod;
import org.springframework.http.HttpStatus;
import org.springframework.http.ResponseEntity;
import org.springframework.web.bind.annotation.GetMapping;
import org.springframework.web.bind.annotation.RequestParam;
import org.springframework.web.bind.annotation.RestController;
import org.springframework.web.client.RestTemplate;
//import org.w3c.dom.Document;
//import org.w3c.dom.DocumentFragment;
import org.jsoup.nodes.Document;
import org.w3c.dom.Node;
import org.xml.sax.InputSource;
import org.xml.sax.SAXException;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
#RestController
public class WebScrapperController {
#GetMapping("/")
public String index() {
return "Greetings from Spring Boot!";
}
// #Autowired
// private RestTemplate restTemplate;
#Autowired
WebScrapperService webScrapperService;
List<String> websites = Arrays.asList("https://pizzerijalimbo.si/meni/",
"https://pizzerijalimbo.si/kontakt/",
"https://pizzerijalimbo.si/my-account/",
"https://pizzerijalimbo.si/o-nas/");
#GetMapping("/podatki")
public ResponseEntity<Object> getData(#RequestParam(required = true) int numberOfWebsites) throws InterruptedException, ExecutionException {
List<SiteResponse> webSitesToScrape = new ArrayList<>();
// List<String> websitesToScrape = websites.subList(0, numberOfWebsites);
List<SiteResponse> responseResults = new ArrayList<SiteResponse>();
CompletableFuture<SiteResponse> futureData1 = webScrapperService.getWebScrappedContent(websites.get(0));
CompletableFuture<SiteResponse> futureData2 = webScrapperService.getWebScrappedContent(websites.get(1));
//CompletableFuture.allOf(futureData1, futureData2).join();
webSitesToScrape.add(futureData1.get());
webSitesToScrape.add(futureData2.get());
List<SiteResponse> result = webSitesToScrape.stream().collect(Collectors.toList());
return ResponseEntity.ok().body(result);
}
}
WebScrapperService.java
package com.stackovertwo.stackovertwo;
import org.jsoup.Jsoup;
import org.jsoup.nodes.Document;
import org.jsoup.select.Elements;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.http.HttpEntity;
import org.springframework.http.HttpHeaders;
import org.springframework.http.HttpMethod;
import org.springframework.http.ResponseEntity;
import org.springframework.scheduling.annotation.Async;
import org.springframework.stereotype.Service;
import org.springframework.web.client.RestTemplate;
import java.util.concurrent.CompletableFuture;
#Service
public class WebScrapperService {
#Autowired
private RestTemplate restTemplate;
Logger logger = LoggerFactory.getLogger(WebScrapperService.class);
#Async
public CompletableFuture<SiteResponse> getWebScrappedContent(String webSiteURL)
//throws InterruptedException
{
logger.info("Starting: getWebScrappedContent for webSiteURL {} with thread {}", webSiteURL, Thread.currentThread().getName());
HttpEntity<String> response = restTemplate.exchange(webSiteURL,
HttpMethod.GET, null, String.class);
//Thread.sleep(1000);
SiteResponse webSiteSummary = null ;
String resultString = response.getBody();
HttpHeaders headers = response.getHeaders();
int statusCode = ((ResponseEntity<String>) response).getStatusCode().value();
System.out.println(statusCode);
System.out.println("HEADERS"+headers);
try
{
Document doc = (Document) Jsoup.parse(resultString);
Elements header = doc.select(".elementor-inner h2.elementor-heading-title.elementor-size-default");
System.out.println(header.get(0).html());
// Return the fragment.
webSiteSummary = new SiteResponse(statusCode, header.get(0).html());
}
catch(Exception e) {
System.out.println("Exception "+e.getMessage());
}
logger.info("Complete: getWebScrappedContent for webSiteURL {} with thread {}", webSiteURL, Thread.currentThread().getName());
return CompletableFuture.completedFuture(webSiteSummary);
}
}
SpringBootApp.java
package com.stackovertwo.stackovertwo;
import org.springframework.boot.SpringApplication;
import org.springframework.boot.autoconfigure.SpringBootApplication;
import org.springframework.context.annotation.Bean;
import org.springframework.http.client.HttpComponentsClientHttpRequestFactory;
import org.springframework.web.client.RestTemplate;
import java.security.KeyManagementException;
import java.security.KeyStoreException;
import java.security.NoSuchAlgorithmException;
import java.security.cert.X509Certificate;
//import javax.net.ssl.HostnameVerifier;
//import javax.net.ssl.HttpsURLConnection;
import javax.net.ssl.SSLContext;
//import javax.net.ssl.SSLSession;
//import javax.net.ssl.TrustManager;
//import javax.net.ssl.X509TrustManager;
//import javax.security.cert.X509Certificate;
import org.apache.http.conn.ssl.TrustStrategy;
import org.apache.http.impl.client.*;
import org.apache.http.conn.ssl.*;
#SpringBootApplication
public class SpringBootApp
{
public static void main(String[] args)
{
SpringApplication.run(SpringBootApp.class, args);
}
#Bean
public RestTemplate restTemplate() throws KeyManagementException, NoSuchAlgorithmException, KeyStoreException {
TrustStrategy acceptingTrustStrategy = (X509Certificate[] chain, String authType) -> true;
SSLContext sslContext = org.apache.http.ssl.SSLContexts.custom()
.loadTrustMaterial(null, acceptingTrustStrategy)
.build();
SSLConnectionSocketFactory csf = new SSLConnectionSocketFactory(sslContext);
CloseableHttpClient httpClient = HttpClients.custom()
.setSSLSocketFactory(csf)
.build();
HttpComponentsClientHttpRequestFactory requestFactory =
new HttpComponentsClientHttpRequestFactory();
requestFactory.setHttpClient(httpClient);
//return new RestTemplate();
RestTemplate restTemplate = new RestTemplate(requestFactory);
return restTemplate;
}
}
Note: I disabled SSL verification while calling the webulr in resttemplate, but its not recommendd inproduction (For assignment its ok). But you need to import the keys via java keystore in case production : https://myshittycode.com/2015/12/17/java-https-unable-to-find-valid-certification-path-to-requested-target-2/

UIMA Pipeline with Stanford NER

I have integrated Stanford NER in UIMA and developed a pipeline.
The pipeline contains a FileSystemCollectionReader,an NERAnnotator and a CasConsumer but the output so come isn't desired. In my input directory i have two files and after running the pipeline, i get two files as ouput but the second file is getting merged with the first file in second ouput. I don't know what's happening here.
The code for CasConsumer:
`
package org.gds.uima;
import java.io.File;
import java.io.FileNotFoundException;
import java.io.FileOutputStream;
import java.io.PrintWriter;
import java.util.ArrayList;
import java.util.Collection;
import java.util.Iterator;
import java.util.List;
import java.util.UUID;
import org.apache.uima.UimaContext;
import org.apache.uima.analysis_component.AnalysisComponent_ImplBase;
import org.apache.uima.analysis_component.CasAnnotator_ImplBase;
import org.apache.uima.analysis_component.JCasAnnotator_ImplBase;
import org.apache.uima.analysis_engine.AnalysisEngineProcessException;
import org.apache.uima.cas.CAS;
import org.apache.uima.cas.CASException;
import org.apache.uima.cas.Type;
import org.apache.uima.cas.text.AnnotationFS;
import org.apache.uima.fit.component.CasConsumer_ImplBase;
import org.apache.uima.fit.component.JCasConsumer_ImplBase;
import org.apache.uima.fit.descriptor.ConfigurationParameter;
import org.apache.uima.fit.util.CasUtil;
import org.apache.uima.jcas.JCas;
import org.apache.uima.resource.ResourceInitializationException;
public class CasConsumer extends JCasConsumer_ImplBase
{
public final static String PARAM_OUTPUT="outputDir";
#ConfigurationParameter(name = PARAM_OUTPUT)
private String outputDirectory;
public final static String PARAM_ANNOTATION_TYPES = "annotationTypes";
enter code here
#ConfigurationParameter(name = PARAM_ANNOTATION_TYPES,defaultValue="String")
public List<String> annotationTypes;
public void initialize(final UimaContext context) throws ResourceInitializationException
{
super.initialize(context);
}
#Override
public void process(JCas jcas)
{
String original = jcas.getDocumentText();
try
{
String onlyText="";
JCas sofaText = jcas.getView(NERAnnotator.SOFA_NAME);
onlyText = sofaText.getDocumentText();
String name = UUID.randomUUID().toString().substring(20);
File outputDir = new File(this.outputDirectory+"/"+name);
System.out.print("Saving file to "+outputDir.getAbsolutePath());
FileOutputStream fos = new FileOutputStream(outputDir.getAbsoluteFile());
PrintWriter pw = new PrintWriter(fos);
pw.println(onlyText);
pw.close();
}
catch(CASException cae)
{
System.out.println(cae);
}
catch(FileNotFoundException fne)
{
System.out.print(fne);
}
}
}
`
}

Consume Java weblogic web service having usernametoken and client certificate in ASP.Net

i need to consume a web service developed with java web logic server, usernametoken and client certificate are used to invoke the service.
and i got the java code to call web service as below
import javax.ejb.CreateException;
import javax.ejb.SessionBean;
import javax.ejb.SessionContext;
import javax.naming.Context;
import javax.naming.InitialContext;
import javax.naming.NamingException;
import java.rmi.RemoteException;
import weblogic.wtc.jatmi.TPException;
import weblogic.wtc.jatmi.TypedString;
import weblogic.wtc.jatmi.Reply;
import weblogic.wtc.jatmi.TuxedoReply;
import weblogic.wtc.jatmi.TPServiceInformation;
import weblogic.wtc.jatmi.TypedFML32;
import weblogic.wtc.jatmi.TuxedoService;
import weblogic.wtc.jatmi.TypedBuffer;
import java.util.StringTokenizer;
import java.io.IOException;
import javax.xml.rpc.ServiceException;
import MMSWbSrv.*;
import java.util.Properties;
import java.text.SimpleDateFormat;
import java.io.BufferedReader;
import java.io.InputStreamReader;
import java.io.FileInputStream;
import java.security.cert.X509Certificate;
import weblogic.security.SSL.TrustManager;
import weblogic.xml.crypto.wss.provider.CredentialProvider;
import weblogic.xml.crypto.wss.WSSecurityContext;
import weblogic.wsee.security.bst.ClientBSTCredentialProvider;
import weblogic.wsee.security.unt.ClientUNTCredentialProvider;
import java.security.cert.X509Certificate;
import weblogic.wsee.security.util.CertUtils;
import javax.xml.rpc.Stub;
import java.util.List;
import java.util.ArrayList;
import java.util.regex.*;
import weblogic.wtc.jatmi.TypedFML32;
public class MMSTest
{
public static void main(String args[]) throws Throwable
{
Properties sysProps = System.getProperties();
sysProps.setProperty("http.proxyHost","XXX.XX.X.XX");
sysProps.setProperty("http.proxyPort","XXXX");
sysProps.setProperty("weblogic.webservice.transport.http.proxy.host", "XXX.XX.X.XX");
sysProps.setProperty("weblogic.webservice.transport.http.proxy.port", "XXXX");
sysProps.setProperty("http.proxySet","true");
sysProps.setProperty("weblogic.webservice.verbose","true");
sysProps.setProperty("javax.net.ssl.trustStore","D:/weblogic92/server/lib/DemoTrust.jks");
System.out.println("proxyHost::"+ sysProps.getProperty("http.proxyHost"));
System.out.println("proxyPort::"+ sysProps.getProperty("http.proxyPort"));
MMSService service = null;
MMSPortType port = null;
try{
System.out.println("before..");
service = new MMSService_Impl("http://XXX.XX.X.XX/MMS/Service?wsdl");
System.out.println("after..");
if(service==null)
System.out.println("NULL SERVICE");
System.out.println("got service...");
port = service.getMMSServicePort();
if(port==null)
System.out.println("NULL PORT");
}
catch(ServiceException svcEx)
{
System.out.println("In ServiceXexception");
}
List credProviders = new ArrayList();
System.out.println("After cred");
String clientCertFile = "F:/client-cert.der";
String keyFile = "F:/client-key-pkcs8.der";
String username = "userName" ;
String password = "Password" ;
CredentialProvider cp = new ClientBSTCredentialProvider(clientCertFile, keyFile);
credProviders.add(cp);
System.out.println("credProviders:"+credProviders);
System.out.println("cp:"+cp+":");
String strret= "";
try{
cp = new ClientUNTCredentialProvider(username, password);
credProviders.add(cp);
Stub stub = (Stub)port;
System.out.println("before");
stub._setProperty(WSSecurityContext.CREDENTIAL_PROVIDER_LIST, credProviders);
System.out.println("after");
stub._setProperty(WSSecurityContext.TRUST_MANAGER,
new TrustManager()
{
public boolean certificateCallback(X509Certificate[] chain, int validateErr)
{
return true;
}
}
);
String strArg = "Parameters";
strret = port.sendData(strArg);
}
catch(Exception e)
{
System.out.println("JJJ");
e.printStackTrace();
}
System.out.println("strret:"+strret);
}
}
and also .pfx format client certificate and .key file .
So using WSE3.0 i try to consume the service like below
WebReference.MMSServiceWse wse = new MMSServiceWse();
UsernameToken token = new UsernameToken("XXXX", "XXXX", PasswordOption.SendPlainText);
wse.RequestSoapContext.Security.Tokens.Add(token);
X509Certificate2 cert = new X509Certificate2(#"D:\\certificate.pfx","",X509KeyStorageFlags.MachineKeySet);
wse.ClientCertificates.Add(cert);
wse.SetPolicy("ClientPolicy");
//System.Net.ServicePointManager.CertificatePolicy =new TrustAllCertificatePolicy();
String strArg = "Param";
strArg= wse.sendData(strArg);
but i have got exception.

When this happens to me, it is usually because the time on the server and the client are too different. The security token has a validity period and if you're not in sync, it seems expired or worse, in the future.

JNA - Query Windows Processes

I'm trying to use JNA to return details on a specific Windows Process. Not exactly sure how to do this. Couldn't find much on the interwebs for help. Some information I would like returned include CPU and memory usage. The below is just an example I found.
import java.io.BufferedReader;
import java.io.BufferedWriter;
import java.io.DataInputStream;
import java.io.FileInputStream;
import java.io.FileWriter;
import java.io.IOException;
import java.io.InputStreamReader;
import com.sun.jna.*;
import com.sun.jna.Library.Handler;
import com.sun.jna.platform.win32.*;
import com.sun.jna.platform.win32.Advapi32Util.*;
import com.sun.jna.platform.win32.WinNT.*;
import com.sun.jna.ptr.IntByReference;
import com.sun.jna.win32.*;
import com.sun.jna.Native;
import com.sun.jna.platform.win32.*;
import com.sun.jna.win32.W32APIOptions;
public class WindowsProcess {
public static void main(String[] args) {
WinNT winNT = (WinNT) Native.loadLibrary(WinNT.class, W32APIOptions.UNICODE_OPTIONS);
WinNT.HANDLE snapshot = winNT.CreateToolhelp32Snapshot(Tlhelp32.TH32CS_SNAPPROCESS, new WinDef.DWORD(0));
Thelp32.PROCESSENTRY32.ByReference processEntry = new Tlhelp32.PROCESSENTRY32.ByReference();
while (winNT.Process32Next(snapshot, processEntry)) {
System.out.println(processEntry.th32ProcessID + "\t" + Native.toString(processEntry.szExeFile));
}
winNT.CloseHandle(snapshot);
}
}

This works with JNA 3.5.0. The example you have isn't compatible with more recent versions of the library, I think.
import com.sun.jna.platform.win32.Kernel32;
import com.sun.jna.platform.win32.Tlhelp32;
import com.sun.jna.platform.win32.WinDef;
import com.sun.jna.platform.win32.WinNT;
import com.sun.jna.win32.W32APIOptions;
import com.sun.jna.Native;
public class ListProcesses {
public static void main(String[] args) {
Kernel32 kernel32 = (Kernel32) Native.loadLibrary(Kernel32.class, W32APIOptions.UNICODE_OPTIONS);
Tlhelp32.PROCESSENTRY32.ByReference processEntry = new Tlhelp32.PROCESSENTRY32.ByReference();
WinNT.HANDLE snapshot = kernel32.CreateToolhelp32Snapshot(Tlhelp32.TH32CS_SNAPPROCESS, new WinDef.DWORD(0));
try {
while (kernel32.Process32Next(snapshot, processEntry)) {
System.out.println(processEntry.th32ProcessID + "\t" + Native.toString(processEntry.szExeFile));
}
}
finally {
kernel32.CloseHandle(snapshot);
}
}
}
See also my answer elsewhere.

java unit test mock HttpClient and webdav

Hello I have a class for doing webdav related operations such as creating a directory, Implementatiion can be seen below (the createDir method). The question is how to test it nicely, perhaps using EasyMock or a similar lib. Any ideas? thanks!
package foobar;
import java.io.File;
import java.io.FileInputStream;
import java.io.IOException;
import org.apache.commons.httpclient.*;
import org.apache.commons.httpclient.HttpMethod;
import org.apache.commons.httpclient.methods.DeleteMethod;
import org.apache.commons.httpclient.methods.GetMethod;
import org.apache.commons.httpclient.methods.InputStreamRequestEntity;
import org.apache.commons.httpclient.methods.PutMethod;
import org.apache.commons.httpclient.methods.RequestEntity;
import org.apache.jackrabbit.webdav.client.methods.DavMethod;
import org.apache.jackrabbit.webdav.client.methods.MkColMethod;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
import mypackage.httpdclient.util.URLHandler;
public class WebDavImpl{
private static final String SEPARATOR = " ----- ";
private HttpClient httpClient;
public StorageSpaceClientImpl() {
httpClient = new HttpClient();
}
public String createDir(String dirName) {
String response = null;
String url = URLHandler.getInstance().getDirectoryUrl(dirName);
DavMethod mkcol = new MkColMethod(url);
try {
httpClient.executeMethod(mkcol);
response = mkcol.getStatusCode() + SEPARATOR + mkcol.getStatusText();
} catch (IOException ex) {
} finally {
mkcol.releaseConnection();
}
return response;
}
}