We Keep Coding

Mock an instance method call of another class that takes a URL and File object and writes a file to a HTTP connection

I'm trying to write UTs for a file called DocumentStoreAccessor.java. Here is the class below:
package com.company.main.accessor;
import com.company.main.dagger.component.AccessorComponent;
import com.company.main.dagger.component.DaggerAccessorComponent;
import com.company.main.util.aws.s3.AWSS3Util;
import lombok.NonNull;
import lombok.RequiredArgsConstructor;
import lombok.extern.slf4j.Slf4j;
import java.io.File;
import java.io.IOException;
import java.net.MalformedURLException;
import java.net.URL;
#Slf4j
#RequiredArgsConstructor
public class DocumentStoreAccessor {
private final DocumentStore documentStoreClient; //Comes from Dagger
public DocumentStoreAccessor() {
AccessorComponent accessorComponent = DaggerAccessorComponent.create();
this.documentStoreClient = accessorComponent.provideDocumentStoreClient();
}
private int putContentsIntoS3(CreateUploadS3UrlResult createUploadS3UrlResponse,
#NonNull File file) {
int uploadStatusCode = 0;
try {
URL url = new URL(createUploadS3UrlResponse.getS3Url());
uploadStatusCode = new AWSS3Util().upload(url, file); //Instance comes from a util class
} catch (MalformedURLException e) {
e.printStackTrace();
} catch (IOException e) {
e.printStackTrace();
}
return uploadStatusCode;
}
public String uploadFile(File file, DocumentFileExtension fileExtension) throws Exception {
String documentId = null;
CreateUploadS3UrlResult createUploadS3Urlresult = documentStoreClient.createUploadS3Url(new CreateUploadS3UrlRequest());
int putContentsStatusCode = putContentsIntoS3(createUploadS3Urlresult, file);
if (putContentsStatusCode == 200) {
CreateDocumentRequest createDocumentRequest = new CreateDocumentRequest()
CreateDocumentResult document = documentStoreClient.createDocument(createDocumentRequest);
documentId = document.getDocumentId();
} else {
throw new Exception("Status code is: " + putContentsStatusCode);
}
return documentId;
}
}
Inside this file I do new AWSS3Util().upload(url, file).
And here is the AWSS3Util.java
package com.company.main.util.aws.s3;
import com.company.main.exception.DataAccessException;
import com.company.main.exception.RetriableDataAccessException;
import lombok.AllArgsConstructor;
import lombok.Builder;
import lombok.NonNull;
import lombok.extern.slf4j.Slf4j;
import java.io.BufferedReader;
import java.io.File;
import java.io.FileInputStream;
import java.io.IOException;
import java.io.InputStream;
import java.io.InputStreamReader;
import java.io.OutputStreamWriter;
import java.io.Reader;
import java.net.HttpURLConnection;
import java.net.URL;
import java.nio.charset.StandardCharsets;
#Slf4j
#AllArgsConstructor
#Builder
public class AWSS3Util {
private static final String PUT_REQUEST_METHOD = "PUT";
public int upload(#NonNull final URL url, #NonNull final File file) throws IOException {
final InputStream inputStream = new FileInputStream(file);
final Reader fileReader = new InputStreamReader(inputStream, StandardCharsets.UTF_8);
final BufferedReader br = new BufferedReader(fileReader);
HttpURLConnection connection = null;
int responseCode = 0;
try {
// Create the connection and use it to upload the new object using the pre-signed URL.
connection = create(url);
connection.setDoOutput(true);
connection.setRequestMethod(PUT_REQUEST_METHOD);
final OutputStreamWriter out = new OutputStreamWriter(connection.getOutputStream(), StandardCharsets.UTF_8);
String st;
while ((st = br.readLine()) != null) {
out.write(st);
}
out.close();
responseCode = connection.getResponseCode();
} catch (IOException e) {
throw new RetriableDataAccessException(String.format("S3 upload request failed with request: %s", url.toString()), e);
} finally {
inputStream.close();
fileReader.close();
br.close();
}
return responseCode;
}
private HttpURLConnection create(URL url) throws IOException {
return (HttpURLConnection) url.openConnection();
}
}
I want to make new AWSS3Util().upload(url, file) return a 200..
I'm unable to do so.. I keep getting a NullPointerException. Here is what I have for the past day:
package com.company.main.accessor;
import com.company.main.util.aws.s3.AWSS3Util;
import org.junit.jupiter.api.BeforeEach;
import org.junit.jupiter.api.Test;
import org.junit.jupiter.api.extension.ExtendWith;
import org.mockito.InjectMocks;
import org.mockito.Mock;
import org.mockito.Mockito;
import org.mockito.junit.jupiter.MockitoExtension;
import java.io.File;
import java.io.IOException;
import java.io.OutputStream;
import java.net.HttpURLConnection;
import java.net.URL;
import java.net.URLConnection;
import java.net.URLStreamHandler;
import static org.junit.jupiter.api.Assertions.assertEquals;
import static org.mockito.Mockito.times;
import static org.mockito.Mockito.verify;
import static org.mockito.Mockito.when;
#ExtendWith(MockitoExtension.class)
public class DocumentStoreAccessorTest {
#Mock
private DocumentStore mockDocumentStoreClient;
#Mock
private HttpURLConnection httpURLConnection;
#Mock
private OutputStream outputStream;
#InjectMocks
private DocumentStoreAccessor classUnderTest;
private URL url;
private AWSS3Util awss3Util;
private CreateUploadS3UrlRequest dummyCreateUploadS3UrlRequest;
private CreateUploadS3UrlResult dummyCreateUploadS3UrlResult;
private CreateDocumentRequest dummyCreateDocumentRequest;
private CreateDocumentResult dummyCreateDocumentResult;
#BeforeEach
void setUp() throws IOException {
awss3Util = new AWSS3Util();
url = getMockUrl(httpURLConnection);
dummyCreateUploadS3UrlRequest = new CreateUploadS3UrlRequest();
dummyCreateUploadS3UrlResult = new CreateUploadS3UrlResult().withS3Url("http://foo.io://:99");
dummyCreateDocumentRequest = new CreateDocumentRequest();
dummyCreateDocumentResult = new CreateDocumentResult().withDocumentId("foo");
}
#Test
void uploadSuccess() throws IOException {
when(mockDocumentStoreClient.createUploadS3Url(dummyCreateUploadS3UrlRequest)).thenReturn(dummyCreateUploadS3UrlResult);
AWSS3Util aSpy = Mockito.spy(awss3Util);
Mockito.when(aSpy.upload(url, getDataToUpload())).thenReturn(200);
when(mockDocumentStoreClient.createDocument(dummyCreateDocumentRequest)).thenReturn(dummyCreateDocumentResult);
String id = classUnderTest.uploadFile(getDataToUpload(), DocumentFileExtension.XLSX);
assertEquals(id, "foo");
verify(mockDocumentStoreClient, times(1)).createUploadS3Url(dummyCreateUploadS3UrlRequest);
}
private File getDataToUpload() {
return new File("TestFileName.xlsx");
}
/**
* We cannot directly use Mockito to mock URL. This helper method, helps us to create the mock url.
* <p>
* https://stackoverflow.com/questions/565535/mocking-a-url-in-java
*/
private URL getMockUrl(HttpURLConnection httpURLConnection) throws IOException {
final URLStreamHandler handler = new URLStreamHandler() {
#Override
protected URLConnection openConnection(final URL arg0)
throws IOException {
return httpURLConnection;
}
};
final URL url = new URL("http://foo.io", "foo.io", 80, "", handler);
return url;
}
}
I'm unable to mock the URL class, so I followed a another Stackoverflow post..
The AWSS3Util cannot come through Dagger, it is a util class that we're all using so this must not change.
I'm not sure if I'm going in the right direction.. I've tried "spying" on that AWSS3Util class. I want this method to return a 200 or any status code of my choice to cover them in UTs by asserting a String return as seen in the example below
I can change the the upload method inside AWSS3Util to static if it helps UTs
Cannot use PowerMockito (this is a last resort)
Try incorporating Yan's comments:
#Test
void verifyCreateUploadS3UrlInvocation() throws Exception {
when(mockDocumentStoreClient.createUploadS3Url(dummyCreateUploadS3UrlRequest)).thenReturn(dummyCreateUploadS3UrlResult);
when(httpURLConnection.getOutputStream()).thenReturn(outputStream);
when(httpURLConnection.getResponseCode()).thenReturn(200);
when(awss3Util.upload(url, getDataToUpload())).thenReturn(200);
when(mockDocumentStoreClient.createDocument(dummyCreateDocumentRequest)).thenReturn(dummyCreateDocumentResult);
String id = classUnderTest.uploadFile(getDataToUpload(), DocumentFileExtension.XLSX);
assertEquals(id, "foo");
verify(mockDocumentStoreClient, times(1)).createUploadS3Url(dummyCreateUploadS3UrlRequest);
}
I get a 405 thrown when I specifically want it to send a 200, and therefore java.lang.Exception: Status code is: 405 thrown from my function.

Firstly I would change a little bit AWSS3Util, because reading binary files as string can lead to very interesting results.
import lombok.NonNull;
import java.io.*;
import java.net.HttpURLConnection;
import java.net.URL;
public class AWSS3Util {
private static final String PUT_REQUEST_METHOD = "PUT";
public int upload(#NonNull final URL url, #NonNull final File file) throws IOException {
HttpURLConnection connection = create(url);
try (BufferedInputStream bis = new BufferedInputStream(new FileInputStream(file))) {
connection.setDoOutput(true);
connection.setRequestMethod(PUT_REQUEST_METHOD);
try (BufferedOutputStream bos = new BufferedOutputStream(connection.getOutputStream())) {
byte[] buffer = new byte[1024];
int len;
while ((len = bis.read(buffer)) > 0) {
bos.write(buffer, 0, len);
}
}
return connection.getResponseCode();
}
}
private HttpURLConnection create(URL url) throws IOException {
return (HttpURLConnection) url.openConnection();
}
}
Test for upload method could be like this:
import org.junit.jupiter.api.Test;
import java.io.*;
import java.net.HttpURLConnection;
import java.net.URL;
import java.net.URLConnection;
import java.net.URLStreamHandler;
import java.nio.file.Files;
import static org.junit.jupiter.api.Assertions.assertArrayEquals;
import static org.junit.jupiter.api.Assertions.assertEquals;
import static org.mockito.Mockito.mock;
import static org.mockito.Mockito.when;
public class MyTest {
#Test
public void test() throws IOException {
final HttpURLConnection mockUrlCon = mock(HttpURLConnection .class);
URLStreamHandler stubUrlHandler = new URLStreamHandler() {
#Override
protected URLConnection openConnection(URL u) throws IOException {
return mockUrlCon;
}
};
URL url = new URL("http://foo.io", "foo.io", 80, "", stubUrlHandler);
when(mockUrlCon.getResponseCode()).thenReturn(200);
ByteArrayOutputStream outputStream = new ByteArrayOutputStream();
when(mockUrlCon.getOutputStream()).thenReturn(outputStream);
File file = new File("c:\\anyFile.png");
int responseCode = new AWSS3Util().upload(url, file);
assertEquals(200, responseCode);
byte[] expectedBytes = Files.readAllBytes(file.toPath());
byte[] actualBytes = outputStream.toByteArray();
assertArrayEquals(expectedBytes, actualBytes);
}
}

I had to violate the second rule - I had to DependencyInject it via Dagger
package com.company.main.dagger.module;
import com.company.main.util.aws.s3.AWSS3Util;
import dagger.Module;
import dagger.Provides;
import javax.inject.Singleton;
#Module
public class AWSS3UtilModule {
#Provides
#Singleton
public static AWSS3Util provideAwss3Util() {
return new AWSS3Util();
}
}
and then in my component
import javax.inject.Singleton;
#Singleton
#Component(modules = {
ShipDocsDMSAccessorModule.class,
AWSS3UtilModule.class
})
public interface AccessorComponent {
ShipDocsDMS provideShipDocsDMSClient();
AWSS3Util provideAwss3Util();
}
and then in my main class
#Slf4j
#RequiredArgsConstructor
public class ShipDocsDMSAccessor {
private final ShipDocsDMS shipDocsDMSClient;
private final AWSS3Util awss3Util;
public ShipDocsDMSAccessor() {
AccessorComponent accessorComponent = DaggerAccessorComponent.create();
this.shipDocsDMSClient = accessorComponent.provideShipDocsDMSClient();
this.awss3Util = accessorComponent.provideAwss3Util();
}
.
.
.
.
and then in my test, I can freely mock AWSS3Util dependency and carry ahead and force out the required response.

Non-blocking FileWalkTree file search with Webflux

I need to create a non-blocking functionality where I search text files within a given folder and it returns the count of search terms found in it.
I am able to execute the test in a blocking manner. I am wondering if anyone could help me transform it into a non-blocking task so that whenever a file is finished being scanned the result delivered without waiting for all the files to be scanned.
The main point is: I don't want to wait for all files to be scanned to start delivering the result to the client (Angular app).
public interface SearchService {
List<SearchResponse> search(SearchRequest searchRequest);
}
import lombok.Getter;
import lombok.RequiredArgsConstructor;
import lombok.ToString;
import java.io.Serializable;
#ToString
#Getter
#RequiredArgsConstructor(staticName = "of")
public class SearchResponse implements Serializable {
private final String server;
private final String filePath;
private final long count;
private final boolean errorReadingFile;
}
import lombok.Getter;
import lombok.RequiredArgsConstructor;
import lombok.ToString;
import javax.validation.constraints.NotNull;
import java.io.Serializable;
#ToString
#Getter
#RequiredArgsConstructor(staticName = "of")
public class SearchRequest implements Serializable {
#NotNull
private final String server;
#NotNull
private final String rootPath;
#NotNull
private final String searchTerm;
}
import lombok.extern.slf4j.Slf4j;
import org.springframework.stereotype.Service;
import java.io.IOException;
import java.nio.file.*;
import java.util.ArrayList;
import java.util.EnumSet;
import java.util.List;
#Slf4j
#Service
public class FileSearchService implements SearchService {
#Override
public List<SearchResponse> search(SearchRequest searchRequest) {
Path start = Paths.get(searchRequest.getRootPath());
EnumSet<FileVisitOption> opts = EnumSet.of(FileVisitOption.FOLLOW_LINKS);
int maxDepth = Integer.MAX_VALUE;
SearchTermFileVisitor visitor = new SearchTermFileVisitor(searchRequest, new ArrayList<>());
try {
Files.walkFileTree(start,opts,maxDepth, visitor);
return visitor.getSearchResponseList();
} catch (IOException e) {
System.out.println(e.getMessage());
}
}
}
import lombok.Getter;
import lombok.RequiredArgsConstructor;
import lombok.extern.slf4j.Slf4j;
import java.io.BufferedReader;
import java.io.IOException;
import java.nio.file.*;
import java.nio.file.attribute.BasicFileAttributes;
import java.util.List;
import java.util.regex.Matcher;
import java.util.regex.Pattern;
import java.util.stream.Stream;
#Slf4j
#Getter
#RequiredArgsConstructor
public class SearchTermFileVisitor extends SimpleFileVisitor<Path> {
private final SearchRequest searchRequest;
private final List<SearchResponse> searchResponseList;
private SearchResponse searchFileContent(Path path, SearchRequest searchRequest) {
SearchResponse response;
try (BufferedReader br = Files.newBufferedReader(path)) {
response = SearchResponse.of(
searchRequest.getServer(),
Paths.get(path.toUri()).toString(),
countWordsInFile(searchRequest.getSearchTerm(), br.lines()),
false);
} catch (Exception e) {
response = SearchResponse.of(
searchRequest.getServer(),
path.toString(),
0,
true);
}
log.debug(response.toString());
return response;
}
private int countWordsInFile(String searchTerm, Stream<String> linesStream) {
return linesStream
.parallel()
.map(line -> countWordsInLine(line, searchTerm))
.reduce(0, Integer::sum);
}
private int countWordsInLine(String line, String searchTerm) {
Pattern pattern = Pattern.compile(searchTerm.toLowerCase());
Matcher matcher = pattern.matcher(line.toLowerCase());
int count = 0;
int i = 0;
while (matcher.find(i)) {
count++;
i = matcher.start() + 1;
}
return count;
}
private boolean isTextFile(Path path) throws IOException {
String type = Files.probeContentType(path);
if (type == null) {
//type couldn't be determined, assume binary
return false;
} else if (type.startsWith("text")) {
return true;
} else {
//type isn't text
return false;
}
}
#Override
public FileVisitResult postVisitDirectory(Path dir, IOException exc) {
log.debug("Visited: " + (Path) dir);
return FileVisitResult.CONTINUE;
}
#Override
public FileVisitResult preVisitDirectory(Path dir, BasicFileAttributes attrs) {
return FileVisitResult.CONTINUE;
}
#Override
public FileVisitResult visitFile(Path file, BasicFileAttributes attrs) throws IOException {
if (attrs.isRegularFile()
&& !attrs.isDirectory()
&& !attrs.isSymbolicLink()
&& isTextFile(file)) {
searchResponseList.add(searchFileContent(file, searchRequest));
}
return FileVisitResult.CONTINUE;
}
#Override
public FileVisitResult visitFileFailed(Path file, IOException exc) {
return FileVisitResult.CONTINUE;
}
}
The test case:
import org.junit.jupiter.api.Assertions;
import org.junit.jupiter.api.BeforeEach;
import org.junit.jupiter.api.Test;
import java.net.URISyntaxException;
import java.nio.file.Paths;
import java.util.List;
class FileSearchServiceTest {
private SearchService searchService = new FileSearchService();
#Test
void test_search_window_root_c_path() {
SearchRequest sr = SearchRequest.of("localhost", "c:\\", "a");
final List<SearchResponse> searchResult = searchService.search(sr);
Assertions.assertNotNull(searchResult.size());
}
}
I want to use WebFlux to receive the results one by one without waiting for all the files to be scanned.

Consider (1) performing the search in a separate thread, (2) using the observer pattern for receiving intermediate results back to the thread created code (references below), and (3) join the search thread to the implementing code so you can return the result list once it is finished. It means you need to pass 'this' to the thread so it has a reference to call back to a separate method for intermediate results. The references below contain sample code.
Think of this like a GUI. You can run the GUI in a separate thread, and for every button click it calls back to the controller code (including a 'finish' once Exit is clicked).
Re: https://dzone.com/articles/the-observer-pattern-using-modern-java
https://en.wikipedia.org/wiki/Observer_pattern

Jar ignoring all methods

I was making a pretty simple jar to unzip a zip and run the jar that was inside of it. The problem I've run into is that it doesn't do anything at all.
This is the main, and only class file for the jar. The manifest does point correctly to it, and it loads without errors.
import java.io.File;
import java.io.IOException;
import java.io.OutputStream;
import java.io.FileWriter;
import java.io.BufferedWriter;
import java.io.BufferedOutputStream;
import java.io.BufferedInputStream;
import java.io.BufferedReader;
import java.io.FileOutputStream;
import java.io.FileInputStream;
import java.io.FileNotFoundException;
import java.io.InputStream;
import java.io.InputStreamReader;
import static java.lang.Integer.parseInt;
import java.net.URLConnection;
import java.net.URL;
import java.util.zip.ZipFile;
import java.util.zip.ZipEntry;
import java.util.zip.ZipInputStream;
import java.util.Enumeration;
import sign.signlink;
import java.nio.file.*;
import java.io.FileReader;
public class ClientUpdater {
private String fileToExtractNew = "/client.zip";
private String getJarDir() throws FileNotFoundException, IOException{
String linebuf="",verStr="";
FileInputStream fis = new FileInputStream("/runLocationURL.txt");
BufferedReader br= new BufferedReader(new InputStreamReader(fis));
while ((linebuf = br.readLine()) != null) {
verStr = linebuf;
}
return verStr;
}
public static void main(String[] args) {
System.out.println("start");
}
private void unZip() {
System.out.println("unzipping");
try {
ZipEntry zipEntry;
//client
BufferedInputStream bufferedInputStreamNew = new BufferedInputStream(new FileInputStream(this.fileToExtractNew));
ZipInputStream zipInputStreamNew = new ZipInputStream(bufferedInputStreamNew);
//client
while ((zipEntry = zipInputStreamNew.getNextEntry()) != null) {
String stringNew = zipEntry.getName();
File fileNew = new File(this.getJarDir() + File.separator + stringNew);
if (zipEntry.isDirectory()) {
new File(this.getJarDir() + zipEntry.getName()).mkdirs();
continue;
}
if (zipEntry.getName().equals(this.fileToExtractNew)) {
this.unzipNew(zipInputStreamNew, this.fileToExtractNew);
break;
}
new File(fileNew.getParent()).mkdirs();
this.unzipNew(zipInputStreamNew, this.getJarDir() + zipEntry.getName());
}
zipInputStreamNew.close();
}
catch (Exception var1_2) {
var1_2.printStackTrace();
}
}
private void unzipNew(ZipInputStream zipInputStreamNew, String stringNew) throws IOException {
System.out.println("unzipping new");
FileOutputStream fileOutputStreamNew = new FileOutputStream(stringNew);
byte[] arrby = new byte[4024];
int n = 0;
while ((n = zipInputStreamNew.read(arrby)) != -1) {
fileOutputStreamNew.write(arrby, 0, n);
}
fileOutputStreamNew.close();
Runtime.getRuntime().exec("java -jar " + getJarDir() + "/Project Pk Client.jar");
System.exit(0);
}
}
It shows the "Start" message, but not the other 2, so it never reaches those methods. Is it because they aren't being called? I'm still learning Java.

You actually have to call your other methods from main. Right now, all you are telling the computer to do is print start and then exit. Functions do not get called simply by existing.
It seems based on a quick glance that you just need to add unzip(); to your main function, right after the System.out.println line.
To do this, you need to say that those other methods are static, so you need to say private static void unZip() instead of private void unZip(). Do this for your other methods too.
import java.io.*;
import static java.lang.Integer.parseInt;
import java.net.URLConnection;
import java.net.URL;
import java.util.zip.ZipFile;
import java.util.zip.ZipEntry;
import java.util.zip.ZipInputStream;
import java.util.Enumeration;
import sign.signlink;
import java.nio.file.*;
public class ClientUpdater {
private String fileToExtractNew = "/client.zip";
private static String getJarDir() throws FileNotFoundException, IOException{
String linebuf="",verStr="";
FileInputStream fis = new FileInputStream("/runLocationURL.txt");
BufferedReader br= new BufferedReader(new InputStreamReader(fis));
while ((linebuf = br.readLine()) != null) {
verStr = linebuf;
}
return verStr;
}
public static void main(String[] args) {
System.out.println("start");
unZip();
}
private static void unZip() {
System.out.println("unzipping");
try {
ZipEntry zipEntry;
//client
BufferedInputStream bufferedInputStreamNew = new BufferedInputStream(new FileInputStream(this.fileToExtractNew));
ZipInputStream zipInputStreamNew = new ZipInputStream(bufferedInputStreamNew);
//client
while ((zipEntry = zipInputStreamNew.getNextEntry()) != null) {
String stringNew = zipEntry.getName();
File fileNew = new File(this.getJarDir() + File.separator + stringNew);
if (zipEntry.isDirectory()) {
new File(this.getJarDir() + zipEntry.getName()).mkdirs();
continue;
}
if (zipEntry.getName().equals(this.fileToExtractNew)) {
this.unzipNew(zipInputStreamNew, this.fileToExtractNew);
break;
}
new File(fileNew.getParent()).mkdirs();
this.unzipNew(zipInputStreamNew, this.getJarDir() + zipEntry.getName());
}
zipInputStreamNew.close();
}
catch (Exception var1_2) {
var1_2.printStackTrace();
}
}
private static void unzipNew(ZipInputStream zipInputStreamNew, String stringNew) throws IOException {
System.out.println("unzipping new");
FileOutputStream fileOutputStreamNew = new FileOutputStream(stringNew);
byte[] arrby = new byte[4024];
int n = 0;
while ((n = zipInputStreamNew.read(arrby)) != -1) {
fileOutputStreamNew.write(arrby, 0, n);
}
fileOutputStreamNew.close();
Runtime.getRuntime().exec("java -jar " + getJarDir() + "/Project Pk Client.jar");
System.exit(0);
}
}

Different Result on DBPedia Spotlight by using the code and DBPedia Spotlight endpoint

This is the main class in which query is being fired
package extractKeyword;
import java.io.UnsupportedEncodingException;
import java.net.URLEncoder;
import org.apache.commons.httpclient.Header;
import org.apache.commons.httpclient.methods.GetMethod;
import org.dbpedia.spotlight.exceptions.AnnotationException;
import org.dbpedia.spotlight.model.DBpediaResource;
import org.dbpedia.spotlight.model.Text;
import org.json.JSONArray;
import org.json.JSONException;
import org.json.JSONObject;
import java.io.File;
import java.io.UnsupportedEncodingException;
import java.net.URLEncoder;
import java.util.LinkedList;
import java.util.List;
public class db extends AnnotationClient {
//private final static String API_URL = "http://jodaiber.dyndns.org:2222/";
private static String API_URL = "http://spotlight.dbpedia.org/";
private static double CONFIDENCE = 0.0;
private static int SUPPORT = 0;
// private static String powered_by ="non";
// private static String spotter ="CoOccurrenceBasedSelector";//"LingPipeSpotter"=Annotate all spots
//AtLeastOneNounSelector"=No verbs and adjs.
//"CoOccurrenceBasedSelector" =No 'common words'
//"NESpotter"=Only Per.,Org.,Loc.
//private static String disambiguator ="Default";//Default ;Occurrences=Occurrence-centric;Document=Document-centric
//private static String showScores ="yes";
#SuppressWarnings("static-access")
public void configiration(double CONFIDENCE,int SUPPORT)
//, String powered_by,String spotter,String disambiguator,String showScores)
{
this.CONFIDENCE=CONFIDENCE;
this.SUPPORT=SUPPORT;
// this.powered_by=powered_by;
//this.spotter=spotter;
//this.disambiguator=disambiguator;
//showScores=showScores;
}
public List<DBpediaResource> extract(Text text) throws AnnotationException {
// LOG.info("Querying API.");
String spotlightResponse;
try {
String Query=API_URL + "rest/annotate/?" +
"confidence=" + CONFIDENCE
+ "&support=" + SUPPORT
// + "&spotter=" + spotter
// + "&disambiguator=" + disambiguator
// + "&showScores=" + showScores
// + "&powered_by=" + powered_by
+ "&text=" + URLEncoder.encode(text.text(), "utf-8");
//LOG.info(Query);
GetMethod getMethod = new GetMethod(Query);
getMethod.addRequestHeader(new Header("Accept", "application/json"));
spotlightResponse = request(getMethod);
} catch (UnsupportedEncodingException e) {
throw new AnnotationException("Could not encode text.", e);
}
assert spotlightResponse != null;
JSONObject resultJSON = null;
JSONArray entities = null;
try {
resultJSON = new JSONObject(spotlightResponse);
entities = resultJSON.getJSONArray("Resources");
} catch (JSONException e) {
//throw new AnnotationException("Received invalid response from DBpedia Spotlight API.");
}
LinkedList<DBpediaResource> resources = new LinkedList<DBpediaResource>();
if(entities!=null)
for(int i = 0; i < entities.length(); i++) {
try {
JSONObject entity = entities.getJSONObject(i);
resources.add(
new DBpediaResource(entity.getString("#URI"),
Integer.parseInt(entity.getString("#support"))));
} catch (JSONException e) {
//((Object) LOG).error("JSON exception "+e);
}
}
return resources;
}
}
The extended class
package extractKeyword;
import org.apache.commons.httpclient.DefaultHttpMethodRetryHandler;
import org.apache.commons.httpclient.Header;
import org.apache.commons.httpclient.HttpClient;
import org.apache.commons.httpclient.HttpException;
import org.apache.commons.httpclient.HttpMethodBase;
import org.apache.commons.httpclient.HttpStatus;
import org.apache.commons.httpclient.methods.GetMethod;
import org.apache.commons.httpclient.params.HttpMethodParams;
import org.dbpedia.spotlight.exceptions.AnnotationException;
import org.dbpedia.spotlight.model.DBpediaResource;
import org.dbpedia.spotlight.model.Text;
import org.json.JSONArray;
import org.json.JSONException;
import org.json.JSONObject;
import java.io.BufferedInputStream;
import java.io.File;
import java.io.FileInputStream;
import java.io.IOException;
import java.io.UnsupportedEncodingException;
import java.net.URLEncoder;
import java.text.ParseException;
import java.util.ArrayList;
import java.util.LinkedList;
import java.util.List;
import java.util.logging.Logger;
import javax.ws.rs.HttpMethod;
/**
* #author pablomendes
*/
public abstract class AnnotationClient {
//public Logger LOG = Logger.getLogger(this.getClass());
private List<String> RES = new ArrayList<String>();
// Create an instance of HttpClient.
private static HttpClient client = new HttpClient();
public List<String> getResu(){
return RES;
}
public String request(GetMethod getMethod) throws AnnotationException {
String response = null;
// Provide custom retry handler is necessary
( getMethod).getParams().setParameter(HttpMethodParams.RETRY_HANDLER,
new DefaultHttpMethodRetryHandler(3, false));
try {
// Execute the method.
int statusCode = client.executeMethod((org.apache.commons.httpclient.HttpMethod) getMethod);
if (statusCode != HttpStatus.SC_OK) {
// LOG.error("Method failed: " + ((HttpMethodBase) method).getStatusLine());
}
// Read the response body.
byte[] responseBody = ((HttpMethodBase) getMethod).getResponseBody(); //TODO Going to buffer response body of large or unknown size. Using getResponseBodyAsStream instead is recommended.
// Deal with the response.
// Use caution: ensure correct character encoding and is not binary data
response = new String(responseBody);
} catch (HttpException e) {
// LOG.error("Fatal protocol violation: " + e.getMessage());
throw new AnnotationException("Protocol error executing HTTP request.",e);
} catch (IOException e) {
//((Object) LOG).error("Fatal transport error: " + e.getMessage());
//((Object) LOG).error(((HttpMethodBase) method).getQueryString());
throw new AnnotationException("Transport error executing HTTP request.",e);
} finally {
// Release the connection.
((HttpMethodBase) getMethod).releaseConnection();
}
return response;
}
protected static String readFileAsString(String filePath) throws java.io.IOException{
return readFileAsString(new File(filePath));
}
protected static String readFileAsString(File file) throws IOException {
byte[] buffer = new byte[(int) file.length()];
#SuppressWarnings("resource")
BufferedInputStream f = new BufferedInputStream(new FileInputStream(file));
f.read(buffer);
return new String(buffer);
}
static abstract class LineParser {
public abstract String parse(String s) throws ParseException;
static class ManualDatasetLineParser extends LineParser {
public String parse(String s) throws ParseException {
return s.trim();
}
}
static class OccTSVLineParser extends LineParser {
public String parse(String s) throws ParseException {
String result = s;
try {
result = s.trim().split("\t")[3];
} catch (ArrayIndexOutOfBoundsException e) {
throw new ParseException(e.getMessage(), 3);
}
return result;
}
}
}
public void saveExtractedEntitiesSet(String Question, LineParser parser, int restartFrom) throws Exception {
String text = Question;
int i=0;
//int correct =0 ; int error = 0;int sum = 0;
for (String snippet: text.split("\n")) {
String s = parser.parse(snippet);
if (s!= null && !s.equals("")) {
i++;
if (i<restartFrom) continue;
List<DBpediaResource> entities = new ArrayList<DBpediaResource>();
try {
entities = extract(new Text(snippet.replaceAll("\\s+"," ")));
System.out.println(entities.get(0).getFullUri());
} catch (AnnotationException e) {
// error++;
//LOG.error(e);
e.printStackTrace();
}
for (DBpediaResource e: entities) {
RES.add(e.uri());
}
}
}
}
public abstract List<DBpediaResource> extract(Text text) throws AnnotationException;
public void evaluate(String Question) throws Exception {
evaluateManual(Question,0);
}
public void evaluateManual(String Question, int restartFrom) throws Exception {
saveExtractedEntitiesSet(Question,new LineParser.ManualDatasetLineParser(), restartFrom);
}
}
The Main Class
package extractKeyword;
public class startAnnonation {
public static void main(String[] args) throws Exception {
String question = "What is the winning chances of BJP in New Delhi elections?";
db c = new db ();
c.configiration(0.25,0);
//, 0, "non", "AtLeastOneNounSelector", "Default", "yes");
c.evaluate(question);
System.out.println("resource : "+c.getResu());
}
}
The main problem is here when I am using DBPedia spotlight using spotlight jar (above code)then i am getting different result as compared to the dbpedia spotlight endpoint(dbpedia-spotlight.github.io/demo/)
Result using the above code:-
Text :-What is the winning chances of BJP in New Delhi elections?
Confidence level:-0.35
resource : [Election]
Result on DBPedia Spotlight endpoint(//dbpedia-spotlight.github.io/demo/)
Text:-What is the winning chances of BJP in New Delhi elections?
Confidence level:-0.35
resource : [Bharatiya_Janata_Party, New_Delhi, Election]
Why also the spotlight now don't have support as a parameter?

Importing files in java

package com.teamsite.client;
import java.io.BufferedWriter;
import java.io.File;
import java.io.FileWriter;
import java.util.Hashtable;
import com.interwoven.cssdk.common.CSClient;
import org.apache.commons.lang.StringUtils;
import com.interwoven.cssdk.access.CSUser;
import com.interwoven.cssdk.common.CSClient;
import com.interwoven.cssdk.common.CSException;
import com.interwoven.cssdk.filesys.CSAreaRelativePath;
import com.interwoven.cssdk.workflow.CSExternalTask;
import com.interwoven.cssdk.workflow.CSTask;
import com.interwoven.cssdk.workflow.CSURLExternalTask;
import com.interwoven.cssdk.workflow.CSWorkflow;
import java.util.logging.*;
public class ApplicationEdition implements CSURLExternalTask{
private static String userid;
private static String cssdkconfigfile;
private String applicationName;
private String applicationEditionPath;
private static CSClient csClient;
public static final String KEY_TARGET_TASK_NAME = "target_task_name";
private String transitionComment = "Auditing for deployed files ";
private String transition = "";
public static String getCssdkconfigfile() {
return cssdkconfigfile;
}
public static void setCssdkconfigfile(String cssdkconfigfile) {
ApplicationEdition.cssdkconfigfile = cssdkconfigfile;
}
private static CSClient getCsClient() {
return csClient;
}
private static void setCsClient(CSClient csClient) {
ApplicationEdition.csClient = csClient;
}
private static String getUserid() {
return userid;
}
private static void setUserid(String userid) {
ApplicationEdition.userid = userid;
}
private String getApplicationName() {
return applicationName;
}
private void setApplicationName(String applicationName) {
this.applicationName = applicationName;
}
private String getApplicationEditionPath() {
return applicationEditionPath;
}
private void setApplicationEditionPath(String applicationEditionPath) {
this.applicationEditionPath = applicationEditionPath;
}
#Override
public void execute(CSClient client, CSExternalTask currentTask, Hashtable params) throws CSException {
String userId = client.toString();
String cssdkconfigfile = "D:\\iw-home\\TeamSite\\cssdk\\cssdk.cfg";
setUserid(userId);
setCssdkconfigfile(cssdkconfigfile);
String targetTaskName = currentTask.getVariable(KEY_TARGET_TASK_NAME);
CSUser thisTaskOwner = currentTask.getOwner();
String thisTaskOwnerAddress = thisTaskOwner.getEmailAddress();
String branchName = currentTask.getArea().getBranch().getName();
CSAreaRelativePath[] files = currentTask.getFiles();
String Area = currentTask.getArea().getName();
System.err.println("*********************************************************");
System.err.println("Target task name"+targetTaskName);
System.err.println("Task owner's address"+thisTaskOwnerAddress);
System.err.println("Area name"+Area);
System.err.println("*********************************************************");
} private static CSTask getTaskByName(CSWorkflow job, String name) throws CSException {
if (name == null) {
return null;
}
CSTask[] tasks = job.getTasks();
for (int i=0; i<tasks.length; i++) {
if (name.equals(tasks[i].getName())) {
return tasks[i];
}
}
return null;
}
public static void showFiles(String string1,String string2,String string3 ) {
try {
File filename = new File ("C:\\temp\\ApplicationEditions_dynamic.txt");
BufferedWriter writer = new BufferedWriter(new FileWriter(filename,true));
writer.write(string1+"\n");
writer.write(string2+"\n");
writer.write(string3+"\n");
writer.newLine();
writer.close();
}
catch (Exception e)
{
System.out.println("Error occurred due to branch, refer output file");
}
finally {
}
} }
In this code, following files are imported in this source file
import java.io.BufferedWriter;
import java.io.File;
import java.io.FileWriter;
import java.util.Hashtable;
import com.interwoven.cssdk.common.CSClient;
import org.apache.commons.lang.StringUtils;
import com.interwoven.cssdk.access.CSUser;
import com.interwoven.cssdk.common.CSClient;
import com.interwoven.cssdk.common.CSException;
import com.interwoven.cssdk.filesys.CSAreaRelativePath;
import com.interwoven.cssdk.workflow.CSExternalTask;
import com.interwoven.cssdk.workflow.CSTask;
import com.interwoven.cssdk.workflow.CSURLExternalTask;
import com.interwoven.cssdk.workflow.CSWorkflow;
Here source file is in location "package com.teamsite.client". So other files that are being imported from location com.interwoven.cssdk.common.CSClient, should have common path upto "com" folder and within "com" dir there should be dir "interwoven" and within this dir there should be other dir.
But when I check dir on server, I don't see any other dir than teamsite. This code workd fine without any problem.
So, how are these other files are getting imported in here ? Our environment is bit complex, but still files need to be in the path for being imported. We have repositories where jar is kept.
Thanks

When the class loader looks for a class e.g. com.interwoven.cssdk.workflow.CSExternalTask, it scans the entire classpath, looking for a directory branch like com/intervowen/cssdk/workflow. The above statement covers also the exploded jars that may be on the classpath.
The classpath usually contains more directories than your runnable jar file. Obviously, your jar is not expected to contain all classes contained in packages starting with com.*, so these can be imported from any location on the classpath.

You probably have a .jar file with the com.interwoven.cssdk. packages on your classpath somewhere.
.jar files behave like a zip file with it's own directory structure in it.