Unit testing Web API consumer module - java

My code is the consumer of an API (www.abc.com/public/news/apple.json). I get a json array in return which I then parse and populate in my own data structure. the code responsible for doing this is:
public Map<String,List<NewsItem>> populateNewsArray() throws Exception
{
url = domain + newsApiStr;
InputStream stream = getNews(url, true);
//jackson library object mapper
ObjectMapper mapper = new ObjectMapper();
//NewsApiObject class implements the structure of the json array returned.
List<NewsApiObject> mappedData = mapper.readValue(stream, NewsApiObject.class));
//populate the properties in a HashMap.
//return HashMap
}
public InputStream getNews(String request, boolean bulk) throws Exception
{
URL url = new URL(request);
connection = (HttpURLConnection) url.openConnection();
connection.setDoOutput(true);
connection.setInstanceFollowRedirects(false);
connection.setRequestMethod("GET");
connection.setRequestProperty("Content-Type", "text/plain");
connection.setRequestProperty("charset", "utf-8");
connection.connect();
return connection.getInputStream();
}
As you can see I am not the controller of the api, only the consumer. It is said that in unit tests, one is not suppose to make http requests. In this scenario, how can I unit test the populateNewsArray() function to see if the object mapping was correct (without any exceptions) and a valid hashmap was returned?

You should extract getNews() into a separate interface, e.g. NewsReader (although the word Reader has a specific meaning in the JDK, I like the name...)
public interface NewsReader {
InputStream getNews(String request, boolean bulk) throws Exception
}
Then implement that interface with using HttpURLConnection as per your code and update your code to allow injection of that particular interface. Then, if you need to test how your code handles an InputStream, you can create a mock of NewsReader which returns a InputStream with well-known content.
Remember to aim for high cohesion: your class shouldn't be an HTTP client and a stream parser.

I would create a subclass and overwrite the method getNews(...). In the subclass you then may return an InputStream for your test.
Since you should not depenend on some external file in a unit test and in order to get a better testable design I'd also change the getNews(...) method to return some kind of value which can be further processed by the mapper.

Related

How can I extend HttpURLConnection to support other methods?

I want to build a protocol on top of http (like WebDAV). So it makes sense to use HttpURLConnection because it contains all basic methods, response codes and so on. Unfortunately it prevents setting other methods like:
HttpURLConnection con = URL.openConnection( "http://abc.def/someServlet" );
con.setRequestMethod( "MyMethod" );
throws a ProtocolException Invalid method MyMethod.
So I extended it and overwrote "setRequestMethod( String )" to accept "MyMethod". This does not work, because URL only returns a HttpURLConnection. Is there any way to use HttpURLConnection to accept other methods?
Please don't, I tell you this because I did the same error when I started programming in java and it did not pay off.
Ask yourself this question, will you ever pass the child class to a function / method / constructor which accepts the parent class?
For example will you put in a collection a mix of HttpURLConnection and MyServiceURLConnection.
If not so, you will be better off with a brand new class which uses a HttpURLConnection to provide the low level operations.
You might wish to consider RESTful interface. You still use http werbs but you are level up, don't see it, can do any interface, use factories, like:
public String readFileAsString(String url) {
String result = null;
try {
Client client = Client.create();
WebResource webResource = client.resource(url);
ClientResponse response = webResource.type("text/plain").get(ClientResponse.class);
result= response.getEntity(String.class);
response.close();
} catch (Exception e) {
e.printStackTrace();
result = "";
}
return result;
}
url dosent return HTTPURLConnection , it returns URLConnection because you didnt cast it
so you cannot override the HTTPURlConnection method setRequestMethod()

Getting InputStream with RestTemplate

I am using URL class to read an InputStream from it. Is there any way I can use RestTemplate for this?
InputStream input = new URL(url).openStream();
JsonReader reader = new JsonReader(new InputStreamReader(input, StandardCharsets.UTF_8.displayName()));
How can I get InputStream with RestTemplate instead of using URL?
The previous answers are not wrong, but they don't go into the depth that I like to see. There are cases when dealing with low level InputStream is not only desirable, but necessary, the most common example being streaming a large file from source (some web server) to destination (a database). If you try to use a ByteArrayInputStream, you will be, not so surprisingly, greeted with OutOfMemoryError. Yes, you can roll your own HTTP client code, but you'll have to deal with erroneous response codes, response converters etc. If you are already using Spring, looking to RestTemplate is a natural choice.
As of this writing, spring-web:5.0.2.RELEASE has a ResourceHttpMessageConverter that has a boolean supportsReadStreaming, which if set, and the response type is InputStreamResource, returns InputStreamResource; otherwise it returns a ByteArrayResource. So clearly, you're not the only one that asked for streaming support.
However, there is a problem: RestTemplate closes the response soon after the HttpMessageConverter runs. Thus, even if you asked for InputStreamResource, and got it, it's no good, because the response stream has been closed. I think this is a design flaw that they overlooked; it should've been dependent on the response type. So unfortunately, for reading, you must consume the response fully; you can't pass it around if using RestTemplate.
Writing is no problem though. If you want to stream an InputStream, ResourceHttpMessageConverter will do it for you. Under the hood, it uses org.springframework.util.StreamUtils to write 4096 bytes at a time from the InputStream to the OutputStream.
Some of the HttpMessageConverter support all media types, so depending on your requirement, you may have to remove the default ones from RestTemplate, and set the ones you need, being mindful of their relative ordering.
Last but not the least, implementations of ClientHttpRequestFactory has a boolean bufferRequestBody that you can, and should, set to false if you are uploading a large stream. Otherwise, you know, OutOfMemoryError. As of this writing, SimpleClientHttpRequestFactory (JDK client) and HttpComponentsClientHttpRequestFactory (Apache HTTP client) support this feature, but not OkHttp3ClientHttpRequestFactory. Again, design oversight.
Edit:
Filed ticket SPR-16885.
Spring has a org.springframework.http.converter.ResourceHttpMessageConverter. It converts Spring's org.springframework.core.io.Resource class.
That Resource class encapsulates a InputStream, which you can obtain via someResource.getInputStream().
Putting this all together, you can actually get an InputStream via RestTemplate out-of-the-box by specifying Resource.class as your RestTemplate invocation's response type.
Here is an example using one of RestTemplate's exchange(..) methods:
import org.springframework.web.client.RestTemplate;
import org.springframework.http.HttpMethod;
import org.springframework.core.io.Resource;
ResponseEntity<Resource> responseEntity = restTemplate.exchange( someUrlString, HttpMethod.GET, someHttpEntity, Resource.class );
InputStream responseInputStream;
try {
responseInputStream = responseEntity.getBody().getInputStream();
}
catch (IOException e) {
throw new RuntimeException(e);
}
// use responseInputStream
You should not get the InputStream directly. RestTemplate is meant to encapsulate processing the response (and request) content. Its strength is handling all the IO and handing you a ready-to-go Java object.
One of RestTemplate's original authors, Brian Clozel, has stated:
RestTemplate is not meant to stream the response body; its contract
doesn't allow it, and it's been around for so long that changing such
a basic part of its behavior cannot be done without disrupting many
applications.
You'll need to register appropriate HttpMessageConverter objects. Those will have access to the response's InputStream, through an HttpInputMessage object.
As Abdull suggests, Spring does come with an HttpMessageConverter implementation for Resource which itself wraps an InputStream, ResourceHttpMessageConverter. It doesn't support all Resource types, but since you should be programming to interfaces anyway, you should just use the superinterface Resource.
The current implementation (4.3.5), will return a ByteArrayResource with the content of the response stream copied to a new ByteArrayInputStream which you can access.
You don't have to close the stream. The RestTemplate takes care of that for you. (This is unfortunate if you try to use a InputStreamResource, another type supported by the ResourceHttpMessageConverter, because it wraps the underlying response's InputStream but is closed before it can be exposed to your client code.)
I encountered the same issue and solved it by extending RestTemplate and closing the connection only after the stream is read.
you can see the code here: https://github.com/ItamarBenjamin/stream-rest-template
Thanks to Abhijit Sarkar's answer for leading the way.
I needed to download a heavy JSON stream and break it into small streamable manageable pieces of data.
The JSON is composed of objects that have big properties: such big properties can be serialized to a file, and thus removed from the unmarshalled JSON object.
Another use case is to download a JSON stream object by object, process it like a map/reduce algorythm and produce a single output without having to load the whole stream in memory.
Yet another use case is to read a big JSON file and only pick a few objects based on a condition, while unmarshalling to Plain Old Java Objects.
Here is an example: we'd like to stream a very huge JSON file that is an array, and we'd like to retrieve only the first object in the array.
Given this big file on a server, available at http://example.org/testings.json :
[
{ "property1": "value1", "property2": "value2", "property3": "value3" },
{ "property1": "value1", "property2": "value2", "property3": "value3" },
... 1446481 objects => a file of 104 MB => take quite long to download...
]
Each row of this JSON array can be parsed as this object:
#lombok.Data
public class Testing {
String property1;
String property2;
String property3;
}
You need this class make the parsing code reusable:
import com.fasterxml.jackson.core.JsonParser;
import java.io.IOException;
#FunctionalInterface
public interface JsonStreamer<R> {
/**
* Parse the given JSON stream, process it, and optionally return an object.<br>
* The returned object can represent a downsized parsed version of the stream, or the result of a map/reduce processing, or null...
*
* #param jsonParser the parser to use while streaming JSON for processing
* #return the optional result of the process (can be {#link Void} if processing returns nothing)
* #throws IOException on streaming problem (you are also strongly encouraged to throw HttpMessageNotReadableException on parsing error)
*/
R stream(JsonParser jsonParser) throws IOException;
}
And this class to parse:
import com.fasterxml.jackson.core.JsonFactory;
import com.fasterxml.jackson.core.JsonParser;
import lombok.AllArgsConstructor;
import org.springframework.http.HttpInputMessage;
import org.springframework.http.HttpOutputMessage;
import org.springframework.http.MediaType;
import org.springframework.http.converter.HttpMessageConverter;
import java.io.IOException;
import java.util.Collections;
import java.util.List;
#AllArgsConstructor
public class StreamingHttpMessageConverter<R> implements HttpMessageConverter<R> {
private final JsonFactory factory;
private final JsonStreamer<R> jsonStreamer;
#Override
public boolean canRead(Class<?> clazz, MediaType mediaType) {
return MediaType.APPLICATION_JSON.isCompatibleWith(mediaType);
}
#Override
public boolean canWrite(Class<?> clazz, MediaType mediaType) {
return false; // We only support reading from an InputStream
}
#Override
public List<MediaType> getSupportedMediaTypes() {
return Collections.singletonList(MediaType.APPLICATION_JSON);
}
#Override
public R read(Class<? extends R> clazz, HttpInputMessage inputMessage) throws IOException {
try (InputStream inputStream = inputMessage.getBody();
JsonParser parser = factory.createParser(inputStream)) {
return jsonStreamer.stream(parser);
}
}
#Override
public void write(R result, MediaType contentType, HttpOutputMessage outputMessage) {
throw new UnsupportedOperationException();
}
}
Then, here is the code to use to stream the HTTP response, parse the JSON array and return only the first unmarshalled object:
// You should #Autowire these:
JsonFactory jsonFactory = new JsonFactory();
ObjectMapper objectMapper = new ObjectMapper();
RestTemplateBuilder restTemplateBuilder = new RestTemplateBuilder();
// If detectRequestFactory true (default): HttpComponentsClientHttpRequestFactory will be used and it will consume the entire HTTP response, even if we close the stream early
// If detectRequestFactory false: SimpleClientHttpRequestFactory will be used and it will close the connection as soon as we ask it to
RestTemplate restTemplate = restTemplateBuilder.detectRequestFactory(false).messageConverters(
new StreamingHttpMessageConverter<>(jsonFactory, jsonParser -> {
// While you use a low-level JsonParser to not load everything in memory at once,
// you can still profit from smaller object mapping with the ObjectMapper
if (!jsonParser.isClosed() && jsonParser.nextToken() == JsonToken.START_ARRAY) {
if (!jsonParser.isClosed() && jsonParser.nextToken() == JsonToken.START_OBJECT) {
return objectMapper.readValue(jsonParser, Testing.class);
}
}
return null;
})
).build();
final Testing firstTesting = restTemplate.getForObject("http://example.org/testings.json", Testing.class);
log.debug("First testing object: {}", firstTesting);
You can pass in your own response extractor. Here is an example where I write out the json to disk in a streaming fashion -
RestTemplate restTemplate = new RestTemplateBuilder().basicAuthentication("user", "their_password" ).build();
int responseSize = restTemplate.execute(uri,
HttpMethod.POST,
(ClientHttpRequest requestCallback) -> {
requestCallback.getHeaders().setContentType(MediaType.APPLICATION_JSON);
requestCallback.getBody().write(body.getBytes());
},
responseExtractor -> {
FileOutputStream fos = new FileOutputStream(new File("out.json"));
return StreamUtils.copy(responseExtractor.getBody(), fos);
}
)
Very simple, yet efficient solution would be using ResponseExtractor. It's especially useful when you want to operate on very large InputStream and your RAM is limited.
Here is how you should be implementing it:
public void consumerInputStreamWithoutBuffering(String url, Consumer<InputStream> streamConsumer) throws IOException {
final ResponseExtractor responseExtractor =
(ClientHttpResponse clientHttpResponse) -> {
streamConsumer.accept(clientHttpResponse.getBody());
return null;
};
restTemplate.execute(url, HttpMethod.GET, null, responseExtractor);
}
And then, invoke the method anywhere you need:
Consumer<InputStream> doWhileDownloading = inputStream -> {
//Use inputStream for your business logic...
};
consumerInputStreamWithoutBuffering("https://localhost.com/download", doWhileDownloading);
Please, be aware of the following common pitfall:
public InputStream getInputStreamFromResponse(String url) throws IOException {
final ResponseExtractor<InputStream> responseExtractor =
clientHttpResponse -> clientHttpResponse.getBody();
return restTemplate.execute(url, HttpMethod.GET, null, responseExtractor);
}
Here InputStream will be closed before you can access it
I solve it by doing that.
I hope it will help you all.
#GetMapping("largeFile")
public ResponseEntity<InputStreamResource> downloadLargeFile(
#RequestParam("fileName") String fileName
) throws IOException {
RestTemplate restTemplate = new RestTemplate();
// Optional Accept header
RequestCallback requestCallback = request -> request.getHeaders()
.setAccept(Arrays.asList(MediaType.APPLICATION_OCTET_STREAM, MediaType.ALL));
// Streams the response instead of loading it all in memory
ResponseExtractor<InputStreamResource> responseExtractor = response -> {
// Here I write the response to a file but do what you like
Path path = Paths.get("tmp/" + fileName);
Files.copy(response.getBody(), path, StandardCopyOption.REPLACE_EXISTING);
return new InputStreamResource(new FileInputStream(String.format("tmp/%s", fileName)));
};
InputStreamResource response = restTemplate.execute(
String.format("http://%s:%s/file/largeFileRestTemplate?fileName=%s", host, "9091", fileName),
HttpMethod.GET,
requestCallback,
responseExtractor
);
return ResponseEntity
.ok()
.header(HttpHeaders.CONTENT_DISPOSITION, String.format("attachment; filename=%s", fileName))
.body(response);
}
As a variant you can consume response as bytes and than convert to stream
byte data[] = restTemplate.execute(link, HttpMethod.GET, null, new BinaryFileExtractor());
return new ByteArrayInputStream(data);
Extractor is
public class BinaryFileExtractor implements ResponseExtractor<byte[]> {
#Override
public byte[] extractData(ClientHttpResponse response) throws IOException {
return ByteStreams.toByteArray(response.getBody());
}
}

Send and receive JSON to REST WebService in Jersey Java

I am new to Jersey Java REST WebService framework. I am trying to write a service method which consumes and produces JSON. My service code is below. It is simplest code, just for studying purpose.
#Path("/myresource")
public class MyResource {
#Path("/sendReceiveJson")
#GET
#Produces(MediaType.APPLICATION_JSON)
#Consumes(MediaType.APPLICATION_JSON)
public String sendReceiveJson(String name)
{
System.out.println("Value in name: " + name);
return "{\"serviceName\": \"Mr.Server\"}";
}
}
And following is JerseyClient code.
public class Program {
public static void main(String[] args) throws Exception{
String urlString="http://localhost:8080/MyWebService/webresources/myresource/sendReceiveJson";
URL url=new URL(urlString);
URLConnection connection=url.openConnection();
connection.setDoOutput(true);
OutputStreamWriter out = new OutputStreamWriter(connection.getOutputStream());
out.write("{\"clientName\": \"Mr.Client\"}");
out.close();
BufferedReader in = new BufferedReader(new InputStreamReader(connection.getInputStream()));
String decodedString;
while ((decodedString = in.readLine()) != null) {
System.out.println(decodedString);
}
in.close();
}
}
But when i run service and then client, i am unable to send/receive JSON data. I get Exception at connection.getInputStream() which is
Server returned HTTP response code: 405 for URL: http://localhost:8080/hellointernet/webresources/myresource/sendReceiveJson
at sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1625)
Please guide me, what needs to correct, or whether i am in wrong direction.
Your resource method is annotated as #GET which means any input data would have to be query string parameters.
In this context #Consumes(MediaType.APPLICATION_JSON) doesn't make a lot of sense as only APPLICATION_FORM_URLENCODED is supported via GET.
When you client calls setDoOutput(true) it probably switches your HTTP call to a POST hence causing the 405 Method Not Allowed.
If you want to consume JSON you should change your #GET annotation with #POST instead. Your client call should then work if it's indeed a POST. You can specify it with the following method:
HttpURLConnection httpCon = (HttpURLConnection) url.openConnection();
httpCon.setDoOutput(true);
httpCon.setRequestMethod("POST");
This API is pretty low level though, so I'd highly recommend you use Jersey's Client API instead. See https://jersey.java.net/documentation/1.17/client-api.html

Multiple #GET for different MIME - how to consume this with plain HttpURLConnection()

I just realized that it is possible to define something like this in my RESTful resource .java file:
#GET
#Produces("text/plain")
public String getPlainTextHello() { ... }
#GET
#Produces("application/json")
public String getJSONHello() { ... }
Isn't that fantastic? But wait the moment....
PROBLEM
I am consuming my API with simple client. Something like this code with help of HttpURLConnection:
URL obj = new URL("http://some.url/res/hello");
HttpURLConnection conn = (HttpURLConnection) obj.openConnection();
conn.setRequestMethod("GET");
... /* get response ... conn.getInputStream() */
How the server 'know' which one method call to serve the client?
Regards.
First of all you should consider using the same method for different types of "produces":
#GET
#Produces({ "application/xml", "text/plain"})
public String getHello() { ... }
The different types of "produces" could be handled by JAXB (in case the response is an object...).
You can define the client side "accept" mime type by using:
String uri =
"http://localhost:8080/hello/";
URL url = new URL(uri);
HttpURLConnection connection =
(HttpURLConnection) url.openConnection();
connection.setRequestMethod("GET");
connection.setRequestProperty("Accept", "application/json");
This question provides more insights (and other client side frameworks) related with this problem: REST. Jersey. How to programmatically choose what type to return: JSON or XML?
You'd probably want a generic function to do all the common work, and then simply pass this work to the response specific functions you outlined.
getHello(String responseType)
{
// do all your work you'd end up doing in both functions
switch (responseType):
case ("json") {
return getJSONHello(work);
}
case ("text") {
return getPlainTextHello(work);
}
}
I made some more checks on this and what works for me is just settings Accept:
...
conn.setRequestMethod("GET");
conn.setRequestProperty("Accept", mime);
...
where mime is "text/plain" or "application/json". This way my server calls one of the GET function.
Anyway I am confused why most answers suggest to use a common one function to serve a #GET and check for type inside this function...

Parse local JSON file with RestTemplate

I would like to parse a local JSON file and marshal it into models using RestTemplate, but can't tell if this is possible.
I'm trying to pre-populate a database on an Android app that is using RestTemplate for syncing with the server. Rather than parsing the local JSON on my own, I thought, why not use RestTemplate? It's made exactly for parsing JSON into models.
But...I can't tell from the docs if there is any way to do this. There is the MappingJacksonHttpMessageConverter class which appears to convert the server's http response into a model...but is there any way to hack that to work with a local file? I tried, but kept getting deeper and deeper down the rabbit hole with no luck.
Figured this out. Instead of using RestTemplate, you can just use Jackson directly. There is no reason RestTemplate needs to be involved in this. It's very simple.
try {
ObjectMapper mapper = new ObjectMapper();
InputStream jsonFileStream = context.getAssets().open("categories.json");
Category[] categories = (Category[]) mapper.readValue(jsonFileStream, Category[].class);
Log.d(tag, "Found " + String.valueOf(categories.length) + " categories!!");
} catch (Exception e){
Log.e(tag, "Exception", e);
}
Yes, I think it is possible(with MappingJacksonHttpMessageConverter).
MappingJacksonHttpMessageConverter has method read() which takes two parameters: Class and HttpInputMessage
MappingJacksonHttpMessageConverter converter = new MappingJacksonHttpMessageConverter();
YourClazz obj = (YourClazz) converter.read(YourClazz.class, new MyHttpInputMessage(myJsonString));
With this method you can read single object from single json message, but YourClazz can be some collection.
Next, You have to create you own HttpInputMessage implementation, in this example it expected json as string but You probably can pass stream to your json file.
public class MyHttpInputMessage implements HttpInputMessage {
private String jsonString;
public MyHttpInputMessage(String jsonString) {
this.jsonString = jsonString;
}
public HttpHeaders getHeaders() {
// no headers needed
return null;
}
public InputStream getBody() throws IOException {
InputStream is = new ByteArrayInputStream(
jsonString.getBytes("UTF-8"));
return is;
}
}
PS. You can publish your app with database

Categories