Related
I am using URL class to read an InputStream from it. Is there any way I can use RestTemplate for this?
InputStream input = new URL(url).openStream();
JsonReader reader = new JsonReader(new InputStreamReader(input, StandardCharsets.UTF_8.displayName()));
How can I get InputStream with RestTemplate instead of using URL?
The previous answers are not wrong, but they don't go into the depth that I like to see. There are cases when dealing with low level InputStream is not only desirable, but necessary, the most common example being streaming a large file from source (some web server) to destination (a database). If you try to use a ByteArrayInputStream, you will be, not so surprisingly, greeted with OutOfMemoryError. Yes, you can roll your own HTTP client code, but you'll have to deal with erroneous response codes, response converters etc. If you are already using Spring, looking to RestTemplate is a natural choice.
As of this writing, spring-web:5.0.2.RELEASE has a ResourceHttpMessageConverter that has a boolean supportsReadStreaming, which if set, and the response type is InputStreamResource, returns InputStreamResource; otherwise it returns a ByteArrayResource. So clearly, you're not the only one that asked for streaming support.
However, there is a problem: RestTemplate closes the response soon after the HttpMessageConverter runs. Thus, even if you asked for InputStreamResource, and got it, it's no good, because the response stream has been closed. I think this is a design flaw that they overlooked; it should've been dependent on the response type. So unfortunately, for reading, you must consume the response fully; you can't pass it around if using RestTemplate.
Writing is no problem though. If you want to stream an InputStream, ResourceHttpMessageConverter will do it for you. Under the hood, it uses org.springframework.util.StreamUtils to write 4096 bytes at a time from the InputStream to the OutputStream.
Some of the HttpMessageConverter support all media types, so depending on your requirement, you may have to remove the default ones from RestTemplate, and set the ones you need, being mindful of their relative ordering.
Last but not the least, implementations of ClientHttpRequestFactory has a boolean bufferRequestBody that you can, and should, set to false if you are uploading a large stream. Otherwise, you know, OutOfMemoryError. As of this writing, SimpleClientHttpRequestFactory (JDK client) and HttpComponentsClientHttpRequestFactory (Apache HTTP client) support this feature, but not OkHttp3ClientHttpRequestFactory. Again, design oversight.
Edit:
Filed ticket SPR-16885.
Spring has a org.springframework.http.converter.ResourceHttpMessageConverter. It converts Spring's org.springframework.core.io.Resource class.
That Resource class encapsulates a InputStream, which you can obtain via someResource.getInputStream().
Putting this all together, you can actually get an InputStream via RestTemplate out-of-the-box by specifying Resource.class as your RestTemplate invocation's response type.
Here is an example using one of RestTemplate's exchange(..) methods:
import org.springframework.web.client.RestTemplate;
import org.springframework.http.HttpMethod;
import org.springframework.core.io.Resource;
ResponseEntity<Resource> responseEntity = restTemplate.exchange( someUrlString, HttpMethod.GET, someHttpEntity, Resource.class );
InputStream responseInputStream;
try {
responseInputStream = responseEntity.getBody().getInputStream();
}
catch (IOException e) {
throw new RuntimeException(e);
}
// use responseInputStream
You should not get the InputStream directly. RestTemplate is meant to encapsulate processing the response (and request) content. Its strength is handling all the IO and handing you a ready-to-go Java object.
One of RestTemplate's original authors, Brian Clozel, has stated:
RestTemplate is not meant to stream the response body; its contract
doesn't allow it, and it's been around for so long that changing such
a basic part of its behavior cannot be done without disrupting many
applications.
You'll need to register appropriate HttpMessageConverter objects. Those will have access to the response's InputStream, through an HttpInputMessage object.
As Abdull suggests, Spring does come with an HttpMessageConverter implementation for Resource which itself wraps an InputStream, ResourceHttpMessageConverter. It doesn't support all Resource types, but since you should be programming to interfaces anyway, you should just use the superinterface Resource.
The current implementation (4.3.5), will return a ByteArrayResource with the content of the response stream copied to a new ByteArrayInputStream which you can access.
You don't have to close the stream. The RestTemplate takes care of that for you. (This is unfortunate if you try to use a InputStreamResource, another type supported by the ResourceHttpMessageConverter, because it wraps the underlying response's InputStream but is closed before it can be exposed to your client code.)
I encountered the same issue and solved it by extending RestTemplate and closing the connection only after the stream is read.
you can see the code here: https://github.com/ItamarBenjamin/stream-rest-template
Thanks to Abhijit Sarkar's answer for leading the way.
I needed to download a heavy JSON stream and break it into small streamable manageable pieces of data.
The JSON is composed of objects that have big properties: such big properties can be serialized to a file, and thus removed from the unmarshalled JSON object.
Another use case is to download a JSON stream object by object, process it like a map/reduce algorythm and produce a single output without having to load the whole stream in memory.
Yet another use case is to read a big JSON file and only pick a few objects based on a condition, while unmarshalling to Plain Old Java Objects.
Here is an example: we'd like to stream a very huge JSON file that is an array, and we'd like to retrieve only the first object in the array.
Given this big file on a server, available at http://example.org/testings.json :
[
{ "property1": "value1", "property2": "value2", "property3": "value3" },
{ "property1": "value1", "property2": "value2", "property3": "value3" },
... 1446481 objects => a file of 104 MB => take quite long to download...
]
Each row of this JSON array can be parsed as this object:
#lombok.Data
public class Testing {
String property1;
String property2;
String property3;
}
You need this class make the parsing code reusable:
import com.fasterxml.jackson.core.JsonParser;
import java.io.IOException;
#FunctionalInterface
public interface JsonStreamer<R> {
/**
* Parse the given JSON stream, process it, and optionally return an object.<br>
* The returned object can represent a downsized parsed version of the stream, or the result of a map/reduce processing, or null...
*
* #param jsonParser the parser to use while streaming JSON for processing
* #return the optional result of the process (can be {#link Void} if processing returns nothing)
* #throws IOException on streaming problem (you are also strongly encouraged to throw HttpMessageNotReadableException on parsing error)
*/
R stream(JsonParser jsonParser) throws IOException;
}
And this class to parse:
import com.fasterxml.jackson.core.JsonFactory;
import com.fasterxml.jackson.core.JsonParser;
import lombok.AllArgsConstructor;
import org.springframework.http.HttpInputMessage;
import org.springframework.http.HttpOutputMessage;
import org.springframework.http.MediaType;
import org.springframework.http.converter.HttpMessageConverter;
import java.io.IOException;
import java.util.Collections;
import java.util.List;
#AllArgsConstructor
public class StreamingHttpMessageConverter<R> implements HttpMessageConverter<R> {
private final JsonFactory factory;
private final JsonStreamer<R> jsonStreamer;
#Override
public boolean canRead(Class<?> clazz, MediaType mediaType) {
return MediaType.APPLICATION_JSON.isCompatibleWith(mediaType);
}
#Override
public boolean canWrite(Class<?> clazz, MediaType mediaType) {
return false; // We only support reading from an InputStream
}
#Override
public List<MediaType> getSupportedMediaTypes() {
return Collections.singletonList(MediaType.APPLICATION_JSON);
}
#Override
public R read(Class<? extends R> clazz, HttpInputMessage inputMessage) throws IOException {
try (InputStream inputStream = inputMessage.getBody();
JsonParser parser = factory.createParser(inputStream)) {
return jsonStreamer.stream(parser);
}
}
#Override
public void write(R result, MediaType contentType, HttpOutputMessage outputMessage) {
throw new UnsupportedOperationException();
}
}
Then, here is the code to use to stream the HTTP response, parse the JSON array and return only the first unmarshalled object:
// You should #Autowire these:
JsonFactory jsonFactory = new JsonFactory();
ObjectMapper objectMapper = new ObjectMapper();
RestTemplateBuilder restTemplateBuilder = new RestTemplateBuilder();
// If detectRequestFactory true (default): HttpComponentsClientHttpRequestFactory will be used and it will consume the entire HTTP response, even if we close the stream early
// If detectRequestFactory false: SimpleClientHttpRequestFactory will be used and it will close the connection as soon as we ask it to
RestTemplate restTemplate = restTemplateBuilder.detectRequestFactory(false).messageConverters(
new StreamingHttpMessageConverter<>(jsonFactory, jsonParser -> {
// While you use a low-level JsonParser to not load everything in memory at once,
// you can still profit from smaller object mapping with the ObjectMapper
if (!jsonParser.isClosed() && jsonParser.nextToken() == JsonToken.START_ARRAY) {
if (!jsonParser.isClosed() && jsonParser.nextToken() == JsonToken.START_OBJECT) {
return objectMapper.readValue(jsonParser, Testing.class);
}
}
return null;
})
).build();
final Testing firstTesting = restTemplate.getForObject("http://example.org/testings.json", Testing.class);
log.debug("First testing object: {}", firstTesting);
You can pass in your own response extractor. Here is an example where I write out the json to disk in a streaming fashion -
RestTemplate restTemplate = new RestTemplateBuilder().basicAuthentication("user", "their_password" ).build();
int responseSize = restTemplate.execute(uri,
HttpMethod.POST,
(ClientHttpRequest requestCallback) -> {
requestCallback.getHeaders().setContentType(MediaType.APPLICATION_JSON);
requestCallback.getBody().write(body.getBytes());
},
responseExtractor -> {
FileOutputStream fos = new FileOutputStream(new File("out.json"));
return StreamUtils.copy(responseExtractor.getBody(), fos);
}
)
Very simple, yet efficient solution would be using ResponseExtractor. It's especially useful when you want to operate on very large InputStream and your RAM is limited.
Here is how you should be implementing it:
public void consumerInputStreamWithoutBuffering(String url, Consumer<InputStream> streamConsumer) throws IOException {
final ResponseExtractor responseExtractor =
(ClientHttpResponse clientHttpResponse) -> {
streamConsumer.accept(clientHttpResponse.getBody());
return null;
};
restTemplate.execute(url, HttpMethod.GET, null, responseExtractor);
}
And then, invoke the method anywhere you need:
Consumer<InputStream> doWhileDownloading = inputStream -> {
//Use inputStream for your business logic...
};
consumerInputStreamWithoutBuffering("https://localhost.com/download", doWhileDownloading);
Please, be aware of the following common pitfall:
public InputStream getInputStreamFromResponse(String url) throws IOException {
final ResponseExtractor<InputStream> responseExtractor =
clientHttpResponse -> clientHttpResponse.getBody();
return restTemplate.execute(url, HttpMethod.GET, null, responseExtractor);
}
Here InputStream will be closed before you can access it
I solve it by doing that.
I hope it will help you all.
#GetMapping("largeFile")
public ResponseEntity<InputStreamResource> downloadLargeFile(
#RequestParam("fileName") String fileName
) throws IOException {
RestTemplate restTemplate = new RestTemplate();
// Optional Accept header
RequestCallback requestCallback = request -> request.getHeaders()
.setAccept(Arrays.asList(MediaType.APPLICATION_OCTET_STREAM, MediaType.ALL));
// Streams the response instead of loading it all in memory
ResponseExtractor<InputStreamResource> responseExtractor = response -> {
// Here I write the response to a file but do what you like
Path path = Paths.get("tmp/" + fileName);
Files.copy(response.getBody(), path, StandardCopyOption.REPLACE_EXISTING);
return new InputStreamResource(new FileInputStream(String.format("tmp/%s", fileName)));
};
InputStreamResource response = restTemplate.execute(
String.format("http://%s:%s/file/largeFileRestTemplate?fileName=%s", host, "9091", fileName),
HttpMethod.GET,
requestCallback,
responseExtractor
);
return ResponseEntity
.ok()
.header(HttpHeaders.CONTENT_DISPOSITION, String.format("attachment; filename=%s", fileName))
.body(response);
}
As a variant you can consume response as bytes and than convert to stream
byte data[] = restTemplate.execute(link, HttpMethod.GET, null, new BinaryFileExtractor());
return new ByteArrayInputStream(data);
Extractor is
public class BinaryFileExtractor implements ResponseExtractor<byte[]> {
#Override
public byte[] extractData(ClientHttpResponse response) throws IOException {
return ByteStreams.toByteArray(response.getBody());
}
}
I want to create an employee information in the system by uploading an image along with employee data. I am able to do it with different rest calls using jersey. But I want to achieve in one rest call.
I provide below the structure. Please help me how to do in this regard.
#POST
#Path("/upload2")
#Consumes({MediaType.MULTIPART_FORM_DATA,MediaType.APPLICATION_XML, MediaType.APPLICATION_JSON})
public Response uploadFileWithData(
#FormDataParam("file") InputStream fileInputStream,
#FormDataParam("file") FormDataContentDisposition contentDispositionHeader,
Employee emp) {
//..... business login
}
Whenever I am trying to do, I get error in Chrome postman. The simple structure of my Employee json is given below.
{
"Name": "John",
"Age": 23,
"Email": "john#gmail.com",
"Adrs": {
"DoorNo": "12-A",
"Street": "Street-11",
"City": "Bangalore",
"Country": "Karnataka"
}
}
However I can do it by making two different call, but I want to achieve in one rest call so that I can receive the file as well as the actual data of the employee.
Request you to help in this regard.
You can't have two Content-Types (well technically that's what we're doing below, but they are separated with each part of the multipart, but the main type is multipart). That's basically what you are expecting with your method. You are expecting mutlipart and json together as the main media type. The Employee data needs to be part of the multipart. So you can add a #FormDataParam("emp") for the Employee.
#FormDataParam("emp") Employee emp) { ...
Here's the class I used for testing
#Path("/multipart")
public class MultipartResource {
#POST
#Path("/upload2")
#Consumes({MediaType.MULTIPART_FORM_DATA})
public Response uploadFileWithData(
#FormDataParam("file") InputStream fileInputStream,
#FormDataParam("file") FormDataContentDisposition cdh,
#FormDataParam("emp") Employee emp) throws Exception{
Image img = ImageIO.read(fileInputStream);
JOptionPane.showMessageDialog(null, new JLabel(new ImageIcon(img)));
System.out.println(cdh.getName());
System.out.println(emp);
return Response.ok("Cool Tools!").build();
}
}
First I just tested with the client API to make sure it works
#Test
public void testGetIt() throws Exception {
final Client client = ClientBuilder.newBuilder()
.register(MultiPartFeature.class)
.build();
WebTarget t = client.target(Main.BASE_URI).path("multipart").path("upload2");
FileDataBodyPart filePart = new FileDataBodyPart("file",
new File("stackoverflow.png"));
// UPDATE: just tested again, and the below code is not needed.
// It's redundant. Using the FileDataBodyPart already sets the
// Content-Disposition information
filePart.setContentDisposition(
FormDataContentDisposition.name("file")
.fileName("stackoverflow.png").build());
String empPartJson
= "{"
+ " \"id\": 1234,"
+ " \"name\": \"Peeskillet\""
+ "}";
MultiPart multipartEntity = new FormDataMultiPart()
.field("emp", empPartJson, MediaType.APPLICATION_JSON_TYPE)
.bodyPart(filePart);
Response response = t.request().post(
Entity.entity(multipartEntity, multipartEntity.getMediaType()));
System.out.println(response.getStatus());
System.out.println(response.readEntity(String.class));
response.close();
}
I just created a simple Employee class with an id and name field for testing. This works perfectly fine. It shows the image, prints the content disposition, and prints the Employee object.
I'm not too familiar with Postman, so I saved that testing for last :-)
It appears to work fine also, as you can see the response "Cool Tools". But if we look at the printed Employee data, we'll see that it's null. Which is weird because with the client API it worked fine.
If we look at the Preview window, we'll see the problem
There's no Content-Type header for the emp body part. You can see in the client API I explicitly set it
MultiPart multipartEntity = new FormDataMultiPart()
.field("emp", empPartJson, MediaType.APPLICATION_JSON_TYPE)
.bodyPart(filePart);
So I guess this is really only part of a full answer. Like I said, I am not familiar with Postman So I don't know how to set Content-Types for individual body parts. The image/png for the image was automatically set for me for the image part (I guess it was just determined by the file extension). If you can figure this out, then the problem should be solved. Please, if you find out how to do this, post it as an answer.
See UPDATE below for solution
And just for completeness...
See here for more about MultiPart with Jersey.
Basic configurations:
Dependency:
<dependency>
<groupId>org.glassfish.jersey.media</groupId>
<artifactId>jersey-media-multipart</artifactId>
<version>${jersey2.version}</version>
</dependency>
Client config:
final Client client = ClientBuilder.newBuilder()
.register(MultiPartFeature.class)
.build();
Server config:
// Create JAX-RS application.
final Application application = new ResourceConfig()
.packages("org.glassfish.jersey.examples.multipart")
.register(MultiPartFeature.class);
If you're having problems with the server configuration, one of the following posts might help
What exactly is the ResourceConfig class in Jersey 2?
152 MULTIPART_FORM_DATA: No injection source found for a parameter of type public javax.ws.rs.core.Response
UPDATE
So as you can see from the Postman client, some clients are unable to set individual parts' Content-Type, this includes the browser, in regards to it's default capabilities when using FormData (js).
We can't expect the client to find away around this, so what we can do, is when receiving the data, explicitly set the Content-Type before deserializing. For example
#POST
#Path("upload2")
#Consumes(MediaType.MULTIPART_FORM_DATA)
public Response uploadFileAndJSON(#FormDataParam("emp") FormDataBodyPart jsonPart,
#FormDataParam("file") FormDataBodyPart bodyPart) {
jsonPart.setMediaType(MediaType.APPLICATION_JSON_TYPE);
Employee emp = jsonPart.getValueAs(Employee.class);
}
It's a little extra work to get the POJO, but it is a better solution than forcing the client to try and find it's own solution.
Another option is to use a String parameter and use whatever JSON library you use to deserialze the String to the POJO (like Jackson ObjectMapper). With the previous option, we just let Jersey handle the deserialization, and it will use the same JSON library it uses for all the other JSON endpoints (which might be preferred).
Asides
There is a conversation in these comments that you may be interested in if you are using a different Connector than the default HttpUrlConnection.
You can access the Image File and data from a form using MULTIPART FORM DATA By using the below code.
#POST
#Path("/UpdateProfile")
#Consumes(value={MediaType.APPLICATION_JSON,MediaType.MULTIPART_FORM_DATA})
#Produces(value={MediaType.APPLICATION_JSON,MediaType.APPLICATION_XML})
public Response updateProfile(
#FormDataParam("file") InputStream fileInputStream,
#FormDataParam("file") FormDataContentDisposition contentDispositionHeader,
#FormDataParam("ProfileInfo") String ProfileInfo,
#FormDataParam("registrationId") String registrationId) {
String filePath= "/filepath/"+contentDispositionHeader.getFileName();
OutputStream outputStream = null;
try {
int read = 0;
byte[] bytes = new byte[1024];
outputStream = new FileOutputStream(new File(filePath));
while ((read = fileInputStream.read(bytes)) != -1) {
outputStream.write(bytes, 0, read);
}
outputStream.flush();
outputStream.close();
} catch (FileNotFoundException e) {
e.printStackTrace();
} catch (IOException e) {
e.printStackTrace();
} finally {
if (outputStream != null) {
try {
outputStream.close();
} catch(Exception ex) {}
}
}
}
When I tried #PaulSamsotha's solution with Jersey client 2.21.1, there was 400 error. It worked when I added following in my client code:
MediaType contentType = MediaType.MULTIPART_FORM_DATA_TYPE;
contentType = Boundary.addBoundary(contentType);
Response response = t.request()
.post(Entity.entity(multipartEntity, contentType));
instead of hardcoded MediaType.MULTIPART_FORM_DATA in POST request call.
The reason this is needed is because when you use a different Connector (like Apache) for the Jersey Client, it is unable to alter outbound headers, which is required to add a boundary to the Content-Type. This limitation is explained in the Jersey Client docs. So if you want to use a different Connector, then you need to manually create the boundary.
Your ApplicationConfig should register the MultiPartFeature.class from the glassfish.jersey.media.. so as to enable file upload
#javax.ws.rs.ApplicationPath(ResourcePath.API_ROOT)
public class ApplicationConfig extends ResourceConfig {
public ApplicationConfig() {
//register the necessary headers files needed from client
register(CORSConfigurationFilter.class);
//The jackson feature and provider is used for object serialization
//between client and server objects in to a json
register(JacksonFeature.class);
register(JacksonProvider.class);
//Glassfish multipart file uploader feature
register(MultiPartFeature.class);
//inject and registered all resources class using the package
//not to be tempered with
packages("com.flexisaf.safhrms.client.resources");
register(RESTRequestFilter.class);
}
I used file upload example from,
http://www.mkyong.com/webservices/jax-rs/file-upload-example-in-jersey/
in my resource class i have below method
#POST
#Path("/upload")
#Consumes(MediaType.MULTIPART_FORM_DATA)
public Response attachupload(#FormDataParam("file") byte[] is,
#FormDataParam("file") FormDataContentDisposition fileDetail,
#FormDataParam("fileName") String flename){
attachService.saveAttachment(flename,is);
}
in my attachService.java i have below method
public void saveAttachment(String flename, byte[] is) {
// TODO Auto-generated method stub
attachmentDao.saveAttachment(flename,is);
}
in Dao i have
attach.setData(is);
attach.setFileName(flename);
in my HBM mapping is like
<property name="data" type="binary" >
<column name="data" />
</property>
This working for all type of files like .PDF,.TXT, .PNG etc.,
The request type is multipart/form-data and what you are sending is essentially form fields that go out as bytes with content boundaries separating different form fields.To send an object representation as form field (string), you can send a serialized form from the client that you can then deserialize on the server.
After all no programming environment object is actually ever traveling on the wire. The programming environment on both side are just doing automatic serialization and deserialization that you can also do. That is the cleanest and programming environment quirks free way to do it.
As an example, here is a javascript client posting to a Jersey example service,
submitFile(){
let data = new FormData();
let account = {
"name": "test account",
"location": "Bangalore"
}
data.append('file', this.file);
data.append("accountKey", "44c85e59-afed-4fb2-884d-b3d85b051c44");
data.append("device", "test001");
data.append("account", JSON.stringify(account));
let url = "http://localhost:9090/sensordb/test/file/multipart/upload";
let config = {
headers: {
'Content-Type': 'multipart/form-data'
}
}
axios.post(url, data, config).then(function(data){
console.log('SUCCESS!!');
console.log(data.data);
}).catch(function(){
console.log('FAILURE!!');
});
},
Here the client is sending a file, 2 form fields (strings) and an account object that has been stringified for transport. here is how the form fields look on the wire,
On the server, you can just deserialize the form fields the way you see fit. To finish this trivial example,
#POST
#Path("/file/multipart/upload")
#Consumes({MediaType.MULTIPART_FORM_DATA})
public Response uploadMultiPart(#Context ContainerRequestContext requestContext,
#FormDataParam("file") InputStream fileInputStream,
#FormDataParam("file") FormDataContentDisposition cdh,
#FormDataParam("accountKey") String accountKey,
#FormDataParam("account") String json) {
System.out.println(cdh.getFileName());
System.out.println(cdh.getName());
System.out.println(accountKey);
try {
Account account = Account.deserialize(json);
System.out.println(account.getLocation());
System.out.println(account.getName());
} catch (Exception e) {
e.printStackTrace();
}
return Response.ok().build();
}
I am working on a project in which I am making a call to one of my servers using RestTemplate which is running a restful service and getting the response back from them.
The response that I will be getting from my server can be either of these error responses (that's all I have for error response) if something has gone wrong -
{"warning": "user_id not found", "user_id": some_user_id}
{"error": "user_id for wrong partition", "user_id": some_user_id, "partition": some_partition}
{"error": "missing client id", "client_id":2000}
or below successful response (it can be any random json string key can also be different) -
{"#data": {"oo":"1205000384","p":"2047935"}
If I am getting any error response as mentioned above, then I am deserializing it (my bad :( ) so that I can log them as an error with a specific error or warning I got front the server which can be for example - user_id not found or missing client id.
If it is a successful response then also I am deserializing it which I don't need for my use case as we don't have any POJO and I just need to return the response as it is which I have got from the server.
In my use case, I don't need to deserialize my response string if it is a successful response as we don't have any POJO for that and we are returning the response string as it is which we have got from the server. But just for logging specific error messages (if I am getting error response from the server) I am deserializing it which I am thinking is unnecessary. There might be better solution for my use case.
Below is my Java client which is calling Callable task using future.get -
public class TestingClient implements IClient {
private ExecutorService service = Executors.newFixedThreadPool(10);
private RestTemplate restTemplate = new RestTemplate();
#Override
public String executeSync(ClientKey keys) {
String response = null;
try {
ClientTask ClientTask = new ClientTask(keys, restTemplate);
Future<String> future = service.submit(ClientTask);
response = handle.get(keys.getTimeout(), TimeUnit.MILLISECONDS);
} catch (TimeoutException e) {
} catch (Exception e) {
}
return response;
}
}
And now below is my ClientTask class which implements Callable interface. In the call method, I am generating an URL and then hit the server using RestTemplate and get the response back -
class ClientTask implements Callable<String> {
private ClientKey cKeys;
private RestTemplate restTemplate;
public ClientTask(ClientKey cKeys, RestTemplate restTemplate) {
this.restTemplate = restTemplate;
this.cKeys = cKeys;
}
#Override
public String call() throws Exception {
// .. some code here
String url = "some_url";
String response = restTemplate.getForObject(url, String.class);
String test = checkJSONResponse(response);
return test;
}
private String checkJSONResponse(final String response) throws Exception {
// may be there are some better way of doing it for my scenario instead of using GSON
Gson gson = new Gson();
String str = null;
JsonObject jsonObject = gson.fromJson(response, JsonObject.class); // parse it, may be performance issues here/
if (jsonObject.has("error") || jsonObject.has("warning")) {
final String error = jsonObject.get("error") != null ? jsonObject.get("error").getAsString() : jsonObject
.get("warning").getAsString();
// log specific `error` here using log4j
str = response;
} else {
str = response;
}
return str;
}
}
As you can see in my above code we are deserializing the JSON string only to log specific error messages if we are getting any error response back. But for successful response we don't need any deserialization but still we are doing it.
Is there any better way of solving this problem? Because currently I am seeing some performance issues with the GSON deserialization.
The only way I can identify successful response along with error response is with error or warning in the response so I am thinking of using regular expressions which can identify error or warning as the key in the response string. If they contain error or warning in the response string then extract the specific error or warning message and log it. But not sure whether this will have any performance benefit or not.
Is there any other better way of solving this problem without using GSON deserialization.
It is a good practice to use HTTP status codes for your responses (e.g. BAD_REQUEST, NOT_FOUND). Return one of them from the server and then check on the client. It will allow to parse response only if some error code is returned:
String result = restTemplate.execute("url", HttpMethod.GET, null, new HttpMessageConverterExtractor<String> {
#Override
public MyEntity extractData(ClientHttpResponse response)
throws IOException {
String result = super.extractData(response);
if (response.getStatusCode() != HttpStatus.OK) {
// parse message and log only for some error code
JsonObject errorJson = parse(result);
log.warn("Got {} status error, with message [{}]", response.getStatusCode(), errorJson.get("warning"));
}
return result;
}
});
You do not need to deserialize to a POJO.
A simple JSON parser such as the one found on json.org will provide minimal JSON parsing an return a JSONObject that you can query.
I very much doubt that
you can come up with a faster parsing of your json responses using regular expressions or otherwise, without taking the risk of failing in corner cases
given the size of your response strings, that the JSON parsing is the performance bottleneck in your code
Unless you have done some serious profiling, I would play safe and follow the first rule of program optimization
I am working on a project in which I am making a call to one of my servers using RestTemplate which is running a restful service and getting the response back from them.
The response that I will be getting from my server can be either of these error responses (that's all I have for error response) if something has gone wrong -
{"warning": "user_id not found", "user_id": some_user_id}
{"error": "user_id for wrong partition", "user_id": some_user_id, "partition": some_partition}
{"error": "missing client id", "client_id":2000}
or below successful response (it can be any random json string key can also be different) -
{"#data": {"oo":"1205000384","p":"2047935"}
If I am getting any error response as mentioned above, then I am deserializing it (my bad :( ) so that I can log them as an error with a specific error or warning I got front the server which can be for example - user_id not found or missing client id.
If it is a successful response then also I am deserializing it which I don't need for my use case as we don't have any POJO and I just need to return the response as it is which I have got from the server.
In my use case, I don't need to deserialize my response string if it is a successful response as we don't have any POJO for that and we are returning the response string as it is which we have got from the server. But just for logging specific error messages (if I am getting error response from the server) I am deserializing it which I am thinking is unnecessary. There might be better solution for my use case.
Below is my Java client which is calling Callable task using future.get -
public class TestingClient implements IClient {
private ExecutorService service = Executors.newFixedThreadPool(10);
private RestTemplate restTemplate = new RestTemplate();
#Override
public String executeSync(ClientKey keys) {
String response = null;
try {
ClientTask ClientTask = new ClientTask(keys, restTemplate);
Future<String> future = service.submit(ClientTask);
response = handle.get(keys.getTimeout(), TimeUnit.MILLISECONDS);
} catch (TimeoutException e) {
} catch (Exception e) {
}
return response;
}
}
And now below is my ClientTask class which implements Callable interface. In the call method, I am generating an URL and then hit the server using RestTemplate and get the response back -
class ClientTask implements Callable<String> {
private ClientKey cKeys;
private RestTemplate restTemplate;
public ClientTask(ClientKey cKeys, RestTemplate restTemplate) {
this.restTemplate = restTemplate;
this.cKeys = cKeys;
}
#Override
public String call() throws Exception {
// .. some code here
String url = "some_url";
String response = restTemplate.getForObject(url, String.class);
String test = checkJSONResponse(response);
return test;
}
private String checkJSONResponse(final String response) throws Exception {
// may be there are some better way of doing it for my scenario instead of using GSON
Gson gson = new Gson();
String str = null;
JsonObject jsonObject = gson.fromJson(response, JsonObject.class); // parse it, may be performance issues here/
if (jsonObject.has("error") || jsonObject.has("warning")) {
final String error = jsonObject.get("error") != null ? jsonObject.get("error").getAsString() : jsonObject
.get("warning").getAsString();
// log specific `error` here using log4j
str = response;
} else {
str = response;
}
return str;
}
}
As you can see in my above code we are deserializing the JSON string only to log specific error messages if we are getting any error response back. But for successful response we don't need any deserialization but still we are doing it.
Is there any better way of solving this problem? Because currently I am seeing some performance issues with the GSON deserialization.
The only way I can identify successful response along with error response is with error or warning in the response so I am thinking of using regular expressions which can identify error or warning as the key in the response string. If they contain error or warning in the response string then extract the specific error or warning message and log it.
I guess there might be some better way of solving this problem without paying the cost for deserialization.
Just relying on a regex is I think to dangerous. What if the server slightly changes the output format?
I would try to make a quick test, possibly with a simple regexp looking for the string "error" and if there is a chance that it is an error response do a full deserialization to determine if it really was an error or not.
You would pay the extra cost only for false positives when a regular response by chance triggers the quick check.
I would use http codes to control success/fail data parsing.
Now I'm learning how to use Gson library to set and get data from webservice in Json format, but its best practices and strategies are a bit dark for me so I will be very delightful if somebody would explain more about it.
I've created an Entity class to get response entity from server:
public class Response
{
#SerializedName("Type")
public String Type;
#SerializedName("result")
public String result;
}
and in AsyncTask class I've used:
Response _Response = new Response();
try
{
String _url = Global.Url_Request ;
Map<String, String> Params = new HashMap<String, String>();
Params.put("PhoneNumber", this.User_PhoneNumber);
String json = new GsonBuilder().create().toJson(Params, Map.class);
HttpClient httpclient = new DefaultHttpClient();
HttpPost httpPost = new HttpPost(_url);
httpPost.setEntity(new StringEntity(json));
httpPost.setHeader("Accept", "application/json");
httpPost.setHeader("Content-type", "application/json");
HttpResponse getResponse = httpclient.execute(httpPost);
HttpEntity returnEntity = getResponse.getEntity();
is = returnEntity.getContent();
Gson gson = new Gson();
Reader reader = new InputStreamReader(is);
_Response = gson.fromJson(reader, Response.class);
}
catch (Exception e)
{
_Response.Type= "Error";
_Response.result= "Data Is Wrong";
}
return _Response;
It works fine with creating an Entity Object for every different http POST call, but my questions are:
What is the best practice for handling webservices with different response objects?
How can I handle this situation: if data sent ok then return specific Jsonarray; if not, return a Response object to detect something is wrong. Should I use Custom typeAdapter?(sample code would be great)
If webservice returns an empty response gson.fromJson would throw an **IllegalStateException: Expected a string but was BEGIN_OBJECT** how can i prevent this?
Thanks in advance
1. What is the best practice for handling webservices with different response objects?
I think that this depends on the kind of control you have. If you code also the webservice, you could create a big container object that has may fields. Each of these fields is one of the possible responses you can pass between client and server. If you have not control on what the server can reply, and it can differ a lot, JsonParser is your best friend. You can use it to snoop inside JSON and decide the right strategy to handle the response.
Let's do an example for case one. I declare these classes:
public static class GenericResponse{
public ServerException exception;
public StandardResponse1 responseType1;
public StandardResponse2 responseType2;
#Override
public String toString() {
return "GenericResponse [exception=" + exception + ", responseType1=" + responseType1 + ", responseType2=" + responseType2 + "]";
}
}
public static class ServerException{
public int error;
public String message;
}
public static class StandardResponse1{
public List<Integer> list;
public Date now;
}
With this kind of classes, I can parse:
{"responseType1":{"list":[1,2],"now":"Nov 25, 2013 9:26:51 PM"}}
or
{"exception":{"error":-1,"message":"Don\u0027t do this at home"}}
For example, if I get from server the second type of response, this code:
GenericResponse out = g.fromJson(fromServerStream, GenericResponse.class);
System.out.println(out);
will return me:
GenericResponse [exception=stackoverflow.questions.Q20187804$ServerException#1e9d085, responseType1=null, responseType2=null]
All you have to do is to check your fields to see what actually the server replied.
Case two. You cannot control the JSON, so the server can reply
[13,17]
or
{"error":-1,"message":"Don\u0027t do this at home"}
In this case you cannot pass directly the class type to Gson as before, but you have to check things. I would solve this problem with a JsonParser.
Gson g = new Gson();
JsonParser jp = new JsonParser();
JsonElement o = jp.parse(s);
if (o.isJsonArray()){
List<Integer> list = (List) g.fromJson(o, listType1);
System.out.print(list);
}
else{
ServerException e = g.fromJson(s, ServerException.class);
System.out.print(e);
}
Using JsonObject/JsonArray and so on, is what happens inside a TypeAdapter. In the adapter you start with the JsonElement that is already parsed. There are many good example of it on SO, this for example.
How can I handle this situation: if data sent ok then return specific Jsonarray; if not, return a Response object to detect something is wrong. Should I use Custom typeAdapter?(sample code would be great)
Do you mean you want to parse this kind of response? Examples of point 1 show this.
If webservice returns an empty response gson.fromJson would throw an **IllegalStateException: Expected a string but was BEGIN_OBJECT how can i prevent this?**
JsonParser/TypeAdapter is again the solution. You can check if JsonElement is null or if is a primitive type (String, Integer, Boolean) and deal it.