We Keep Coding

API call with Java + STS returning "Content type 'application/octet-stream' not supported"

I am working on part of an API, which requires making a call to another external API to retrieve data for one of its functions. The call was returning an HTTP 500 error, with description "Content type 'application/octet-stream' not supported." The call is expected to return a type of 'application/json."
I found that this is because the response received doesn't explicitly specify a content type in its header, even though its content is formatted as JSON, so my API defaulted to assuming it was an octet stream.
The problem is, I'm not sure how to adjust for this. How would I get my API to treat the data it receives from the other API as an application/json even if the other API doesn't specify a content type? Changing the other API to include a contenttype attribute in its response is infeasible.
Code:
The API class:
#RestController
#RequestMapping(path={Constants.API_DISPATCH_PROFILE_CONTEXT_PATH},produces = {MediaType.APPLICATION_JSON_VALUE})
public class GetProfileApi {
#Autowired
private GetProfile GetProfile;
#GetMapping(path = {"/{id}"})
public Mono<GetProfileResponse> getProfile(#Valid #PathVariable String id){
return GetProfile.getDispatchProfile(id);
}
The service calling the external API:
#Autowired
private RestClient restClient;
#Value("${dispatch.api.get_profile}")
private String getDispatchProfileUrl;
#Override
public Mono<GetProfileResponse> getDispatchProfile(String id) {
return Mono.just(id)
.flatMap(aLong -> {
MultiValueMap<String, String> headers = new HttpHeaders();
headers.add(HttpHeaders.CONTENT_TYPE, MediaType.APPLICATION_JSON_VALUE);
return restClient.get(getDispatchProfileUrl, headers);
}).flatMap(clientResponse -> {
HttpStatus status = clientResponse.statusCode();
log.info("HTTP Status : {}", status.value());
return clientResponse.bodyToMono(GetProfileClientResponse.class);
// the code does not get past the above line before returning the error
}).map(GetProfileClientResponse -> {
log.debug("Response : {}",GetProfileClientResponse);
String id = GetProfileClientResponse.getId();
log.info("SubscriberResponse Code : {}",id);
return GetProfileResponse.builder()
// builder call to be completed later
.build();
});
}
The GET method for the RestClient:
public <T> Mono<ClientResponse> get(String baseURL, MultiValueMap<String,String> headers){
log.info("Executing REST GET method for URL : {}",baseURL);
WebClient client = WebClient.builder()
.baseUrl(baseURL)
.defaultHeaders(httpHeaders -> httpHeaders.addAll(headers))
.build();
return client.get()
.exchange();
}
One solution I had attempted was setting produces= {MediaType.APPLICATION_JSON_VALUE} in the #RequestMapping of the API to produces= {MediaType.APPLICATION_OCTET_STREAM_VALUE}, but this caused a different error, HTTP 406 Not Acceptable. I found that the server could not give the client the data in a representation that was requested, but I could not figure out how to correct it.
How would I be able to treat the response as JSON successfully even though it does not come with a content type?
Hopefully I have framed my question well enough, I've kinda been thrust into this and I'm still trying to figure out what's going on.

Are u using jackson library or jaxb library for marshalling/unmarshalling?
Try annotating Mono entity class with #XmlRootElement and see what happens.

How to consume a HTTPS GET service with Spring Boot

I am trying to consume the following HTTPS endpoints from Yahoo Weather Service:
Yahoo Weather Service API
I am doing some special query according to the API to get the current weather at some parametrized location.
#Service("weatherConditionService")
public class WeatherConditionServiceImpl implements WeatherConditionService {
private static final String URL = "http://query.yahooapis.com/v1/public/yql";
public WeatherCondition getCurrentWeatherConditionsFor(Location location) {
RestTemplate restTemplate = new RestTemplate();
StringBuilder stringBuilder = new StringBuilder();
stringBuilder.append(URL);
stringBuilder.append("?q=select%20item.condition%20from%20weather.forecast%20where%20woeid%20in%20(select%20woeid%20from%20geo.places(1)%20where%20text%3D%22");
// TODO: Validate YQL query injection
stringBuilder.append(location.getName());
stringBuilder.append("%22)&format=json&env=store%3A%2F%2Fdatatables.org%2Falltableswithkeys");
WeatherQuery weatherQuery = restTemplate.getForObject(stringBuilder.toString(), WeatherQuery.class);
// TODO: Test Json mapping response
Condition condition = weatherQuery.getQuery().getResults().getChannel().getItem().getCondition();
return new WeatherCondition(condition.getDate(), Integer.parseInt(condition.getTemp()), condition.getText());
}
Location is a class that provides the attribute "name" that is a String description of the location, such as "New York" or "Manila".
Condition an other classes just map the returning object.
When executing I get the following HTTP response:
org.springframework.web.client.HttpClientErrorException: 403 Forbidden
So this means I am not authorized to access the resource from what I understand.
The URL works great if I just copy & paste it in a web browser:
Yahoo Weather Query
I think that mapping is not a problem since I am not getting "400" (Bad Request) but "403" (Forbidden)
There must be some error on the way I use the RestTemplate object. I am researching but I can't find an answer.

The docs say you need an api key. But when I make a call like this:
fetch('https://query.yahooapis.com/v1/public/yql?q=select%20*%20from%20weather.forecast%20where%20woeid%20in%20(select%20woeid%20from%20geo.places(1)%20where%20text%3D%22nome%2C%20ak%22)&format=json&env=store%3A%2F%2Fdatatables.org%2Falltableswithkeys')
.then(resp=> resp.json())
.then((res)=>console.log(res.query.results))
https://repl.it/NeoM
It works fine without one. Perhaps you've been blackisted for hitting the api too often.
Your code seems fine.

I finally found the answer. It finally WAS a Bad Request because I needed to pass the parameters differently (not as part of the URL).
I found the answer here. Here goes the code for my particular Yahoo Weather API call return a String (I still will have to do some work to use the mapping).
private static final String URL = "http://query.yahooapis.com/v1/public/yql";
public String callYahooWeatherApi() {
RestTemplate restTemplate = new RestTemplate();
HttpHeaders headers = new HttpHeaders();
headers.set("Accept", MediaType.APPLICATION_JSON_VALUE);
UriComponentsBuilder builder = UriComponentsBuilder.fromHttpUrl(URL)
.queryParam("q", "select wind from weather.forecast where woeid=2460286")
.queryParam("format", "json");
HttpEntity<?> entity = new HttpEntity<>(headers);
HttpEntity<String> response = restTemplate.exchange(
builder.build().encode().toUri(),
HttpMethod.GET,
entity,
String.class);
return response.getBody();
}

How to get websocket in webclient of gargoylesoftware

Is Websocket supported in gargoylesoftware library? I want to get websocket object in webclient.

Yes, WebSocket is supported since version 2.11. However, it is always recommended to use the latest version.
Please ensure you use BrowserVersion with recent browser, e.g. CHROME, FIREFOX_38, or INTERNET_EXPLORER_11.
E.g.:
try (final WebClient webClient = new WebClient(BrowserVersion.CHROME)) {
HtmlUnit will automatically handle the JavaScript with WebSocket.
Update:
To intercept the requests and responses, you can use:
new WebConnectionWrapper(webClient) {
public WebResponse getResponse(WebRequest request) throws IOException {
WebResponse response = super.getResponse(request);
if (request.getUrl().toExternalForm().contains("my_url")) {
String content = response.getContentAsString("UTF-8");
//change content
WebResponseData data = new WebResponseData(content.getBytes("UTF-8"),
response.getStatusCode(), response.getStatusMessage(), response.getResponseHeaders());
response = new WebResponse(data, request, response.getLoadTime());
}
return response;
}
};

Efficient way to detect strings?

I am working on a project in which I am making a call to one of my servers using RestTemplate which is running a restful service and getting the response back from them.
The response that I will be getting from my server can be either of these error responses (that's all I have for error response) if something has gone wrong -
{"warning": "user_id not found", "user_id": some_user_id}
{"error": "user_id for wrong partition", "user_id": some_user_id, "partition": some_partition}
{"error": "missing client id", "client_id":2000}
or below successful response (it can be any random json string key can also be different) -
{"#data": {"oo":"1205000384","p":"2047935"}
If I am getting any error response as mentioned above, then I am deserializing it (my bad :( ) so that I can log them as an error with a specific error or warning I got front the server which can be for example - user_id not found or missing client id.
If it is a successful response then also I am deserializing it which I don't need for my use case as we don't have any POJO and I just need to return the response as it is which I have got from the server.
In my use case, I don't need to deserialize my response string if it is a successful response as we don't have any POJO for that and we are returning the response string as it is which we have got from the server. But just for logging specific error messages (if I am getting error response from the server) I am deserializing it which I am thinking is unnecessary. There might be better solution for my use case.
Below is my Java client which is calling Callable task using future.get -
public class TestingClient implements IClient {
private ExecutorService service = Executors.newFixedThreadPool(10);
private RestTemplate restTemplate = new RestTemplate();
#Override
public String executeSync(ClientKey keys) {
String response = null;
try {
ClientTask ClientTask = new ClientTask(keys, restTemplate);
Future<String> future = service.submit(ClientTask);
response = handle.get(keys.getTimeout(), TimeUnit.MILLISECONDS);
} catch (TimeoutException e) {
} catch (Exception e) {
}
return response;
}
}
And now below is my ClientTask class which implements Callable interface. In the call method, I am generating an URL and then hit the server using RestTemplate and get the response back -
class ClientTask implements Callable<String> {
private ClientKey cKeys;
private RestTemplate restTemplate;
public ClientTask(ClientKey cKeys, RestTemplate restTemplate) {
this.restTemplate = restTemplate;
this.cKeys = cKeys;
}
#Override
public String call() throws Exception {
// .. some code here
String url = "some_url";
String response = restTemplate.getForObject(url, String.class);
String test = checkJSONResponse(response);
return test;
}
private String checkJSONResponse(final String response) throws Exception {
// may be there are some better way of doing it for my scenario instead of using GSON
Gson gson = new Gson();
String str = null;
JsonObject jsonObject = gson.fromJson(response, JsonObject.class); // parse it, may be performance issues here/
if (jsonObject.has("error") || jsonObject.has("warning")) {
final String error = jsonObject.get("error") != null ? jsonObject.get("error").getAsString() : jsonObject
.get("warning").getAsString();
// log specific `error` here using log4j
str = response;
} else {
str = response;
}
return str;
}
}
As you can see in my above code we are deserializing the JSON string only to log specific error messages if we are getting any error response back. But for successful response we don't need any deserialization but still we are doing it.
Is there any better way of solving this problem? Because currently I am seeing some performance issues with the GSON deserialization.
The only way I can identify successful response along with error response is with error or warning in the response so I am thinking of using regular expressions which can identify error or warning as the key in the response string. If they contain error or warning in the response string then extract the specific error or warning message and log it. But not sure whether this will have any performance benefit or not.
Is there any other better way of solving this problem without using GSON deserialization.

It is a good practice to use HTTP status codes for your responses (e.g. BAD_REQUEST, NOT_FOUND). Return one of them from the server and then check on the client. It will allow to parse response only if some error code is returned:
String result = restTemplate.execute("url", HttpMethod.GET, null, new HttpMessageConverterExtractor<String> {
#Override
public MyEntity extractData(ClientHttpResponse response)
throws IOException {
String result = super.extractData(response);
if (response.getStatusCode() != HttpStatus.OK) {
// parse message and log only for some error code
JsonObject errorJson = parse(result);
log.warn("Got {} status error, with message [{}]", response.getStatusCode(), errorJson.get("warning"));
}
return result;
}
});

You do not need to deserialize to a POJO.
A simple JSON parser such as the one found on json.org will provide minimal JSON parsing an return a JSONObject that you can query.
I very much doubt that
you can come up with a faster parsing of your json responses using regular expressions or otherwise, without taking the risk of failing in corner cases
given the size of your response strings, that the JSON parsing is the performance bottleneck in your code
Unless you have done some serious profiling, I would play safe and follow the first rule of program optimization

How to use regular expressions to identify the error message in the JSON String?

I am working on a project in which I am making a call to one of my servers using RestTemplate which is running a restful service and getting the response back from them.
The response that I will be getting from my server can be either of these error responses (that's all I have for error response) if something has gone wrong -
{"warning": "user_id not found", "user_id": some_user_id}
{"error": "user_id for wrong partition", "user_id": some_user_id, "partition": some_partition}
{"error": "missing client id", "client_id":2000}
or below successful response (it can be any random json string key can also be different) -
{"#data": {"oo":"1205000384","p":"2047935"}
If I am getting any error response as mentioned above, then I am deserializing it (my bad :( ) so that I can log them as an error with a specific error or warning I got front the server which can be for example - user_id not found or missing client id.
If it is a successful response then also I am deserializing it which I don't need for my use case as we don't have any POJO and I just need to return the response as it is which I have got from the server.
In my use case, I don't need to deserialize my response string if it is a successful response as we don't have any POJO for that and we are returning the response string as it is which we have got from the server. But just for logging specific error messages (if I am getting error response from the server) I am deserializing it which I am thinking is unnecessary. There might be better solution for my use case.
Below is my Java client which is calling Callable task using future.get -
public class TestingClient implements IClient {
private ExecutorService service = Executors.newFixedThreadPool(10);
private RestTemplate restTemplate = new RestTemplate();
#Override
public String executeSync(ClientKey keys) {
String response = null;
try {
ClientTask ClientTask = new ClientTask(keys, restTemplate);
Future<String> future = service.submit(ClientTask);
response = handle.get(keys.getTimeout(), TimeUnit.MILLISECONDS);
} catch (TimeoutException e) {
} catch (Exception e) {
}
return response;
}
}
And now below is my ClientTask class which implements Callable interface. In the call method, I am generating an URL and then hit the server using RestTemplate and get the response back -
class ClientTask implements Callable<String> {
private ClientKey cKeys;
private RestTemplate restTemplate;
public ClientTask(ClientKey cKeys, RestTemplate restTemplate) {
this.restTemplate = restTemplate;
this.cKeys = cKeys;
}
#Override
public String call() throws Exception {
// .. some code here
String url = "some_url";
String response = restTemplate.getForObject(url, String.class);
String test = checkJSONResponse(response);
return test;
}
private String checkJSONResponse(final String response) throws Exception {
// may be there are some better way of doing it for my scenario instead of using GSON
Gson gson = new Gson();
String str = null;
JsonObject jsonObject = gson.fromJson(response, JsonObject.class); // parse it, may be performance issues here/
if (jsonObject.has("error") || jsonObject.has("warning")) {
final String error = jsonObject.get("error") != null ? jsonObject.get("error").getAsString() : jsonObject
.get("warning").getAsString();
// log specific `error` here using log4j
str = response;
} else {
str = response;
}
return str;
}
}
As you can see in my above code we are deserializing the JSON string only to log specific error messages if we are getting any error response back. But for successful response we don't need any deserialization but still we are doing it.
Is there any better way of solving this problem? Because currently I am seeing some performance issues with the GSON deserialization.
The only way I can identify successful response along with error response is with error or warning in the response so I am thinking of using regular expressions which can identify error or warning as the key in the response string. If they contain error or warning in the response string then extract the specific error or warning message and log it.
I guess there might be some better way of solving this problem without paying the cost for deserialization.

Just relying on a regex is I think to dangerous. What if the server slightly changes the output format?
I would try to make a quick test, possibly with a simple regexp looking for the string "error" and if there is a chance that it is an error response do a full deserialization to determine if it really was an error or not.
You would pay the extra cost only for false positives when a regular response by chance triggers the quick check.

I would use http codes to control success/fail data parsing.