I have a REST api that responds with some additional non JSON data in the body content. This breaks the use of RestTemplate and jackson. Can I intercept the http response body prior to the parsing?
I am using RestTemplate.getForObject.
I've taken a look at the RestTemplate and couldn't see an appropriate method.
You can try to implement ClientHttpRequestInterceptor and assign it to restTemplate. Implement intercept method:
#Override
public ClientHttpResponse intercept(HttpRequest httpRequest, byte[] bytes,
ClientHttpRequestExecution clientHttpRequestExecution) throws IOException {
ClientHttpResponse response=clientHttpRequestExecution.execute(httpRequest, bytes);
//...do magic with response body from getBody method
return response;
}
You might have to extend AbstractClientHttpResponse with your own implementation to do that.
Another option could be to treat the response from the REST API as String, then format the string as needed and explicitly map it to object using ObjectMapper.
Then in your restTemplate you would have:
String result = restTemplate.getForObject(url, String.class, host);
//..trim the extra stuff
MyClass object=objectMapper.readValue(result, MyClass.class);
Yet another option would be to implement your own HttpMessageConverter which extends AbstractJackson2HttpMessageConverter and register it with restTemplate. In my opinion that would be the cleaneast from the Spring point of view
Another way would be to unwrap the response by implementing a ClientHttpRequestInterceptor along with a ClientHttpResponse.
#Component
public class MyInterceptor implements ClientHttpRequestInterceptor {
#Autowired
Function<ClientHttpResponse, MyResponseWrapper> responseWrapperBeanFactory;
#Autowired
MyRequestAdvice requestAdvice;
#Override
public ClientHttpResponse intercept(HttpRequest httpRequest, byte[] bytes, ClientHttpRequestExecution clientHttpRequestExecution) throws IOException {
byte[] wrappedBody = requestAdvice.wrapRequest(bytes);
ClientHttpResponse res = clientHttpRequestExecution.execute(httpRequest, wrappedBody);
return responseWrapperBeanFactory.apply(res);
}
}
Here's the bean config for the MyResponseWrapper:
#Bean
Function<ClientHttpResponse, MyResponseWrapper> responseWrapperBeanFactory() {
return this::getMyResponseWrapper;
}
#Bean
#Scope(ConfigurableBeanFactory.SCOPE_PROTOTYPE)
public MyResponseWrapper getMyResponseWrapper(ClientHttpResponse originalResponse) {
return new MyResponseWrapper(originalResponse);
}
#Bean
public RestTemplate restTemplate(#Autowired MyInterceptor interceptor) {
RestTemplate t = new RestTemplate();
t.setInterceptors(Arrays.asList(interceptor));
// other setup code
return t;
}
And here's the ClientHttpResponse implementation:
public class MyResponseWrapper implements ClientHttpResponse {
private byte[] filteredContent;
private ByteArrayInputStream responseInputStream;
private ClientHttpResponse originalResponse;
public MyResponseWrapper(ClientHttpResponse originalResponse) {
this.originalResponse = originalResponse;
try {
filteredContent = MyContentUnwrapper.unwrapResponse(originalResponse.getBody().readAllBytes());
} catch (Exception e) {
throw new RuntimeException("There was a problem reading/decoding the response coming from the service ", e);
}
}
#Override
public HttpStatus getStatusCode() throws IOException {
return originalResponse.getStatusCode();
}
#Override
public int getRawStatusCode() throws IOException {
return originalResponse.getRawStatusCode();
}
#Override
public String getStatusText() throws IOException {
return originalResponse.getStatusText();
}
#Override
public void close() {
if (responseInputStream != null) {
try {
responseInputStream.close();
} catch (IOException e) { /* so long */}
}
}
#Override
public InputStream getBody() throws IOException {
if (responseInputStream == null) {
responseInputStream = new ByteArrayInputStream(filteredContent);
}
return responseInputStream;
}
#Override
public HttpHeaders getHeaders() {
return originalResponse.getHeaders();
}
}
From your Controller you can try to return a ResponseEntity and manipulate the entity object manually
If you don't need these extra properties you may add:
#JsonIgnoreProperties(ignoreUnknown = true)
to your mapping class.
From docs:
Property that defines whether it is ok to just ignore any unrecognized
properties during deserialization. If true, all properties that are
unrecognized -- that is, there are no setters or creators that accept them
-- are ignored without warnings (although handlers for unknown properties,
if any, will still be called) without exception.
Does not have any effect on serialization.
Related
I am implementing a filter to find out any json mapping or parsing issues using ContainerRequestFilter interface. It logs error for all json parsing or mapping issues but for request which doesn't have any json validation issues it throws 400 Bad request. If I remove the filter from the code it works fine for proper json request. Below is the code for the same ..
#Provider
public class AppFilter implements ContainerRequestFilter {
private static final Logger LOG = Logger.getLogger(AppFilter.class);
#Inject
ObjectMapper objectMapper;
#Override
public void filter(ContainerRequestContext context) throws IOException {
try(InputStream stream = new CloseShieldInputStream(context.getEntityStream())) {
objectMapper.readValue(stream,AppRequest.class);
}catch(Exception ex) {
LOG.error(ExceptionUtils.getRootCauseMessage(ex));
}
}
}
#Path("/app")
public class MatchResource {
#POST
#Produces(MediaType.APPLICATION_JSON)
public Uni<Response> processRequest(AppRequest request,
#Context SecurityContext securityContext) {
String tranId = UUID.randomUUID().toString();
ZonedDateTime zonedDateTime = ZonedDateTime.now(TIME_ZONE_ID);
return Uni.createFrom().item(request)
.onItem().invoke(() -> {
Set<ConstraintViolation<AppRequest>> violations = validator.validate(request);
if (!violations.isEmpty()) {
throw new RequestViolationsException(violations,
request.header.applicationReferenceId);
}
}).onFailure()
.transform(t -> LOG.error(ExceptionUtils.getRootCauseMessage(t)));
}
Let me know if I am missing anything in this ..
I am using Feign as a HTTP client. Now I am facing an API using Google Protobuf. Feign does not provide any encoder/decoder for Protobuf so it is necessary to implement custom ones.
import feign.RequestTemplate
import feign.codec.EncodeException
import feign.codec.Encoder
import java.lang.reflect.Type
class ProtobufEncoder implements Encoder {
#Override
void encode(Object object, Type bodyType, RequestTemplate template) throws EncodeException {
// I am stuck here
}
}
Someone solved this before?
Update: I am looking for a solution that uses plain Feign without Spring.
One way to achieve protobuf encoding (described in this GitHub issue) is to use ProtobufHttpMessageConverter class with SpringEncoder \ SpringDecoder from spring-cloud-openfeign library.
In your application you need to configure the bean:
import org.springframework.http.converter.protobuf.ProtobufHttpMessageConverter;
#Configuration
public class ProtoBufConfig {
#Autowired
private ObjectFactory<HttpMessageConverters> messageConverters;
#Bean
ProtobufHttpMessageConverter protobufHttpMessageConverter() {
return new ProtobufHttpMessageConverter();
}
//override the encoder
#Bean
public Encoder springEncoder(){
return new SpringEncoder(this.messageConverters);
}
#Bean
public Decoder springDecoder(){
return new ResponseEntityDecoder(new SpringDecoder(this.messageConverters));
}
}
And to you can use this configuration in the FeignClient annotation as such:
#FeignClient(name = "example-svc", configuration=ProtoBufConfig.class)
public interface KittensApiResource {
#RequestMapping(value="/v1/api/kittens", method = RequestMethod.GET, consumes="application/x-protobuf", produces="application/x-protobuf")
public ResponseEntity<KittenResponse> kittens();
}
That is what I came up with. I also had to encode/decode with Base64 because it was required by the API I was working with.
Decoder:
#Slf4j
class ProtobufBase64Decoder implements Decoder {
Map<Type, Message> protoMessagesCache = new HashMap<>()
#Override
Object decode(Response response, Type type) throws IOException, DecodeException, FeignException {
if (response.body() == null) {
return null
}
InputStream inputStream = BaseEncoding.base64().decodingStream(response.body().asReader(Charset.defaultCharset()))
Message message = computeMessageForType(type)
message.toBuilder().mergeFrom(inputStream).build()
}
private Message computeMessageForType(Type type) {
if (!protoMessagesCache.containsKey(type)) {
if (Message.class.isAssignableFrom(type)) {
Method builderMethod = type.getMethod("newBuilder")
Message message = ((Message.Builder) builderMethod.invoke(type)).getDefaultInstanceForType()
protoMessagesCache.put(type, message)
} else {
throw new NoClassDefFoundError()
}
}
protoMessagesCache.get(type)
}
}
Encoder:
#Slf4j
class ProtobufBase64Encoder implements Encoder {
#Override
void encode(Object object, Type bodyType, RequestTemplate template) throws EncodeException {
try {
Message message = Message.cast(object)
template.body(BaseEncoding.base64().encode(message.toByteArray()))
} catch (ClassCastException ignored) {
log.error("Unable to encode {} into protobuf", object)
}
}
}
I had developed a Rest API application and have handled Authentication and Authorization using custom JWT.
I want to further make the application secure from XSS attacks or validation for untrusted data which could be handled for each and every field of JSON request.
Can I get some help in this regard so that efficient data processing will happen at the entry-level of the request without touching internal business validation?
You don't filter or escape data in a restful API. API's should be client agnostic. It is the clients responsibility to provide XSS protection. If the clients are doing their job appropriately you will end up with doubly escaped data. Remember potential clients can be:
Mobile Apps
Backend Web Servers
Web Browsers
Desktop Applications
Embedded systems/ IoT
In the above only a limited number of clients and configurations are vulnerable to XSS.
Need to override the HttpServletRequest in a Servlet Filter(if you are using Servlet).
Extends HttpServletRequestWrapper that stores JSON body(intention is to sanitize JSON body).
Strip/ escape the eligible JSON value
Extented "HttpServletRequestWrapper" :
public class SanitizationRequestWrapper extends HttpServletRequestWrapper {
public byte[] getBody() {
return body;
}
public void setBody(byte[] body) {
this.body = body;
}
private byte[] body;
public SanitizationRequestWrapper(HttpServletRequest request) throws IOException {
super(request);
try {
body = IOUtils.toByteArray(super.getInputStream());
}catch (NullPointerException e){
}
}
#Override
public ServletInputStream getInputStream() throws IOException {
return new ServletInputStreamImpl(new ByteArrayInputStream(body));
}
#Override
public BufferedReader getReader() throws IOException {
String enc = getCharacterEncoding();
if (enc == null) enc = "UTF-8";
return new BufferedReader(new InputStreamReader(getInputStream(), enc));
}
private class ServletInputStreamImpl extends ServletInputStream {
private InputStream is;
public ServletInputStreamImpl(InputStream is) {
this.is = is;
}
public int read() throws IOException {
return is.read();
}
public boolean markSupported() {
return false;
}
public synchronized void mark(int i) {
throw new RuntimeException(new IOException("mark/reset not supported"));
}
public synchronized void reset() throws IOException {
throw new IOException("mark/reset not supported");
}
}
}
Servlet filter which sanitize request body:
public class XSSSanitizeFilters implements Filter {
#Override
public void destroy() {
}
#Override
public void doFilter(ServletRequest arg0, ServletResponse arg1, FilterChain arg2) throws IOException, ServletException {
HttpServletRequest request = (HttpServletRequest) arg0;
HttpServletResponse response = (HttpServletResponse) arg1;
SanitizationRequestWrapper sanitizeRequest = new SanitizationRequestWrapper(request);
if (null != sanitizeRequest.getBody()) {
try {
sanitizeJson(sanitizeRequest);
} catch (ParseException e) {
LOG.error("Unable to Sanitize the provided JSON .");
}
arg2.doFilter(sanitizeRequest, arg1);
} else {
arg2.doFilter(arg0, arg1);
}
}
public void init(FilterConfig filterConfig) throws ServletException {
}
private void sanitizeJson(SanitizationRequestWrapper sanitizeRequest ) throws IOException, ParseException {
JSONParser parser= new JSONParser();
Object obj = parser.parse(sanitizeRequest.getReader());
ObjectMapper oMapper = new ObjectMapper();
Map <String, Object> map = oMapper.convertValue(obj, Map.class);
sanitizeRequest.setBody((new JSONObject(map)).toString().getBytes());
}
For this you need XSS filter using HTMLUtils which will filter any injected script and prevent your site. Please refer my answer https://stackoverflow.com/a/55741351/10232467 for its complete code and implementation.
If your API doesn't accecpt any HTML Characters then you can follow the below logic.
You can Sanitize the Input Payload with EncodeHtml and Compare it with Provided Payload.
If both Sanitized Payload and Provided payload doesn't match then there exists some Html Content and straight way throw an Excpetion.
String unsanitizedPayload = IOUtils.toString(multiReadRequest.getReader());
String sanitizedPayload = Encode.forHtmlContent(unsanitizedPayload);
if(!unsanitizedPayload.equals(sanitizedPayload)) {
throw new Exception("Improper Payload");
}
If you're using Spring, Spring security guarantees basic level of protection against XSS attack. You can also use
#SafeHtml
private String value;
You will also need to add org.jsoup dependency.
I'm trying to catch an exception thrown in my implementation of StreamingResponseBody, I can see the exception being thrown inside the class however the thrown exception isn't visible to the method body or my Controller Advice. So none of my handling seems to work, just interested to know which is the correct way to handle exceptions in this case.
#GetMapping(path = "/test", produces = "application/json")
public StreamingResponseBody test(#RequestParam(value = "var1") final String test)
throws IOException{
return new StreamingResponseBody() {
#Override
public void writeTo(final OutputStream outputStream) throws IOException{
try {
// Some operations..
} catch (final SomeCustomException e) {
throw new IOException(e);
}
}
};
}
I would expect my ControllerAdvice to return an ResponseEntity with a Http Status of 500.
The best way I discovered to handle errors/exceptions in the web environment is to create your custom exception with the disabled stack trace, and handle it with #ControllerAdvice.
import lombok.Getter;
import org.springframework.http.HttpStatus;
public class MyException extends RuntimeException {
#Getter private HttpStatus httpStatus;
public MyException(String message) {
this(message, HttpStatus.INTERNAL_SERVER_ERROR);
}
public MyException(String message, HttpStatus status) {
super(message, null, false, false);
this.httpStatus = status;
}
}
And then handle it in #ControllerAdvice like this:
#ExceptionHandler(MyException.class)
public ResponseEntity handleMyException(MyException exception) {
return ResponseEntity.status(exception.getHttpStatus()).body(
ErrorDTO.builder()
.message(exception.getMessage())
.description(exception.getHttpStatus().getReasonPhrase())
.build());
}
where ErrorDTO is just a simple DTO with two fields:
#Value
#Builder
public class ErrorDTO {
private final String message;
private final String description;
}
In my project, I have a set of api calls which should filtered through certain set of common validation. In that case, I have to intercept the request before it hits the REST controller, read the request body, do the validations and pass it to the controller if the request passes the validations.
Since the HttpServletRequest cannot be deserialized more than once, I used a HttpServletRequestWrapper to make a copy of the actual request. Using the copy it makes, I do the validations.
Following is the configuration class for intercepting the requests.
public class InterceptorConfig extends WebMvcConfigurerAdapter {
#Autowired
CustomInterceptor customInterceptor;
#Override
public void addInterceptors(InterceptorRegistry registry) {
registry.addInterceptor(customInterceptor).addPathPatterns("/signup/**");
}
}
Here is my preHandle method inside CustomInterceptor class which extends HandlerInterceptorAdaptor
#Override
public boolean preHandle(HttpServletRequest request, HttpServletResponse response, Object handler) throws Exception {
ServletRequest copiedRequest = new HttpRequestWrapper(request);
Map<String, Object> jsonMap = mapper.readValue(copiedRequest.getInputStream(), Map.class);
if(jsonMap.containsKey("userId")){
long userId = jsonMap.get("userId");
MyClass myObject= myAutowiredService.getMyObject(userId);
if(myObject == null){
response.setStatus(HttpStatus.SC_NOT_ACCEPTABLE);
return false;
}
// some more validations which end up returning false if they are met
}
return true;
}
This is my HttpRequestWrapper
public class HttpRequestWrapper extends HttpServletRequestWrapper {
private byte[] requestBody;
public HttpRequestWrapper(HttpServletRequest request) throws IOException{
super(request);
try {
requestBody = IOUtils.toByteArray(request.getInputStream());
} catch (IOException ex) {
requestBody = new byte[0];
}
}
#Override
public ServletInputStream getInputStream() throws IOException {
final ByteArrayInputStream byteArrayInputStream = new ByteArrayInputStream(requestBody);
return new ServletInputStream() {
#Override
public boolean isFinished() {
return byteArrayInputStream.available() == 0;
}
#Override
public boolean isReady() {
return true;
}
#Override
public void setReadListener(ReadListener listener) {
throw new RuntimeException("Not implemented");
}
public int read () throws IOException {
return byteArrayInputStream.read();
}
};
}
}
All set now. Now, when I send a request to any url with the pattern of /signup/**, all the validations are happening fine. However, once the request hits the controller method, error pops out saying the request body is not available.
Required request body is missing: public
com.mypackage.myResponseObject
com.mypackage.myController.myControllerMethod(com.mypackage.myDTO)
I am struggling to find the reason for this and also a way to overcome the issue. Is there anything I have done wrong in RequestWrapper class? or anything missing?
Help me to sort this thing out.
Thanks!
The Problem seems to be that you are using an Interceptor to read the HttpServletRequest's InputStream and just wrap it in HttpRequestWrapper but the wrapper is never returned.
I think you should use a Filter
public class CustomFilter extends OncePerRequestFilter {
public void doFilterInternal(HttpServletRequest request, HttpServletResponse response, FilterChain filterChain) throws ServletException, IOException {
ServletRequest copiedRequest = new HttpRequestWrapper(request);
Map<String, Object> jsonMap = mapper.readValue(copiedRequest.getInputStream(), Map.class);
if(jsonMap.containsKey("userId")){
long userId = jsonMap.get("userId");
MyClass myObject= myAutowiredService.getMyObject(userId);
if(myObject == null){
response.setStatus(HttpStatus.SC_NOT_ACCEPTABLE);
//return false;
}
// some more validations which end up returning false if they are met
}
filterChain.doFilter(copiedRequest, (ServletResponse) response);
}
}
And you need to use this Filter in either web.xml or WebApplicationInitializer