We Keep Coding

WebClient doesn't fetch data as expected

I'm trying to reactively fetch data from external API using two methods from some Service class.
I'm new to reactive and Spring in general, so it could be a very obvious mistake but I just can't find it
These are the two methods:
public Mono<SomeClass> get(int value) {
return webClient.get()
.uri("/" + value)
.retrieve()
.onRawStatus(HttpStatus.CONFLICT::equals, clientResponse -> {
return Mono.error(new SomeException1("Some message", clientResponse.rawStatusCode()));
})
.onRawStatus(HttpStatus.NOT_FOUND::equals, clientResponse -> {
return requestGeneration(value)
.flatMap(res -> Mono.error(new SomeException1("Some message", clientResponse.rawStatusCode())));
})
.bodyToMono(SomeClass.class)
.retryWhen(Retry.backoff(5, Duration.ofSeconds(8))
.filter(throwable -> throwable instanceof SomeException1));
}
private Mono<Void> requestGeneration(int value) {
return webClient.post()
.uri("/" + value)
.retrieve()
.onRawStatus(HttpStatus.BAD_REQUEST::equals, clientResponse -> {
return Mono.error(new SomeException2("Wrong value", clientResponse.rawStatusCode()));
})
.bodyToMono(Void.class);
}
Baasically what I'm trying to achieve is:
first GET from http://api.examplepage.com/{value}
if that API returns HTTP404 it means I need to first call POST to the same URL, because the data is not yet generated
the second function does the POST call and returns Mono<Void> because it is just HTTP200 or HTTP400 on bad generation seed (i don't need to process the response)
first function (GET call) could also return HTTP429 which means the data is generating right now, so I need to call again after some time period (5-300 seconds) and check if data has been generated already
then after some time it results in HTTP200 with generated data which I want to map to SomeClass and then return mapped data in controller below
#PostMapping("/follow/{value}")
public Mono<ResponseEntity<?>> someFunction(#PathVariable int value) {
return Mono.just(ResponseEntity.ok(service.get(value)));
}
all the code I posted is very simplified to the issues I'm struggling with and doesn't contain some things I think are not important in this question
and now the actual question:
it doesn't actually make the call? i really don't know what is happening
program doesn't enter onRawStatus, even if i change it to onStatus 2xx or whatever other httpstatus and log inside i see nothing as if it doesn't even enter the chains
when i manually call with postman it seems like the program calls have never been made because the GET call returns 404 (the program didn't request to generate data)
the controller only returns "scanAvailable": true, when i expect it to return mapped SomeClass json
// edit
i changed the code to be a full chain as suggested and it didn't solve the problem. all the status are still unreachable (code inside any onStatus nor onRawStatus never executes)

How to save responses to database in spring boot

I have a simple post method that accepts temperature and checks is the temperature is greater than or equals to 37.4 and returns a response as cleared or not cleared.
#PostMapping(value = "/temperature")
public ResponseEntity<?> Temperature(#Valid #RequestBody Activation activation) throws UnsupportedEncodingException {
Temperature temperature = new Temperature();
temperature.setStatus(activation.getStatus());
temperature.setTemperature(activation.getTemp());
if (db_activation.getCode().equals(code)) {
temperature.setSource("VENUE");
temperatureRepository.save(temperature);
return ResponseEntity.ok(activation.getTemp() < 37.4 ? "Cleared" : "Not cleared");
}
}
how can I insert the response to Status column (which can be cleared or not cleared) to the database and will it be a post request or a put? Please suggest how to do that

To set the status, you can use something like below,
temperature.setStatus(activation.getTemp() < 37.4 ? "Cleared" : "Not cleared")
If you want to create the resource, POST should be the method, for updates, PUT should be used.

Choosing the right request type is just a matter of convention. Technically you could perform any operation with any type of request. By convention: POST sends some fresh data tonthe server, PUT sends an update of the server's data. So if it's a persisted data, POST would be "add" and PUT would be "update".
You could do it in the same method, like so:
if (activation.getTemp()< 37.4) {
temperature.setStatus("Cleared");
} else {
temperature.setStatus("Not cleared");
}
instead of this line:
temperature.setStatus(activation.getStatus());

REST - Download list of users

Context:
I want to write an endpoint that will return a Collection of users based on their usernames. How should those username be passed to the REST endpoint - note that I can (potentially) have a lot of usernames (say > 5000)?
Solution #1:
Use a GET endpoint, concatenate the usernames on client side and pass them as a single request parameter. Split the request parameter on server side to get the list of usernames.
#RestController
public class UserController {
#GetMapping
// able to deserialize `filename1,filename2` to List out of the box
public Collection<User> getUser(#RequestParam List<String> usernames) {
return userService.getUsersByUsername(usernames);
}
}
Solution #2:
Use a POST endpoint and pass the list of usernames as request body. Although cleaner form a coding perspective, I end up using a POST to fetch data.
#RestController
public class UserController {
#PostMapping
public Collection<User> getUser(#RequestBody List<String> usernames) {
return userService.getUsersByUsername(usernames);
}
}
Questions:
Which of the two solutions would be the better approach?
Do you have a better approach to pass the list of usernames into the endpoint?
Edits:
I've updated the signature of the first solution based on suggestions from answers. Spring is able to deserialize filename1,filename2 to List out of the box for #RequestParam.

POST looks like a cleaner approach in this case because -
Sending a huge string in a URL is not a good idea and there is scope for error
You need to write additional code (logic) to create the string on frontend and split it on backend.
Sending a huge string in a URL is not scalable as there are limits on the length of URL.

Get approach might result into an issue since URL length is limited and then you have to limit your query parameters.
Though its not a post request but in your case i think post is the only way out.

I would agree with all the answers given above. I would like to specify one more point , if you are going with post request you might have to increase the payload capacity a server can receive , the default post capacity(The maximum size in bytes) of spring boot is 2mb (based on your server). While testing your code might work fine with 1000-2000 usernames but make sure to change the property to accept more bytes in request.

GET is not limited, yet the browser is. Your server client does not seem to be the browser, so I would say GET is the way to go.
P.S GET can receive a body (not so great, but POST is not also the best match).
You don need to concatenated the string and add extra computation on server server, GET can receive a list of separate strings.
UPDATE with example:
#RestController
public class MyController {
#GetMapping(value = "/test")
public List<String> getTestParams(#RequestParam List<String> params) {
return params;
}
}
The test with 3000 params
#RunWith(SpringRunner.class)
#SpringBootTest(webEnvironment = SpringBootTest.WebEnvironment.RANDOM_PORT)
public class TestMyController {
#Autowired
private TestRestTemplate restTemplate;
#Test
public void testRequestWithParamsList() {
List<String> params = new ArrayList<>();
for (int i = 0; i < 3000; i++) {
params.add(String.valueOf(i));
}
List<String> result = restTemplate.getForObject(buildUrl(params),
List.class);
assertEquals(params, result);
}
private String buildUrl(List<?> params) {
return "/test?params=" + getUrlParameter(params);
}
private String getUrlParameter(List<?> params) {
return params.stream()
.map(Object::toString)
.collect(Collectors.joining(","));
}
}
If you are using tomcat you must specify also the max http header property in application.properties
server.max-http-header-size=30000

Array as QueryParam is not being processed in POST

I have the following Java REST method I implemented using Jersey:
#POST
#Path("copy")
public List<Integer> copyCompanionTextRule(#QueryParam("ruleid") List<Integer> ruleIdList,
#QueryParam("workgroupid") List<WorkgroupId> workgroupIds,
#Context HttpHeaders hh)
throws ETMSException
{
List<Integer> insertedItems = new ArrayList<Integer>();
if ( null != ruleIdList ){
for(Integer ruleId : ruleIdList) {
insertedItems.addAll(copyCompanionTextRule(ruleId, workgroupIds));
}
}
return insertedItems;
}
It receives a list of integer and a list of objects of type WorkgroupId as well as the context for some extra processing I'll do later.
I'm working the client with Sencha EXTJS 4.2 and my request is being performed this way:
Ext.Ajax.request({
url: '/sysadmin/companiontextrules/copy',
method: 'POST',
showException: true,
scope: this,
params: {
ruleid: Ext.encode(ruleIdsArray),
workgroupid: toWorkgroups
},
callback: function(options, success, response) {
me.setLoading(false);
if (!success) {
return;
}
this.destroy();
}
});
The ruleIdsArray is just an array of integers: [1274,1292,1745].
The toWorkgroups is an array of objects which has a model that is related to the WorkgroupId entity.
As you can see, both lists are being processed as query parameters and I'm using the "params" config in the Ajax request; however, this is not working.
Seems like the ruleId array is empty, when it tries to iterate the rulesIdList is empty so the method POST works but it is returning always an empty list.
I know I cannot use them in the form "url?ruleid=a&workgroupid=b". When I tried it just by curiosity, I got a QueryParamException and NumberFormatException saying that the rule array is being considered as string.
When I use the "Ext.encode" for both params I receive a message in browser console that the Maximum callstack size exceeded.
This is what I got from Chrome Console:
I've tried almost everything, but maybe some more eyes can help me in this, I'd really appreciate comments or any kind of help.
Thanks in advance.

Looks like your parameters are going in the POST body instead of as query parameters.
url?ruleid=a&workgroupid=b is getting a NumberFormatException because ruleId is supposed to be Integer.
url?ruleid=1&workgroupid=b or url?ruleid=1&ruleid=2&workgroupid=b should work

Java Jersey REST Request Parameter Sanitation

I'm trying to make sure my Jersey request parameters are sanitized.
When processing a Jersey GET request, do I need to filter non String types?
For example, if the parameter submitted is an integer are both option 1 (getIntData) and option 2 (getStringData) hacker safe? What about a JSON PUT request, is my ESAPI implementation enough, or do I need to validate each data parameter after it is mapped? Could it be validated before it is mapped?
Jersey Rest Example Class:
public class RestExample {
//Option 1 Submit data as an Integer
//Jersey throws an internal server error if the type is not Integer
//Is that a valid way to validate the data?
//Integer Data, not filtered
#Path("/data/int/{data}/")
#GET
#Produces(MediaType.TEXT_HTML)
public Response getIntData(#PathParam("data") Integer data){
return Response.ok("You entered:" + data).build();
}
//Option 2 Submit data as a String, then validate it and cast it to an Integer
//String Data, filtered
#Path("/data/string/{data}/")
#GET
#Produces(MediaType.TEXT_HTML)
public Response getStringData(#PathParam("data") String data) {
data = ESAPI.encoder().canonicalize(data);
if (ESAPI.validator().isValidInteger("data", data, 0, 999999, false))
{
int intData = Integer.parseInt(data);
return Response.ok("You entered:" + intData).build();
}
return Response.status(404).entity("404 Not Found").build();
}
//JSON data, HTML encoded
#Path("/post/{requestid}")
#POST
#Consumes({MediaType.APPLICATION_FORM_URLENCODED, MediaType.APPLICATION_JSON})
#Produces(MediaType.TEXT_HTML)
public Response postData(String json) {
json = ESAPI.encoder().canonicalize(json);
json = ESAPI.encoder().encodeForHTML(json);
//Is there a way to iterate through each JSON KeyValue and filter here?
ObjectMapper mapper = new ObjectMapper();
DataMap dm = new DataMap();
try {
dm = mapper.readValue(json, DataMap.class);
} catch (Exception e) {
e.printStackTrace();
}
//Do we need to validate each DataMap object value and is there a dynamic way to do it?
if (ESAPI.validator().isValidInput("strData", dm.strData, "HTTPParameterValue", 25, false, true))
{
//Is Integer validation needed or will the thrown exception be good enough?
return Response.ok("You entered:" + dm.strData + " and " + dm.intData).build();
}
return Response.status(404).entity("404 Not Found").build();
}
}
Data Map Class:
public class DataMap {
public DataMap(){}
String strData;
Integer intData;
}

The short answer is yes, though by "filter" I interpret it as "validate," because no amount of "filtering" will EVER provide you with SAFE data. You can still run into integer overflows in Java, and while those may not have immediate security concerns, they could still put parts of your application in an unplanned for state, and hacking is all about perturbing the system in ways you can control.
You packed waaaaay too many questions into one "question," but here we go:
First off, the lines
json = ESAPI.encoder().canonicalize(json);
json = ESAPI.encoder().encodeForHTML(json);
Aren't doing what you think they're doing. If your JSON is coming in as a raw String right here, these two calls are going to be applying mass rules across the entire string, when you really need to handle these with more surgical precision, which you seem to at least be subconsciously aware of in the next question.
//Is there a way to iterate through each JSON KeyValue and filter
here?
Partial duplicate of this question.
While you're in the loop discussed here, you can perform any data transformations you want, but what you should really be considering is using the JSONObject class referenced in that first link. Then you'll have JSON parsed into an object where you'll have better access to JSON key/value pairs.
//Do we need to validate each DataMap object value and is there a
dynamic way to do it?
Yes, we validate everything that comes from a user. All users are assumed to be trained hackers, and smarter than you. However if you handled filtering before you do your data mapping transformation, you don't need to do it a second time. Doing it dynamically?
Something like:
JSONObject json = new JSONObject(s);
Iterator iterator = json.keys();
while( iterator.hasNext() ){
String data = iterator.next();
//filter and or business logic
}
^^That syntax is skipping typechecks but it should get you where you need to go.
/Is Integer validation needed or will the thrown exception be good
enough?
I don't see where you're throwing an exception with these lines of code:
if (ESAPI.validator().isValidInput("strData", dm.strData, "HTTPParameterValue", 25, false, true))
{
//Is Integer validation needed or will the thrown exception be good enough?
return Response.ok("You entered:" + dm.strData + " and " + dm.intData).build();
}
Firstly, in java we have autoboxing which means this:
int foo = 555555;
String bar = "";
//the code
foo + bar;
Will be cast to a string in any instance. The compiler will promote the int to an Integer and then silently call the Integer.toString() method. Also, in your Response.ok( String ); call, THIS is where you're going to want to encodeForHTML or whatever the output context may be. Encoding methods are ALWAYS For outputting data to user, whereas canonicalize you want to call when receiving data. Finally, in this segment of code we also have an error where you're assuming that you're dealing with an HTTPParameter. NOT at this point in the code. You'll validate http Parameters in instances where you're calling request.getParameter("id"): where id isn't a large blob of data like an entire JSON response or an entire XML response. At this point you should be validating for things like "SafeString"
Usually there are parsing libraries in Java that can at least get you to the level of Java objects, but on the validation side you're always going to be running through every item and punting whatever might be malicious.
As a final note, while coding, keep these principles in mind your code will be cleaner and your thought process much more focused:
user input is NEVER safe. (Yes, even if you've run it through an XSS filter.)
Use validate and canonicalize methods whenever RECEIVING data, and encode methods whenever transferring data to a different context, where context is defined as "Html field. Http attribute. Javascript input, etc...)
Instead of using the method isValidInput() I'd suggest using getValidInput() because it will call canonicalize for you, making you have to provide one less call.
Encode ANY time your data is going to be passed to another dynamic language, like SQL, groovy, Perl, or javascript.