We Keep Coding

How to simulate timeout in response to a Rest request in Spring?

I have a Rest API implemented with Spring Boot 2. To check some client behavior on timeout, how can I simulate that condition in my testing environment? The server should regularly receive the request and process it (in fact, in production timeouts happen due to random network slowdowns and large big response payloads).
Would adding a long sleep be the proper simulation technique? Is there any better method to really have the server "drop" the response?

Needing sleeps to test your code is considered bad practice. Instead you want to replicate the exception you receive from the timeout, e.g. java.net.SocketTimeoutException when using RestTemplate.exchange.
Then you can write a test as such:
public class FooTest
#Mock
RestTemplate restTemplate;
#Before
public void setup(){
when(restTemplate.exchange(...)).thenThrow(new java.net.SocketTimeoutException())
}
#Test
public void test(){
// TODO
}
}
That way you wont be twiddling your thumbs waiting for something to happen.

Sleep is one way to do it, but if you're writing dozens of tests like that then having to wait for a sleep will create a really long-running test suite.
The alternative would be to change the 'threshold' for timeout on the client side for testing. If in production your client is supposed to wait 5 seconds for a response, then in test change it to 0.5 seconds (assuming your server takes longer than that to respond) but keeping the same error handling logic.
The latter might not work in all scenarios, but it will definitely save you from having a test suite that takes 10+ mins to run.

You can do one thing which I did in my case .
Actually in my case, when my application is running in a production environment, we keep on polling trades from the API and sometimes it drops the connection by throwing an Exception SSLProtocolException.
What we did
int retryCount =5;
while (true ){
count++;
try{
//send an api request here
}catch (Exception e){
if(retryCount == count ) {
throw e
// here we can do a thread sleep. and try after that period to reconnect
}
}
}
Similarly in your case some Exception it will throw catch that Exception and put your thread in Sleep for a while and after that again try for Connection the retryCount you can modify as per your requirment, in my case it was 5.

Establish 50+ URLConnections simultaneously within 1-2 seconds

I'm trying to scrape live data from 50+ dynamic webpages and need the data to be updated every 1-2 seconds. To do so, I have a Timer scheduled every 1/2 second that iterates through the following method 50 times (for 50 URLs):
public double fetchData(String link) {
String data = null;
try {
URL url = new URL();
urlConn = url.openConnection(link);
InputStreamReader inStream = new InputStreamReader(urlConn.getInputStream());
BufferedReader buff = new BufferedReader(inStream);
/*code that scrapes webpage, stores value in "data"*/
inStream.close();
buff.close();
} catch (IOException e) {
e.printStackTrace();
}
return data;
}
This method works but takes about a second per URL, or 50 sec total. I've also tried JSoup in hopes that the delay may be overcome using the following code:
public double fetchData(String link, String identifier) {
Document doc;
String data = null;
try {
doc = Jsoup.connect(link).timeout(10*1000).get();
data = doc.getElementById(identifier).parent().child(0).text();
} catch (IOException e) {
e.printStackTrace();
}
return data;
}
but have run into approximately the same processing time. Are there any faster ways to draw data from dynamic webpages simultaneously, whether through URLConnection, JSoup, or some other method?

The short answer is "use threads". Create a thread for each of the 50+ URLs that you want to scrape repeatedly.
It will most likely make little difference if you use URLConnection, JSoup or some other way do the scraping. The actual bottleneck is likely to be due to:
load and performance of the load on the server(s) you are scraping from
network bandwidth
network latency
The first of those is outside of your control (in a positive way!). The last two ... you might be able to address but only by throwing money at the problem. For example, you could pay for a better network connection / path, or pay for alternative hosting to move your scraper close to the sites you are trying to scrape.
Switching to multi-threaded scraping will ameliorate some of those bottlenecks, but not eliminate them.
But I don't think what you are doing is a good idea.
If you write something that repeatedly re-scrapes the same pages once every 1 or 2 seconds, they are going to notice. And they are going to take steps to stop you. Steps that will be difficult to deal with. Things like:
rate limiting your requests
blocking your IPs or IP range
sending you "cease and desist" letters
And if that doesn't help, maybe more serious things.
The real solution may be to get the information a more efficient way; e.g. via an API. This may cost you money too. Because (when it boils down to it) your scraping will be costing them money for either no return ... or a negative return if your activity ends up reducing real peoples' clicks on their site.

Are there major scaling limits with play framework and jdbc blocking io calls

I am using the playframework (2.4) for Java and connecting it to Postgres. The play framework is being used as a restful service and all it is doing is insert,updates,reads and deletes using JDBC. On this play page https://www.playframework.com/documentation/2.3.x/JavaAsync it states clearly that JDBC is blocking and that play has few threads. For the people who know about this, how limiting could this be and is there some way I can work around this? My specific app can have a few hundred database calls per second. I will have all the hardware and extra servers but do not know how play can handle this or scale to handle this in the code. My code in play looks like this:
public static Result myprofile() {
DynamicForm requestData = Form.form().bindFromRequest();
Integer id = Integer.parseInt(requestData.get("id"));
try {
JSONObject jo = null;
Connection conn = DB.getConnection();
ResultSet rs;
JSONArray ja = new JSONArray();
PreparedStatement ps = conn.prepareStatement("SELECT p.fullname as fullname, s.post as post,to_char(s.created_on, 'MON DD,YYYY') as created_on,s.last_reply as last_reply,s.id as id,s.comments as comments,s.state as state,s.city as city,s.id as id FROM profiles as p INNER JOIN streams as s ON (s.profile_id=p.id) WHERE s.profile_id=? order by created_on desc");
ps.setInt(1, id);
rs = ps.executeQuery();
while (rs.next()) {
jo = new JSONObject();
jo.put("fullname", rs.getString("fullname"));
jo.put("post", rs.getString("post"));
jo.put("city", rs.getString("city"));
jo.put("state", rs.getString("state"));
jo.put("comments", rs.getInt("comments"));
jo.put("id", rs.getInt("id"));
jo.put("last_reply", difference(rs.getInt("last_reply"), rs.getString("created_on")));
ja.put(jo);
}
JSONObject mainObj = new JSONObject();
mainObj.put("myprofile", ja);
String total = mainObj.toString();
System.err.println(total);
conn.close();
return ok(total);
} catch (Exception e) {
e.getMessage();
}
return ok();
}
I also know that I can try to wrap that in a futures promise however the blocking still occurs. As stated before I will have all the servers and the other stuff taken care of, but would the play framework be able to scale to hundreds of requests per second using jdbc? I am asking and learning now to avoid serious mistakes later on.

Play can absolutely handle this load.
The documentation states that blocking code should be avoided inside controller methods - the default configuration is tuned for them to have asynchronous execution. If you stick some blocking calls in there, your controller will now be waiting for that call to finish before it can process another incoming request - this is bad.
You can’t magically turn synchronous IO into asynchronous by wrapping
it in a Promise. If you can’t change the application’s architecture to
avoid blocking operations, at some point that operation will have to
be executed, and that thread is going to block. So in addition to
enclosing the operation in a Promise, it’s necessary to configure it
to run in a separate execution context that has been configured with
enough threads to deal with the expected concurrency. See
Understanding Play thread pools for more information.
https://www.playframework.com/documentation/2.4.x/JavaAsync#Make-controllers-asynchronous
I believe you are aware of this but I wanted to point out the bolded section. Your database has a limited number of threads that are available for applications to make calls on - it may be helpful to track this number down, create a new execution context that is turned for these threads, and assign that new execution context to a promise that wraps your database call.
Check out this post about application turning for Play, it should give you an idea of what this looks like. I believe he is using Akka Actors, something that might be out of scope for you, but the idea for thread tuning is the same:
Play 2 is optimized out-of-the-box for HTTP requests which don’t
contain blocking calls (i.e. asynchronous). Most database-driven apps
in Java use synchronous calls via JDBC so Play 2 needs a bit of extra
configuration to tune Akka for these types of requests.
http://www.jamesward.com/2012/06/25/optimizing-play-2-for-database-driven-apps
If you try to execute a massive number of requests on the database without turning the threads, you run the risk of starving the rest of your application of threads, which will halt your application. For the load you are expecting, the default tuning might be ok, but it is worth performing some additional investigating.
Getting started with thread tuning:
https://www.playframework.com/documentation/2.4.x/ThreadPools
You should update your controller to return Promise and there is also no reason to make it static anymore with Play 2.4. https://www.playframework.com/documentation/2.4.x/Migration24#Routing
Define an execution context in the application.conf with name "jdbc-execution-context"
//reference to context
ExecutionContext jdbcExecutionContext = Akka.system().dispatchers()
.lookup("jdbc-execution-context");
return promise(() -> {
//db call
}, jdbcExecutionContext)
.map(callResult -> ok(callResult));

Downloading html source code is slow

I'm using jsoup in my android app but the problem is, the html source takes too much time to download. Here is my code:
long t = System.currentTimeMillis();
String url = "http://www.stackoverflow.com/";
Document doc = null;
try {
Connection c = Jsoup.connect(url);
doc = c.get();
System.out.println(System.currentTimeMillis() - t);
} catch (IOException e) {
e.printStackTrace();
}
Executing this code takes 1.265 seconds which feels really weird because i can download the whole website (with images and all that good stuff) using web browser in less than a 0.5 seconds on the same device. Did I do something wrong? Or maybe there is a faster way for getting html source of website? Thanks in advance.

Where are you trying this code on? Your device? If you are using the LTE/3G network it wouldn't be too much off.
The other reason that I could think is that your wireless router is not situated in the best place from your device in case you are using Wifi.
From that code I don't see anything that could cause more delay. 1.2 secs may not be that bad if you dont have the host DNS entry cached and the server is far away from you.
Also, try setting the Agent to the same as your browser when comparing times. It may happen that the server gives different priorities based on the user agent. In this case you are using the default Java user agent.

How do I time an ajax query (send query, processing, receive response)

Edited to clarify my intent: (based on initial answers)
I have a web app. The server consists of a set of Java POJOs and I'm using Jersey to expose these as REST APIs. The browser calls these APIs, using jquery ajax, and does stuff.
I want to log the duration that my ajax queries take, and I want a break down by
How long it took to send the query from browser to server (into my Java POJO)
How long it took for the Java POJO to process the request
How long it took for response to be transmitted to browser (from POJO exit point to onComplete: entry point in javascript)
I'm also looking for a code based solution that I can apply systematically across my app.
So, 2 on its own is trivial, and timing the whole sequence is trivial. But I'm not sure about getting the breakdown for 1 & 3. I was initially going to pass the system time as a GET param, and compare to currentTimeMillis on the server - but this relies on the browser system time being in sync with the server system time. Unlikely to be reliable.
Are there any other suggestions?

See "How do you determine exactly when, accounting for network latency, a HTTP request was sent?", the answer explains how you can "calibrate" the client-side time with the server-side time. Note that this calibration will not be exact. Hope this helps.

Use Fiddler.

The answer is you can't. Certainly not from within a browser using Ajax.
The server could put the processing time in a custom response header, the client knows the send/receive times, from this it easy to derive the round trip time and assumes symmetrical link performance.
Going beyond that effectively means you want to measure TCP performance.

If this is only for a couple of pages and a couple of calls i would suggest using Network tab in Chrome Dev Tools. It provides plenty of info.

Is this something that you are looking for...
function myAjax(){
var timer = {};
timer["start"] = new Date().getTime();
if(window.XMLHttpRequest){
var req = new XMLHttpRequest();
}
else{
var req = new ActiveXObject("Microsoft.XMLHTTP");
}
req.onreadystatechange=function(){
if(req.readyState===2){
timer['recieved'] = new Date().getTime() - timer['start'];
}
if(req.readyState===3 && !timer['processing']){
timer['processing'] = new Date().getTime();
}
if(req.readyState===4 && req.status===200){
timer['processing'] = new Date().getTime() - timer['processing'];
timer['full'] = new Date().getTime() - timer['start'];
console.log(timer);
}
}
req.open("get", "myScript.php", true);
req.send();
}
Times are held in the timer variable. This obviously is not exact and may not even be what you are looking for, but is a simplified version of some AJAX times.