In my app i'm using code like the following to download several images.
Is it High-Performance to do it like that or can I reuse the connection somehow?
for(int i = 0; i < 100; i++){
URL url = new URL("http://www.android.com/image" + i + ".jpg");
HttpURLConnection urlConnection = (HttpURLConnection) url.openConnection();
try {
InputStream in = new BufferedInputStream(urlConnection.getInputStream());
readStream(in);
finally {
urlConnection.disconnect();
}
}
}
You won't really get any benefit from reuse of the HttpURLConnection.
One thing that will greatly benefit your application is if you spend some time looking into Async Tasks, which will allow you to harness the power of multi threaded HTTP requests with callbacks to your main code.
See:
http://www.vogella.com/articles/AndroidPerformance/article.html
for a good example of how Async Tasks can be utilised.
A good starting point though is of course the Android Developers Blog, where they have an example for downloading an image from a server asynchronously, which will match your requirements nicely. With some adaptation you can have your application sending multiple asynchronous requests at once for good performance.
The Google article can be found at:
http://android-developers.blogspot.co.uk/2009/05/painless-threading.html
The key area to look at is:
public void onClick(View v) {
new DownloadImageTask().execute("http://example.com/image.png");
}
private class DownloadImageTask extends AsyncTask {
protected Bitmap doInBackground(String... urls) {
return loadImageFromNetwork(urls[0]);
}
protected void onPostExecute(Bitmap result) {
mImageView.setImageBitmap(result);
}
}
The loadImageFromNetwork method is where the downloading takes place, and will be completely asynchronous away from your main UI thread.
As a basic example, you could modify your application to call this like so:
for(int i = 0; i < 100; i++){
new DownloadImageTask().execute("http://www.android.com/image" + i + ".jpg");
}
Though for an optimisation, I wouldn't throw 100 requests out at once, maybe creating a Threaded queue system which will allow maybe 4 or 5 concurrent connections and then keep the rest coming through when another finishes by maintaining an ArrayList of pending requests to read off.
No matter how you do it, you are going to end up opening multiple connections, one to get each image. This is how any image is received. And there's no way to change the HttpURLConnection anyways. So this could looks fine from that sense.
However, you could attempt to load multiple images at the same time through threading. It would be somewhat complex to implement such a scheme, but entirely possible. It would speed up the process by requesting multiple images at the same time.
Related
I'm trying to scrape live data from 50+ dynamic webpages and need the data to be updated every 1-2 seconds. To do so, I have a Timer scheduled every 1/2 second that iterates through the following method 50 times (for 50 URLs):
public double fetchData(String link) {
String data = null;
try {
URL url = new URL();
urlConn = url.openConnection(link);
InputStreamReader inStream = new InputStreamReader(urlConn.getInputStream());
BufferedReader buff = new BufferedReader(inStream);
/*code that scrapes webpage, stores value in "data"*/
inStream.close();
buff.close();
} catch (IOException e) {
e.printStackTrace();
}
return data;
}
This method works but takes about a second per URL, or 50 sec total. I've also tried JSoup in hopes that the delay may be overcome using the following code:
public double fetchData(String link, String identifier) {
Document doc;
String data = null;
try {
doc = Jsoup.connect(link).timeout(10*1000).get();
data = doc.getElementById(identifier).parent().child(0).text();
} catch (IOException e) {
e.printStackTrace();
}
return data;
}
but have run into approximately the same processing time. Are there any faster ways to draw data from dynamic webpages simultaneously, whether through URLConnection, JSoup, or some other method?
The short answer is "use threads". Create a thread for each of the 50+ URLs that you want to scrape repeatedly.
It will most likely make little difference if you use URLConnection, JSoup or some other way do the scraping. The actual bottleneck is likely to be due to:
load and performance of the load on the server(s) you are scraping from
network bandwidth
network latency
The first of those is outside of your control (in a positive way!). The last two ... you might be able to address but only by throwing money at the problem. For example, you could pay for a better network connection / path, or pay for alternative hosting to move your scraper close to the sites you are trying to scrape.
Switching to multi-threaded scraping will ameliorate some of those bottlenecks, but not eliminate them.
But I don't think what you are doing is a good idea.
If you write something that repeatedly re-scrapes the same pages once every 1 or 2 seconds, they are going to notice. And they are going to take steps to stop you. Steps that will be difficult to deal with. Things like:
rate limiting your requests
blocking your IPs or IP range
sending you "cease and desist" letters
And if that doesn't help, maybe more serious things.
The real solution may be to get the information a more efficient way; e.g. via an API. This may cost you money too. Because (when it boils down to it) your scraping will be costing them money for either no return ... or a negative return if your activity ends up reducing real peoples' clicks on their site.
I will try to make this a simple as possible (which is more than I can say for the Java HTTP setups I have seen).
I have a decision tree (pseudo) within my Activity:
private void okOnClick(View v){
if(HttpService.isCredentialValid()){
//wait to do something
} else {
//wait to do something else
}
}
Then I have an HttpService:
public class HttpService {
public static boolean isCredentialValid(){
//GET `http://my_server:8080/is-valid?someParam=123`
//the endpoint will return a 200 or 500
}
}
I don't want isCredentialValid to do any actions to the UI, I just want it to tell me, true or false.
I don't want to tightly couple it to a button.setText() or any of that, I just want a simple contract response.code == 200
In nearly every language this is not that difficult. Can someone please set me straight here.
...Sorry for any sounds of hostility. This is one of the most elementary mechanics of nearly every codebase I have ever worked with. And I am only finding Asynchronous patterns that have no way to return something substantial to the method caller. Or I am finding methods that endanger the main thread with no way to catch errors (when there is no connection for instance).
So far I have tried something like below (adjusted the code for simplification). I have allowed this to operate on the main thread, because I do want it to synchronously block. However there is no way to catch on the scenario for bad internet connection or when the remote server does not respond:
public static boolean isCredentialValid(){
String url = "http://my_server:8080?param=123";
StrictMode.ThreadPolicy policy = new
StrictMode.ThreadPolicy.Builder().permitAll().build();
StrictMode.setThreadPolicy(policy);
try {
OkHttpClient client = new OkHttpClient.Builder()
.connectTimeout(1, TimeUnit.SECONDS)
.writeTimeout(1, TimeUnit.SECONDS)
.readTimeout(1, TimeUnit.SECONDS)
.build();
Request request = new Request.Builder()
.url(url)
.build();
Response response = client.newCall(request).execute();
return response.code() == 200;
} catch(Exception e){
//
//THIS DOES NOT GET HIT WHEN THERE
//IS A BAD CONNECTION OR REMOTE SERVER FAILS TO RESPOND
//the app just hangs then quits
//
Log.d("ERROR:", e.toString());
return false;
}
}
First, you shouldn't execute your request on the main thread. Also, on Android HTTP requests are meant to be executed asynchronously, it's an extremely bad practice and a code smell if you need to execute them synchronously as a return for a method. The proper way of doing what you are trying to achieve is through the usage of the callback pattern. Your method shouldn't return anything, but invoke a callback which should be received as one of its parameters. If you still have the extreme necessity of doing things synchronously because either you don't know how to handle asynchronous calls or your architecture doesn't allow it, how about using a CountDownLatch? Excuse my Kotlin, but basically it works like this:
val countDownLatch = CountDownLatch(1)
// Execute your request
countDownLatch.countDown()
try {
countDownLatch.await(30, TimeUnit.SECONDS) // Give it a 30 seconds timeout
// return the response code here.
} catch (ex: InterruptedException) {
// Catch the timeout exception
}
Anyway, you should probably re-think about the necessity of actually returning from that method in particular instead of using callbacks, what I proposed isn't exactly the best practice.
PS: This code below is REALLY a bad idea. Basically what you are doing is forcing Android to allow HTTP requests on the main thread, which will completely block the UI of your application.
StrictMode.ThreadPolicy policy = new StrictMode.ThreadPolicy.Builder().permitAll().build();
StrictMode.setThreadPolicy(policy);
I have a following piece for loop in a function which I intended to parallelize but not sure if the load of multiple threads will overweight the benefit of concurrency.
All I need is to send different log files to corresponding receivers. For the timebeing lets say number of receivers wont more than 10. Instead of sending log files back to back, is it more efficient if I send them all parallel?
for(int i=0; i < receiversList.size(); i++)
{
String receiverURL = serverURL + receiversList.get(i);
HttpPost method = new HttpPost(receiverURL);
String logPath = logFilesPath + logFilesList.get(i);
messagesList = readMsg(logPath);
for (String message : messagesList) {
StringEntity entity = new StringEntity(message);
log.info("Sending message:");
log.info(message + "\n");
method.setEntity(entity);
if (receiverURL.startsWith("https")) {
processAuthentication(method, username, password);
}
httpClient.execute(method).getEntity().getContent().close();
}
Thread.sleep(500); // Waiting time for the message to be sent
}
Also please tell me how can I make it parallel if it is gonna work? Should I do it manual or use ExecutorService?
All I need is to send different log files to corresponding receivers. For the time being lets say number of receivers won't be more than 10. Instead of sending log files back to back, is it more efficient if I send them all parallel?
There are a lot of questions to be asked before we can determine if doing this in parallel will buy you anything. You mentioned "receivers" but are you really talking about different receiving servers on different web addresses or are all threads sending their log files to the same server? If it is the latter then chances are you will get very little improvement in speed with concurrency. A single thread should be able to fill the network pipeline just fine.
Also, you probably would get no speed up if the messages are small. Only large messages would take any time and give you any true savings if they were sent in parallel.
I'm most familiar with the ExecutorService classes. You could do something like:
ExecutorService threadPool = Executors.newFixedThreadPool(10);
...
threadPool.submit(new Runnable() {
// you could create your own Runnable class if each one needs its own httpClient
public void run() {
StringEntity entity = new StringEntity(message);
...
// we assume that the client is some sort of pooling client
httpClient.execute(method).getEntity().getContent().close();
}
}
});
What will be good is if you want to queue up these messages and send them in a background thread to not slow down your program. Then you could submit the messages to the threadPool and keep on moving. Or you could put them in BlockingQueue<String> and have a thread taking from the BlockingQueue and calling the httpClient.execute(...).
More implementation details from this good ExecutorService tutorial.
Lastly, how about putting all of your messages into one entity and divide the messages on the server. That would be the most efficient although you might not control the server handler code.
Hello ExecutorService is certainly an option. You have 4 ways to do it in Java.
Using Threads (exposes to many details easy to make mistake)
Executor service as you have already mentioned. It comes from Java 6
Here is a tutorial demonstrating ExecutorService http://tutorials.jenkov.com/java-util-concurrent/executorservice.html
ForkJoin framework comes from Java 7
ParallelStreams comes from Java 8 bellow is a solution using ParallelStreams
Going for higher level api will spare you some errors you might otherwise do.
receiversList.paralelstream().map(t->{
String receiverURL = serverURL + receiversList.get(i);
HttpPost method = new HttpPost(receiverURL);
String logPath = logFilesPath + logFilesList.get(i);
return readMsg(logPath);
})
.flatMap(t->t.stream)
.forEach(t->{
StringEntity entity = new StringEntity(message);
log.info("Sending message:");
log.info(message + "\n");
method.setEntity(entity);
if (receiverURL.startsWith("https")) {
processAuthentication(method, username, password);
}
httpClient.execute(method).getEntity().getContent().close();})
I want to draw five different routes in Google Maps API v3, GWT 2.5.1. I initialize a route which sets its DirectionDisplay and DirectionsRequest in this class.
When I start my web project, sometimes only my first route is shown, sometimes all five, so I decided to make a System.out.print(m);.
The results:
01234 -> as expected, all routes shown
10234 -> error, only first route shown.
Why does Google serve my second request before my first? I tried to use Thread.sleep(1000) to ensure that my requests have time to come back in order, also Timer/TimerTasks, no success. Any ideas?
DirectionsService o = DirectionsService.newInstance();
for (Integer i = 0; i < 4; i++) { //routes.size()
final int m = i;
final Route route = new Route("Route " + i.toString());
route.initRoute(m, getRoutingPresenter(), adressData, addressIndex);
//here i initialize the DirectionsRequests and its Displays, which
//i set in this class after execution.
o.route(directionsRequest, new DirectionsResultHandler() {
#Override
public void onCallback(DirectionsResult result,DirectionsStatus status) {
if (status == DirectionsStatus.OK) {
System.out.print(m);
...
}
}
);
}
}
Google can take as long as they like to handle your requests and you should code accordingly. This is true of any HTTP traffic. Even if the remote server guaranteed a fixed service time for all requests, the Internet does not and your requests could be taking any old route through it.
You can either right your handling code so that the response order doesn't matter, or write it so that it waits until all responses are back and then sort out the order itself.
I would recommend the first unless there are very specific an important reasons not to.
I have asked variations of this question over the past couple of days. I am not understanding or making the proper connections etc. so I am getting really frustrated not understanding which path I should be on.
I have developed a XAMLX WCF workflow service and my windows clients interact with it just fine. I am new to android and java and thus basically a NOOB. My service uses BasicHttpBinding and expects a couple of string values (uid & pwd) in the request. The response is simply a string.
Now on the android I am totally lost. I have read to use the AsyncTask, I've heard it has major issues and to use RoboSpice. Looking at RoboSpice it looks very solid but very stiff learning curve. I've heard for the stiff learning curve to use LoopJ as it will be easier when starting out. Throw in there the people who want you to use JSON and yeah. I'm definitely lost.
So I have been pulled in a lot of different directions. I know that I need it to be Async so it doesn't lock the ui thread and I would like to get to a solid robust solution but for now a baby step of just interacting with the service even synchronously would be encouraging.
Here are my basics:
Send a request to the service.
Have the application pause/loop etc. A progress bar would be wonderful but a simple loop that doesn't hang would be just fine for now.
RX the response and continue processing.
This is so simple for me in .NET I thought it would be simple. Just goes to show that pride cometh before the fall.
TIA
JB
Direct Question
As has been commented there are several implied questions in my OP. That is because I don't know what to ask. So let's go with AsynchTask for now. Let's call that my baby step.
Here are my questions:
HOW do I associate values to populate the request? In .NET I would create an object that matches what the request expects. So my object would have a UserName property and Password property. I would assign them their values like obj.UserName = "Joe"; I would then call the service operation and pass it obj.
HOW do I ensure I am using the proper libraries? I have read that AsyncTask uses the Apache Http Client library. OK I google that to get the jar's and apparently it's end of life. But still available so I download the latest and put it in my lib folder....but I didn't have to because I can import org.apache.http.HttpResponse without it so which version am I using then and should I use an older one or the latest.
**Finally how do I get the app to pause because this code does something (although it never shows in the logs of my service machine) and while it is off doing something my code continues to execute and crashes when it reaches the point where it needs the data from the service.....but it's not there. A progress bar would be the bees knees.
Here is my code for the implementation:
public class AsyncHttpRequestManager extends AsyncTask<String, Boolean, String>
{
private static final String serviceUrl = "http://www.serviceurl.com/service.xamlx";
#Override
protected String doInBackground(String... params)
{
try
{
DefaultHttpClient client = new DefaultHttpClient();
HttpPut putRequest = new HttpPut(serviceUrl);
putRequest.setHeader("Content-type", "text/plain");
putRequest.setHeader("Accept", "text/plain");
putRequest.setEntity(new StringEntity(params[0])); //This one is connected to my first question. How do I get the proper associations for the operations contract.
HttpResponse response = client.execute(putRequest);
InputStream content = response.getEntity().getContent();
BufferedReader buffer = new BufferedReader(new InputStreamReader(content));
String result = "";
String s = "";
while ((s = buffer.readLine()) != null)
{
result += s;
}
return result;
}
catch (UnsupportedEncodingException e)
{
Log.e("AsyncOperationFailed", e.getMessage());
e.printStackTrace();
} catch (ClientProtocolException e)
{
Log.e("AsyncOperationFailed", e.getMessage());
e.printStackTrace();
} catch (IOException e)
{
Log.e("AsyncOperationFailed", e.getMessage());
e.printStackTrace();
}
return null;
}
protected void onPostExecute(String result)
{
Log.d("MESSAGE", result);
}
}
I call this like this:
new AsyncHttpRequestManager().execute("joe","asdf");
If more info is needed I will gladly post.
You should ask one question at a time, it's how SO is designed, and curated, to work.
For HTTP authentication
http://developer.android.com/reference/org/apache/http/auth/UsernamePasswordCredentials.html
E.g.
HttpClient client = new HttpClient();
client.getParams().setAuthenticationPreemptive(true);
Credentials defaultcreds = new UsernamePasswordCredentials("username", "password");
client.getState().setCredentials(new AuthScope("myhost", 80, AuthScope.ANY_REALM), defaultcreds);
Asynctask does not require any external libraries it's part of the Android platform. Don't get confused about HTTPClient and Asynctask. Asyntask is a generic class intended to call background processing on a separate thread from the main UI thread to avoid blocking. It just so happens that you are considering using an Asynctask to process your web request but they are not linked in any way.
Your question about "how to return data to my app" is best posed separately but, in essence, you're thinking sequentially. Your app should not pause. There are many methods, progress dialogs being one, to properly handle this.