im having problems downloading html source from a web page in android. I run the http client in a different thread and it is able to get the html text ( i logged the result) but later when i try to work with the downloaded html text the variable seems to be empty from the main thread. I assume the problem is rising because im unable to synchronize threads but i don't know how to fix it for now. When i debug the code, the global variable contains data in the run function but when i join threads and look after the join method the variable is empty. Here is my code (class which i run in a different thread)
public class LutrijaHr {
public String url;
public String savedHtml;
public LutrijaHr(String s){
this.url = s;
savedHtml = "";
}
public String donwloadSource(String passedUrl) throws Exception{
HttpClient client = new DefaultHttpClient();
HttpGet request = new HttpGet(passedUrl);
HttpResponse response = client.execute(request);
String html = "";
InputStream in = response.getEntity().getContent();
BufferedReader reader = new BufferedReader(new InputStreamReader(in));
StringBuilder str = new StringBuilder();
String line = null;
while((line = reader.readLine()) != null)
{
str.append(line);
}
in.close();
html = str.toString();
savedHtml += html;
return html;
}
}
Parts of code from the main class:
String test = "";
LutrijaHr lhr = new LutrijaHr("https://www.lutrija.hr");
#Override
public void run() {
try {
test = lhr.donwloadSource(lhr.url);
} catch (Exception e) {
// TODO Auto-generated catch block
e.printStackTrace();
}
lhr.savedHtml = test;
Log.d("test", test);
}
Here is the part where i try to join the threads but the variable is empty
if (v.getId() == R.id.checkNumber){
Thread t = new Thread(new LotoMain(), "Page thread");
t.start();
try {
t.join();
etCheckedNumber.setText(lhr.savedHtml);
String smrki = test;
Log.d("testdsadasd", lhr.savedHtml);
Log.d("BOZO BOZO" ,test) ;
} catch (InterruptedException e) {
// TODO Auto-generated catch block
Log.d("BOZO BOZO BOZO" ,test) ;
e.printStackTrace();
}
}
I would like to solve this problem without using the android asynctask class because i want to learn a bit about threads and how they operate.
Use "Lock"
Add this code to main class:
public Lock workingLock = new ReentrantLock();
String test = "";
LutrijaHr lhr = new LutrijaHr("https://www.lutrija.hr");
#Override
public void run() {
try {
workingLock.lock();
test = lhr.donwloadSource(lhr.url);
} catch (Exception e) {
// TODO Auto-generated catch block
e.printStackTrace();
}
lhr.savedHtml = test;
workingLock.unlock;
Log.d("test", test);
}
Now use it in:
if (v.getId() == R.id.checkNumber){
Thread t = new Thread(new LotoMain(), "Page thread");
try {
try {
workingLock.lock();
} catch (Exception e) {
e.printStackTrace();
}
etCheckedNumber.setText(lhr.savedHtml);
String smrki = test;
Log.d("testdsadasd", lhr.savedHtml);
Log.d("BOZO BOZO" ,test) ;
workingLock.unlock();
} catch (InterruptedException e) {
// TODO Auto-generated catch block
Log.d("BOZO BOZO BOZO" ,test) ;
e.printStackTrace();
}
}
Try this, it will return the source of a given page as one long string which you can then manipulate however you need, and as its a standalone class/method you can call it on the UI thread or asyc or however you choose to.
import java.io.BufferedReader;
import java.io.IOException;
import java.io.InputStreamReader;
import java.net.URL;
import java.net.URLConnection;
public class WebPage {
public static String getWebSource(String Url) throws IOException {
URL url = new URL(Url);
URLConnection urlConnection = url.openConnection();
BufferedReader br = new BufferedReader(new InputStreamReader(
urlConnection.getInputStream(), "UTF-8"));
String inputLine;
StringBuilder sb = new StringBuilder();
while ((inputLine = br.readLine()) != null)
sb.append(inputLine);
br.close();
return sb.toString();
}
}
Edit: If you want to call it from the UI thread, android won't by default let you do that. you will need to change the apps thread policy which can by done by running this when the app starts (Required a min API of 9)
StrictMode.ThreadPolicy policy = new StrictMode.ThreadPolicy.Builder()
.permitAll().build();
StrictMode.setThreadPolicy(policy);
I "solved" the problem with declaring
String test = "";
as
static String test = "";
Even if this soultion works i don't understand why it wouldn't work with my original solution. If someone could light up this for me it would be really helpfull
Related
This question already has answers here:
Android "Only the original thread that created a view hierarchy can touch its views."
(33 answers)
Closed 4 years ago.
In my CountryInfoActivity.java I have an Async Class which retrieves a JSON from this website: https://pt.wikipedia.org/w/api.php?format=json&action=query&prop=extracts&exintro=&explaintext=&titles=Portugal.
It then parses the extract node into a String so I can set it in my TextView.
The problem is, whenever I set the text in the text view, my app crashes. The JSON parse is correct as it is retrieving all the information I want...
These are the classes I use to retrieve the data and, in the last one I try to set the data textoSobrePais into my TextView... By the way, in my onCreate method I called the class this way new DownloadTask().execute(url);
public class DownloadTask extends AsyncTask<String,Integer,Void>{
#Override
protected Void doInBackground(String... params) {
String url = params[0];
getJSONFromURL(url);
return null;
}
}
public String getJSONFromURL(String url){
String json_string = null;
try{
URL obj = new URL(url);
HttpURLConnection http = (HttpURLConnection) obj.openConnection();
http.setRequestMethod("GET");
int response = http.getResponseCode();
Log.i("response",Integer.toString(response));
BufferedReader reader = new BufferedReader(new InputStreamReader(http.getInputStream()));
StringBuilder sb = new StringBuilder();
String line = null;
while ((line = reader.readLine())!= null){
sb.append(line+"\n");
}
reader.close();
json_string = sb.toString();
Log.i("json_string",json_string);
} catch (UnsupportedEncodingException e){
e.printStackTrace();
} catch (ClientProtocolException e){
e.printStackTrace();
} catch (IOException e){
e.printStackTrace();
}
ParseJson(json_string);
return null;
}
public void ParseJson (String json){
JSONObject obj = null;
try {
obj = new JSONObject(json);
} catch (JSONException e) {
e.printStackTrace();
}
try {
JSONArray pageIdObj = obj.getJSONObject("query").getJSONObject("pages").names();
String page = String.valueOf(pageIdObj.get(0));
Log.i("ASdasd",page);
textoSobrePais = obj.getJSONObject("query").getJSONObject("pages").getJSONObject(page).getString("extract");
page = "";
Log.i("texte",textoSobrePais);
txtInfoPais = findViewById(R.id.txtInfoPais);
txtInfoPais.setText(textoSobrePais);
} catch (JSONException e) {
e.printStackTrace();
}
}
This is the error that is giving me when it crashes:
https://pastebin.com/PJh5r36u
Can somebody help, please?
We can't update UI from Background Thread. You have to set text on main thread
Run on main thread like this
runOnUiThread(new Runnable() {
#Override
public void run() {
txtInfo.setText(textoPais);
}
});
You cannot update UI components from non-UI Thread. Run the update of TextView on UI thread as follows:
runOnUiThread(new Runnable() {
#Override
public void run() {
txtInfoPais.setText(textoSobrePais);
}
});
Consider following
Code
private String url = "https://celestrak.org/NORAD/elements/resource.txt";
#Override
public Boolean crawl() {
try {
// Timeout is set to 20s
Connection connection = Jsoup.connect(url).userAgent(USER_AGENT).timeout(20 * 1000);
Document htmlDocument = connection.get();
// 200 is the HTTP OK status code
if (connection.response().statusCode() == 200) {
System.out.println("\n**Visiting** Received web page at " + url);
} else {
System.out.println("\n**Failure** Web page not recieved at " + url);
return Boolean.FALSE;
}
if (!connection.response().contentType().contains("text/plain")) {
System.out.println("**Failure** Retrieved something other than plain text");
return Boolean.FALSE;
}
System.out.println(htmlDocument.text()); // Here it print whole text file in one line
} catch (IOException ioe) {
// We were not successful in our HTTP request
System.err.println(ioe);
return Boolean.FALSE;
}
return Boolean.TRUE;
}
Output
SCD 1 1 22490U 93009B 16329.83043855 .00000228 00000-0 12801-4 0 9993 2 22490 24.9691 122.2579 0043025 337.9285 169.5838 14.44465946256021 TECHSAT 1B (GO-32) 1 25397U ....
I am trying to read an online-txt file (from https://celestrak.org/NORAD/elements/resource.txt). Problem is that while I print or save the body's text it prints whole online-txt file in one line. But I want to read it as splited by \n so that I can read it line by line. Am I making mistake while reading online-txt file?
I am using JSoup.
you can do it without using jsoup in the following manner:
public static void main(String[] args) {
String data;
try {
data = IOUtils.toString(new URL("https://celestrak.com/NORAD/elements/resource.txt"));
for (String line : data.split("\n")) {
System.out.println(line);
}
} catch (IOException e1) {
e1.printStackTrace();
}
}
the above code uses org.apache.commons.io.IOUtils
if adding the commons library is a issue you can use the below code:
public static void main(String[] args) {
URLReader reader;
try {
reader = new URLReader(new URL("https://celestrak.com/NORAD/elements/resource.txt"));
BufferedReader bufferedReader = new BufferedReader(reader);
String sCurrentLine;
while ((sCurrentLine = bufferedReader.readLine()) != null) {
System.out.println(sCurrentLine);
}
bufferedReader.close();
} catch (MalformedURLException e1) {
e1.printStackTrace();
} catch (IOException e) {
e.printStackTrace();
}
}
Since the file is already delimited by line separator, we can simple take the input stream from URL to read the contents
String url = "https://celestrak.com/NORAD/elements/resource.txt";
List<String> text = new BufferedReader(new InputStreamReader(new URL(url).openStream())).lines().collect(Collectors.toList());
To convert to a String
String content = new BufferedReader(new InputStreamReader(new URL(url).openStream())).lines()
.collect(Collectors.joining(System.getProperty("line.separator")));
What I want to do is to open an internet page in my browser (chrome) and get the html source code of the page just opened with my java application.
I don't want to get the source code of an url, I want a program that connects to the browser and gets the html code of the page that is open.
For example, if I open youtube in my browser, I want my application to get the current pages html code (in that case youtube code). Sorry if my english is not very good.
You can do this:
import java.util.*;
public static void main(String[] args) {
Scanner input = new Scanner(System.in);
URL url;
InputStream is = null;
BufferedReader br;
String line;
try {
String urlInput = input.nextLine();
url = new URL(urlInput);
is = url.openStream(); // throws an IOException
br = new BufferedReader(new InputStreamReader(is));
while ((line = br.readLine()) != null) {
System.out.println(line);
}
} catch (MalformedURLException mue) {
mue.printStackTrace();
} catch (IOException ioe) {
ioe.printStackTrace();
} finally {
try {
if (is != null) is.close();
} catch (IOException ioe) {
// nothing to see here
}
}
}
I got this from here: How do you Programmatically Download a Webpage in Java
Try this out:
You must pass in the URL as the argument and you'll have the HTML code
public static void main(String[] args) throws IOException {
URL u = null;
try {
u = new URL(args[0]);
} catch (MalformedURLException e) {
// TODO Auto-generated catch block
e.printStackTrace();
}
BufferedReader in = new BufferedReader(new InputStreamReader(u.openStream()));
String line = null;
while((line = in.readLine()) != null){
System.out.print(line);
}
}
I have got problem with read output form request.
public JSONArray listLights()
{
try
{
URL adres = new URL("https://api.lifx.com/v1/lights/all");
HttpURLConnection polaczenie = (HttpURLConnection) adres.openConnection();
polaczenie.setRequestProperty("Authorization", "Bearer " + apiKey);
polaczenie.setRequestMethod("GET");
BufferedReader wejscie = new BufferedReader(new InputStreamReader((polaczenie.getInputStream())));
StringBuilder odpowiedz = new StringBuilder();
String json;
while ((json = wejscie.readLine()) != null)
odpowiedz.append(json);
wejscie.close();
return new JSONArray(odpowiedz.toString());
}
catch (Exception wyjatek)
{
wyjatek.printStackTrace();
}
return new JSONArray();
}
StackTrace
I added to AndroidManifest Internet access too.
Welcome to leave any comments. :P
EDIT:
I google internet and found partial solution. Added AsyncTask, but now I'm receiving '429' response code.
public class JSONTask extends AsyncTask<String, String, String>
{
String apiKey = "blah_blah_blah";
String txtresult;
#Override
protected String doInBackground(String... params) {
HttpsURLConnection connection = null;
BufferedReader reader = null;
try
{
URL adres = new URL(params[0]);
HttpsURLConnection polaczenie = (HttpsURLConnection) adres.openConnection();
polaczenie.setRequestProperty("Authorization", "Bearer " + apiKey);
polaczenie.setRequestMethod("GET");
System.out.println(polaczenie.getResponseCode());
InputStream stream = polaczenie.getInputStream();
reader = new BufferedReader(new InputStreamReader(stream));
StringBuffer buffer = new StringBuffer();
String line = "";
while ((line = reader.readLine()) != null)
{
buffer.append(line);
}
return buffer.toString();
}
catch (MalformedURLException e)
{
e.printStackTrace();
}
catch (IOException e)
{
e.printStackTrace();
}
finally {
if (connection != null)
connection.disconnect();
try
{
if (reader != null)
reader.close();
}
catch (IOException e)
{
e.printStackTrace();
}
}
return null;
}
#Override
protected void onPostExecute(String s)
{
super.onPostExecute(s);
widok.setText(s);
}
}
My current StackTrace
EDIT2:
New day, new surprise. I figure out that I'm making connection with Bulb once/twice on every 10 attempts. Any ideas?
HTTP Status code 429 means too many requests in a given an amount of time. So how many requests exactly are you doing?
android.os.NetworkOnMainThreadException it means, that You have to make a htttp request from another threat than UIthread. Why are you using async task ?
Edit: You can also try make a call from postman and maybe You will see the problem.
In the end, everything is working. Problem was on the side of bulb or Lifx Cloud.
There's a website which provides some javascript which emits information I need, is there a way through androids APIs, to use this javascript to emit whatever information it has and parse through it?
This is the javascript:
<script type="text/javascript" src="http://pulllist.comixology.com/js/pulllist/5b467b28e73595311e02fe32c3081e4a.js?date=2013-05-15"></script>
If you run it in a html file you'll notice it displays images and text it retrieves from a website, I'd like to get all that information in an android app.
I don't know if you can directly run javascript code, but you can run it within a webview (maybe a hidden one?) and intercept javascript calls with a javascript interface, in combination with injection into the webview I think you should be able to do almost everything you want.
If you'd need it, you could also download the js source, parse it as you wish and then feed it to the webview. If you want more info let me know ^^
the web is full of example anyway, here an example and another one
EDIT
ok you don't need to actually run that javascript, parsing it is enough, here a working example in android implementing #T.S. parsing method, to run it you just need to add android.permission.INTERNET to your manifest file and having a textview with myTextView id set.
public class MainActivity extends Activity implements Runnable{
TextView myTextView;
public Handler handler = new Handler(){
#Override
public void handleMessage(Message msg){
String source = (String)msg.obj;
// The index of the item
int i = 0;
// We search the data-string from the start
int position1 = 0;
// While there are items found, continue
while(true) {
// Look for the pattern a[index]={json data};
String lookFor1 = "a[" + String.valueOf(i) + "]={";
String lookFor2 = "};";
position1 = source.indexOf(lookFor1, position1+1);
// Check if we have a match
if(position1 > -1) {
// Find the end of the match
int position2 = source.indexOf(lookFor2, position1);
// Get the result
String result = source.substring(position1 + lookFor1.length() - 1, position2 + 1);
// Increase the index an check if we can find a next item
i++;
// Print out this row, which is a JSON representation of the data you want
Log.e("res",result);
try {
JSONObject obj = new JSONObject(result);
String title = obj.getString("title");
String img = obj.getString("img");
String src = new JSONObject(img).getString("src");
myTextView.append("\ntitle: "+title+"\nimgurl: "+src+"\n");
} catch (JSONException e) {
// TODO Auto-generated catch block
e.printStackTrace();
}
} else {
// We haven't found a match, break out of while loop
break;
}
}
}
};
#Override
protected void onCreate(Bundle savedInstanceState) {
super.onCreate(savedInstanceState);
setContentView(R.layout.activity_main);
myTextView = (TextView) findViewById(R.id.myTextView);
Thread thread = new Thread(this);
thread.start();
}
#Override
public void run() {
String data = retrieve_data();
Message msg = handler.obtainMessage();
msg.obj = data;
handler.sendMessage(msg);
}
private String retrieve_data(){
String data = "";
String url = "http://pulllist.comixology.com/js/pulllist/5b467b28e73595311e02fe32c3081e4a.js?date=2013-05-15";
HttpClient httpclient = new DefaultHttpClient();
HttpGet request;
try {
request = new HttpGet(new URI(url));
request.addHeader("User-Agent", "Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:21.0) Gecko/20100101 Firefox/21.0");
HttpResponse response = httpclient.execute(request);
StatusLine statusLine = response.getStatusLine();
if (statusLine.getStatusCode() == HttpStatus.SC_OK) {
ByteArrayOutputStream out = new ByteArrayOutputStream();
response.getEntity().writeTo(out);
out.close();
data = out.toString();
}
} catch (URISyntaxException e1) {
// TODO Auto-generated catch block
e1.printStackTrace();
} catch (ClientProtocolException e) {
// TODO Auto-generated catch block
e.printStackTrace();
} catch (IOException e) {
// TODO Auto-generated catch block
e.printStackTrace();
} catch (Exception e) {
e.printStackTrace();
}
return data;
}
}
If it's just for displaying you could use the WebView with some custom styling.
Link: http://developer.android.com/reference/android/webkit/WebView.html
String html = "<script type='text/javascript' src='http://pulllist.comixology.com/js/pulllist/5b467b28e73595311e02fe32c3081e4a.js?date=2013-05-15'></script>";
String mime = "text/html";
String encoding = "utf-8";
WebView myWebView = (WebView)this.findViewById(R.id.myWebView);
myWebView.getSettings().setJavaScriptEnabled(true);
myWebView.loadDataWithBaseURL(null, html, mime, encoding, null);
Update
I wrote a quick and dirty example of how you can extract the data from that javascript file:
import java.util.*;
import java.lang.*;
import java.net.*;
import java.io.*;
public class JavascriptToUsableData {
// Method to get a string from an URL
// Thanks to http://stackoverflow.com/a/4328733/1226267
public static String getText(String url) throws Exception {
URL website = new URL(url);
URLConnection connection = website.openConnection();
BufferedReader in = new BufferedReader(
new InputStreamReader(
connection.getInputStream()));
StringBuilder response = new StringBuilder();
String inputLine;
while ((inputLine = in.readLine()) != null)
response.append(inputLine);
in.close();
return response.toString();
}
public static void main (String args[]) throws Exception {
// Get the data
String source = JavascriptToUsableData.getText("http://pulllist.comixology.com/js/pulllist/5b467b28e73595311e02fe32c3081e4a.js?date=2013-05-15");
// The index of the item
int i = 0;
// We search the data-string from the start
int position1 = 0;
// While there are items found, continue
while(true) {
// Look for the pattern a[index]={json data};
String lookFor1 = "a[" + String.valueOf(i) + "]={";
String lookFor2 = "};";
position1 = source.indexOf(lookFor1, position1+1);
// Check if we have a match
if(position1 > -1) {
// Find the end of the match
int position2 = source.indexOf(lookFor2, position1);
// Get the result
String result = source.substring(position1 + lookFor1.length() - 1, position2 + 1);
// Increase the index an check if we can find a next item
i++;
// Print out this row, which is a JSON representation of the data you want
System.out.println(result);
// I'm not in an Android environment right now, but there is a JSON reader you can use in Android
// I think it works somthing like this:
// resultObj = new JSONObject(result);
// And then access the data like this:
// resultObj.getString("title");
// resultObj.getString("guid");
// resultObj.getString("img");
// etc
} else {
// We haven't found a match, break out of while loop
break;
}
}
}
}
It can be optimized a lot and probably isn't fail-safe in it's current state, but it might give you some hints on how to start.