I am trying to extract the HTML content from a Webview. I found interesting subject on stackoverflow, but all of these answers loads the URL in order to get the HTML content. Here, I need to extract the HTML content of a webpage that has been generated from a POST method.
Using, the java method below, the HTML content loaded will just be (because it loads the url within the method, instead of directly extracting the html content from the webview)
<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 3.2 Final//EN">
<title>400 Bad Request</title>
<h1>Bad Request</h1>
<p>The browser (or proxy) sent a request that this server could not understand.</p>
private static class RetrieveHTML extends AsyncTask<String, String, String> {
private static String htmlContent;
protected String doInBackground(String... url) {
return getRemoteContent(url[0]);
}
protected void onProgressUpdate(Integer... progress) {
}
protected void onPostExecute(Long result) {
}
private static String getRemoteContent(String url)
{
HttpPost pageGet = new HttpPost(url);
HttpClient client = new DefaultHttpClient();
ResponseHandler<String> handler = new ResponseHandler<String>()
{
public String handleResponse(HttpResponse response) throws ClientProtocolException, IOException
{
HttpEntity entity = response.getEntity();
String html;
if (entity != null)
{
html = EntityUtils.toString(entity);
return html;
}
else
{
return null;
}
}
};
String pageHTML = null;
try
{
pageHTML = client.execute(pageGet, handler);
//if you want to manage http sessions then you have to add localContext as a third argument to this method and have uncomment below line to sync cookies.
//syncCookies();
}
catch (Exception e)
{
e.printStackTrace();
}
// you can filter your html content here if you wish before displaying
// in webview
try {
Log.d("TEST", pageHTML);
}
catch (Exception e){
e.printStackTrace();
}
htmlContent = pageHTML;
return pageHTML;
}
}
Thanks in advance
EDIT : I forgot to say why I am trying to do this :
I am adapting a Desktop website into an android application (mostly showing webview of mobile templates). I have a map into my desktop website, and markers are placed on it (those markers are transmitted via a json String through Flask+jinja). I got the idea to hide thoses markers in an html hidden tag. I could then extract the html and then parse the right part of this html content in order to get this json string into my java application (and then, use google-maps method thats exists in android studio)
Finally I decided to do what I wanted by an other way.
Everytime I do this post request, i generate a temp html file in which I write all the information that I need to get within my Java Application. I can then call this page from java (using the method above) because there is no data to re-send (since it is not a post-generated page)
Related
I've answered my own question here. See the first code block where I use php://input to get the posted data.
I'm trying to send a post request from my app to a webserver, and I'm checking for the post vars using PHP:
if( isset( $_POST['name'] ) ){
echo json_encode(['status' => 1]);
}else if( $posted_data = (string) #file_get_contents('php://input') ) {
echo json_encode(['status' => 2]);
}else{
echo json_encode($_POST);
}
The request always returns and empty json encoded array.
I'm using the latest Android Studio, and the latest OkHttp, com.squareup.okhttp3:okhttp:3.4.1. For me, this is like the "Hello World" of OkHttp in Android Studio.
In MainActivity.java:
public void postSomething(View view) {
String url = "https://example.com/json_api_test.php";
String json = "{\"name\":\"cholula\"}";
OkHttpPostHandler handler = new OkHttpPostHandler();
String result = "";
try {
result = handler.execute(url, json).get();
} catch (Exception e) {
e.printStackTrace();
}
displayPostResponse(result + "\n");
}
My OkHttpPostHandler.java:
public class OkHttpPostHandler extends AsyncTask<String, Void, String> {
OkHttpClient client = new OkHttpClient();
public static final MediaType JSON
= MediaType.parse("application/json; charset=utf-8");
#Override
protected String doInBackground(String... params) {
RequestBody body = RequestBody.create(JSON, params[1]);
Request request = new Request.Builder()
.url(params[0])
.post(body)
.build();
try {
Response response = client.newCall(request).execute();
return response.body().string();
}
catch( Exception e ){
return "HTTP Request Error";
}
}
}
When I debug, I can see that the params[1] value is the expected json-like string, but that's the last time I see it.
I've tried forming the json in a number of ways, so I'm not sure if that's the problem. I just don't understand why I can't see the posted vars when the post request gets to the server.
How can I see the posted vars on the webserver? What am I doing wrong? I've only been using Android Studio and Java for less than a week, so I have no clue. I've really looked around the internet a lot for the answer, and so posting here is the last resort.
Thanks!
It turns out that one must use php://input to get the posted data.
I have made my rest web service code to start sever like this :
static final String BASE_URI = "http://10.236.51.14:9000/abcd/";
public static void main(String[] args) {
try {
HttpServer server = HttpServerFactory.create(BASE_URI);
server.start();
System.out.println("Press Enter to stop the server. ");
System.in.read();
server.stop(0);
} catch (IllegalArgumentException e) {
e.printStackTrace();
} catch (IOException e) {
e.printStackTrace();
}
}
And in the rest web service I have made a basic code to receive 2 arguments and show their sum like this :
#GET
#Path("/add/{a}/{b}")
#Produces(MediaType.TEXT_XML)
public String add(#PathParam("a") double a, #PathParam("b") double b) {
return "<?xml version=\"1.0\"?>" + "<result>" + (a + b) + "</result>";
}
I want to send Json data (image) from my android app to this webservice but I don't know how to receive it in webservice and display it.
Here is the code from my android app. In this I have converted a bitmap to string using Base64. How should I send it to my webservice?
ByteArrayOutputStream baos = new ByteArrayOutputStream();
mybitmap.compress(Bitmap.CompressFormat.PNG, 100, baos);
byte[] b = baos.toByteArray();
String strBitMap = Base64.encodeToString(b, Base64.DEFAULT);
Any help will be appreciated :)
I have searched a lot but cant find appropriate code for my webservice to receive and display the json data. I am also struggling in sending this base64 string to the webservice in form of json.
Please help me out.
Best regards :)
I have a question: Does your example WebService work? I mean the one with the two arguments. If you call http://10.236.51.14:9000/abcd/add/1/2 in your browser does it display 3 correctly? If not you should have an ApplicationConfig containing your REST-interfaces. Those should be added as resource classes for example like this:
#ApplicationPath("api")
public class ApplicationConfig extends Application {
#Override
public Set<Class<?>> getClasses() {
Set<Class<?>> resources = new HashSet<>();
resources.addAll(addServiceClasses());
resources.addAll(addFilterClasses());
return resources;
}
private Set<Class<?>> addServiceClasses() {
// add all your REST-classes here
Set<Class<?>> resources = new HashSet<>();
resources.add(YourCalculatorRestServiceClass.class);
resources.add(YourImageConsumingRestServiceClass.class);
return resources;
}
private Set<Class<?>> addFilterClasses() {
// add all your filter classes here (if you have any)
Set<Class<?>> resources = new HashSet<>();
resources.add(YourAuthenticationFilterClass.class);
resources.add(OtherFilterClass.class);
return resources;
}
#Override
public Map<String, Object> getProperties() {
Map<String, Object> properties = new HashMap<>();
// in Jersey WADL generation is enabled by default, but we don't
// want to expose too much information about our apis.
// therefore we want to disable wadl (http://localhost:8080/service/application.wadl should return http 404)
// see https://jersey.java.net/nonav/documentation/latest/user-guide.html#d0e9020 for details
properties.put("jersey.config.server.wadl.disableWadl", true);
// we could also use something like this instead of adding each of our resources
// explicitly in getClasses():
// properties.put("jersey.config.server.provider.packages", "com.nabisoft.tutorials.mavenstruts.service");
return properties;
}
}
That should make the deal and you should be able to call http://10.236.51.14:9000/abcd/api/add/1/2. ApplicationConfig is annotated with #Path("api"). That means all classes registered in this config have the root path http://your.server.address/api/.
Now to your problem. I assume your server is working and you can reach your Webservice /add/1/2 displaying the result 3 in your browser.
Now you need another service listening for a POST. I'd take your already prepared String as the posted body.
#Path("image")
public class ImageReceiverRestService {
#POST
public Response checkAssignable(String base64ImageString) {
// code here working with the base64ImageString
// response code according to whatever happened during your algorithm
return Response.ok().build();
}
}
For appropriate HTTP response codes see this Wikipedia article for a quick overview HTTP Status Codes
So now you'd need a corresponding client on your android app. For example:
public class ImageSendingRestClient {
private final static String SERVER_BASE_URI = "http://10.236.51.14:9000/abcd/api/";
private final static String API_ADDRESS = "image/";
public ImageSendingRestClient() {
}
#Override
public void sendImageStringForProcessing(String base64ImageString) throws Exception {
Entity<String> entity = Entity.json(base64ImageString);
Response response = ClientBuilder.newClient()
.target(SERVER_BASE_URI)
.path(API_ADDRESS)
.request()
.post(entity);
try {
if (response.getStatus() == Response.Status.OK.getStatusCode()) {
return;
}
if (response.getStatus() == Response.Status.NOT_FOUND.getStatusCode()) {
throw new Exception;
}
} finally {
response.close();
}
}
}
All dependencies needed are JAX-RS implementations like JAX-RS reference implementation Jersey. Maybe you should also check the Jersey Guide with many examples providing most of the information you need Jersey User Guide
I am currently using this code to add a custom header to android WebView
Map<String, String> extraHeaders = new HashMap<String, String>();
extraHeaders.put("example", "header");
webView.loadUrl(url, extraHeader);
Above code is working but only on the main page. So if I write this code echo $_SERVER['example'] it prints header. But there is an iframe in the loaded URL which shows an undefined error when I try the same code. Is there any way I can fix this?
So what I want to do is add custom header not only to the main loaded URL but also on the iframe of the loaded page.
This worked for me:
mWebView.setWebViewClient(new MyWebViewClient(token));
in MyWebViewClient
#Override
public WebResourceResponse shouldInterceptRequest(WebView view, String url) {
try {
OkHttpClient httpClient = new OkHttpClient();
Request request = new Request.Builder()
.url(url.trim())
.addHeader("token", mToken) //add headers
.build();
Response response = httpClient.newCall(request).execute();
return new WebResourceResponse(
getMimeType(url), // set content-type
response.header("content-encoding", "utf-8"),
response.body().byteStream()
);
} catch (IOException e) {
return null;
}
}
//get mime type by url
public String getMimeType(String url) {
String type = null;
String extension = MimeTypeMap.getFileExtensionFromUrl(url);
if (extension != null) {
if (extension.equals("js")) {
return "text/javascript";
}
else if (extension.equals("woff")) {
return "application/font-woff";
}
else if (extension.equals("woff2")) {
return "application/font-woff2";
}
else if (extension.equals("ttf")) {
return "application/x-font-ttf";
}
else if (extension.equals("eot")) {
return "application/vnd.ms-fontobject";
}
else if (extension.equals("svg")) {
return "image/svg+xml";
}
type = MimeTypeMap.getSingleton().getMimeTypeFromExtension(extension);
}
return type;
}
You can put this setting to your web setting, every request from webview will be using this User-Agent header.
webview.getSettings().setUserAgentString("user-agent-string");
No, that is not possible with Android WebView itself. You have to work around either in your page code, or in your app's code, or on the server.
For fixing this on the page's side, you can use XMLHttpRequest for loading subresources. But for that you will have basically to construct the page on the fly.
On the app's side, you can use WebViewClient.shouldInterceptRequest, to intercept all the network requests. You are not allowed to just modify the provided request, instead, you will need to make a new request yourself, but there you will be able to set any headers you want. See this example: Android WebViewClient url redirection (Android URL loading system)
On the server side, you can look into Referer header of subresources, which must contain the url of the page that has requested it.
Just add this piece of code before load URL:
if (Build.VERSION.SDK_INT >= Build.VERSION_CODES.LOLLIPOP) {
CookieManager.getInstance().removeAllCookies(null);
}
I'm in the midst of testing my application which is using an HTTP-server. Instead of mocking I decided to go with a HTTP server fixture. Meaning that I do not have to mock any productional code. To accomplish this goal I currently chose for a free to use 3rd party library fixd.
I was able to successfully create several unit tests - which are working by means of a GET request. Most are quite simple, i.e.:
#Test
public void verifyConnectionTest()
{
try
{
final String body = FileUtils.readFileToString(RESOURCE);
final String path = "/";
this.server.handle(Method.GET, path).with(
new HttpRequestHandler() {
#Override
public void handle(final HttpRequest request,
final HttpResponse response)
{
response.setStatusCode(200);
response.setContentType("text/xml");
response.setBody(body);
}
});
// Setting up my HTTP client
// Execute some tasks
// asserting of everything was valid
}
catch (final IOException e)
{
fail(e.getMessage());
}
}
But I now have to send a POST request with multipart/form-data. Which does not make much of a difference other than changing the method and content-type:
#Test
public void executeStepTest()
{
try
{
final String body = FileUtils.readFileToString(SERVICE_RESPONSE);
final String path = "/";
this.server.handle(Method.POST, path, "multipart/form-data").with(
new HttpRequestHandler() {
#Override
public void handle(final HttpRequest request,
final HttpResponse response)
{
response.setStatusCode(200);
response.setContentType("text/xml");
response.setBody(body);
}
});
// Setting up my HTTP client
// Execute some tasks
// asserting of everything was valid
}
catch (final IOException e)
{
fail(e.getMessage());
}
}
However I get the following error: [ERROR] could not find a handler for POST - / - multipart/form-data; boundary=bqCBI7t-VW1xaJW7BADmTiGMg9w_YM2sHH8ukJYx and my guess is that fixd doesn't recognize the boundary-party. Since the documentation does not show an example I'm quite stuck on this part.
I tried using some wildcards such as '*', no succes. Thus; I need a way to either tell fixd to accept that boundary or use some wildcards I didn't yet discover. Any help would be great, thanks!
I've been making some debug and it seems to be that the problem is in the fixd core.
Basically, fixd indexes every RequestHandlerImpl by a HandlerKey (which includes ContentType as part of the key) in the map handlerMap. See method org.bigtesting.fixd.core.FixtureContainer#resolve.
...
HandlerKey key = new HandlerKey(method, route, contentType);
RequestHandlerImpl handler = handlerMap.get(key);
if (handler == null) {
// Error
}
...
Problem: When the request is multipart/form-data, boundary data (which it's generated dinamically every request) is part of the content type. So, any handler is found in handlerMap because the key changes with every running.
I've made a little test only to check that this is the cause of the problem, passing the contentType to fixd server.handle after the creation of the multipart request, and it works fine.
See the test below:
#Test
public void verifyConnectionTest_multipart() {
try {
// 1. Create multipart request (example with http-commons 3.1)
PostMethod filePost = new PostMethod(url);
Part[] parts = { new StringPart("param", "value") };
MultipartRequestEntity request = new MultipartRequestEntity(parts, filePost.getParams());
filePost.setRequestEntity(request);
// 2. fixd server handle (passing the request content type)
this.server.handle(Method.POST, "/", request.getContentType()).with(
new HttpRequestHandler() {
#Override
public void handle(final HttpRequest request,
final HttpResponse response) {
response.setStatusCode(200);
response.setContentType("text/xml");
}
});
// 3. Execute multipart request
HttpClient client = new HttpClient();
int status = client.executeMethod(filePost);
// 4. Assertions
Assert.assertEquals(200, status);
} catch (Exception e) {
Assert.fail(e.getMessage());
}
}
Hope it helps you to clarify the problem. Cheers
This was a bug in fixd, and has been fixed in version 1.0.3. Your original code should work using this new version of fixd.
I've seen multiple posts about this topic, but none of them seem to be the solution to my problem.
The problem is that the JSON response from the server is getting cut off and therefore I'm getting a JSONException when trying to get the response into a JSONArray.
json = new JSONArray(EntityUtils.toString(response.getEntity()));
Here is the whole code:
private class AsyncFetchForms extends AsyncTask<String, Void, JSONArray> {
private HttpClient mClient = new DefaultHttpClient();
private AsyncTaskCompleteListener<JSONArray> listener;
private String serverUrl;
private String credentials;
private ProgressDialog progressDialog;
private HttpGet httpGet;
private String response;
private BasicResponseHandler responseHandler;
private boolean showDialog;
private JSONArray json;
public AsyncFetchForms(String url, String message, AsyncTaskCompleteListener<JSONArray> listener, boolean showDialog)
{
serverUrl = Utils.getServerUrl(context) + url;
credentials = Utils.getUserCredentials(context);
this.listener = listener;
this.showDialog = showDialog;
httpGet = new HttpGet(serverUrl);
httpGet.setHeader("Authorization", "Basic " + credentials);
httpGet.setHeader("Accept", "application/json");
httpGet.setHeader("Accept-Encoding", "gzip");
httpGet.setHeader("Connection", "keep-alive");
responseHandler = new BasicResponseHandler();
if(showDialog)
{
progressDialog = new ProgressDialog(context);
progressDialog.setMessage(message);
progressDialog.setProgressStyle(ProgressDialog.STYLE_SPINNER);
progressDialog.show();
}
}
#Override
protected JSONArray doInBackground(String... params) {
try {
HttpResponse response = mClient.execute(httpGet);
if (response.getStatusLine().getStatusCode() == 200) {
json = new JSONArray(EntityUtils.toString(response.getEntity()));
return json;
}
} catch (ClientProtocolException e) {
e.printStackTrace();
} catch (IOException e) {
e.printStackTrace();
} catch (ParseException e) {
e.printStackTrace();
} catch (JSONException e) {
e.printStackTrace();
}
return null;
}
#Override
protected void onPostExecute(JSONArray result) {
System.out.println(result.toString());
}
}
Can anyone see the problem?
Logcat can only show about 4000 characters. So you will have to implement a recursive function to see the entire log. Use below function to see the entire log:
public static void longInfo(String str) {
if (str.length() > 4000) {
Log.d("", str.substring(0, 4000));
longInfo(str.substring(4000));
} else
Log.d("", str);
}
If you (or your team) implement the server side yourself, first thing I'd check is if the server is returning the correct HTTP response. In particular, if you transfer the data by HTTP, you need to have correct Content-Length or otherwise your data will be cut off. Also, Content-Length must be the length of data after any Transfer Encodings are applied, in other words, after the length of the data after being gzipped. Alternatively, use chunked transfer.
Second, make sure that your server is generating valid JSON. Maybe you missed a closing parentheses or so. Maybe you need to parse JSON Object rather JSON Array.
Also, if you receive exceptions, please always post the the entire traceback.
First of all, try to log the EntityUtils.toString(response.getEntity()) response and make sure that it starts with "[" not "{" ie. it's a jsonArray not jsonObject.
Then try to open the url in your browser ,if avilable, and make sure that there are no encoding issues.
Finally, if the problem is still exists please send us the error log output.
This answer is completely out of the subject but :
What are you trying do here ? Do you know that there are libraries which are doing all this boring job for you ?
When I talk about boring job, I'm talking about managing all the background running stuff (like AsyncTask), JSON decoding and HTTP response. I know that it's sometimes a pain in the a** (I've been there) but now I've choose to not worry anymore and use a dedicated library : http://square.github.io/retrofit/
This little baby will contact the Webservice of your choice, download the JSON and put it into a custom java class with all the attributes you want to deal with.
If you plug it with something like ORMLite, it can even save your JSON response object into a SQLite DB that you can access in the same way (it "populates" a java class object by setting all the attributes for you).
Personally I can't imagine myself doing all this stuff by hand anymore, it's just trouble without the benefits =)