java.lang.arrayindexoutofboundsexception jsoup - java

I'm trying to pull all images from a website and
analyze each one using AWS image recognition API. It works for some websites, however some websites return an error saying `500 server error java.lang.arrayindexoutofboundsexception index:281 size 281.
Basically I'm scraping images using jsoup and then creating an object to store the name and image URL for each image. After that, I call the API and check each image in the ArrayList. For some reason it only works for some websites.
Can someone please explain what I'm doing wrong and how to prevent this error?
#WebServlet(name = "HelloAppEngine", urlPatterns = {
"/hello"
})
public class HelloAppEngine extends HttpServlet {
static ArrayList < ResponseData > testImages = new ArrayList < > ();
static AmazonRekognition rekognitionClient = AmazonRekognitionClientBuilder.defaultClient();
public static void getimages() throws MalformedURLException, IOException {
System.out.println("getImages called" + testImages);
int index = 0;
for (ResponseData data: testImages) {
System.err.println("open stream for:" + data.getUrl());
ByteBuffer imageBytes = null;
try (InputStream inputStream = new URL(data.getUrl()).openStream()) {
System.out.println(inputStream);
imageBytes = ByteBuffer.wrap(IOUtils.toByteArray(inputStream));
System.out.println(imageBytes);
} catch (IOException e1) {
System.err.println(e1.getMessage());
}
//
DetectLabelsRequest request = new DetectLabelsRequest().withImage(new Image().withBytes(imageBytes)); //.withMaxLabels(10).withMinConfidence(77F);
try {
DetectLabelsResult result = rekognitionClient.detectLabels(request);
List < Label > labels = result.getLabels();
//System.out.println(labels);
//System.out.println("Detected labels for " + photo+""+labels);
for (Label label: labels) {
//loop through all labels of object
//create new responsedata object for each image
//where im getting error
if (testImages.get(index) != null) {
ResponseData d = testImages.get(index);
d.setName(label.getName());
testImages.set(index, d);
//increment for making new image url and name
index++;
System.out.println(label.getName() + ": " + label.getConfidence().toString());
}
}
//
} catch (AmazonRekognitionException e) {
System.err.println(e.getMessage());
}
}
}
private static final long serialVersionUID = 1 L;
protected static final Gson GSON = new GsonBuilder().create();
// This is just a test array
ArrayList < String > list = new ArrayList < String > ();
#Override
protected final void doPost(HttpServletRequest req, HttpServletResponse resp) throws ServletException, IOException {
resp.setContentType("text/json");
String servlet = req.getServletPath();
System.setProperty("http.proxyHost", "192.168.5.1");
System.setProperty("http.proxyPort", "1080");
log("servlet:" + servlet);
if (servlet.equalsIgnoreCase("/main")) {
log("if body start");
String urlString = java.net.URLDecoder.decode(req.getParameter("url"), "UTF-8");
// Connect to website. This can be replaced with your file loading
// implementation
Document doc = Jsoup.connect(urlString).get();
// Get all img tags
Elements img = doc.getElementsByTag("img");
Elements media = doc.select("[src]");
int counter = 0;
// Loop through img tags
for (Element src: media) {
if (src.tagName().equals("img")) {
counter++;
//create reposnsedata object for each image url
ResponseData data = new ResponseData();
//set object url to image url
data.setUrl(src.attr("abs:src"));
//set data name from aws
data.setName(" ");
testImages.add(data);
// getimages();
}
if (src.tagName().equals("link[href~=.*\\.(ico|png)]")) {
System.out.println("image is logo");
}
if (src.tagName().equals("meta[itemprop=image]")) {
System.out.println("image is logosss");
}
}
}
//log("list" + testImages);
getimages();
//
// getimages();
System.err.println(GSON.toJson(testImages));
resp.getWriter().println(GSON.toJson(testImages));
}
#Override
protected final void doGet(HttpServletRequest req, HttpServletResponse resp) throws ServletException, IOException {
doPost(req, resp);
}
}

You're trying to get 282nd image (index=281) from testImages but there's only 281 (index=280). You're getting each image for each label and it's possible there's more labels than images.
Try displaying the amount of both of them:
System.out.println("testImages.size() is: " + testImages.size());
System.out.println("labels.size() is: " + labels.size());
To avoid getting more images than labels try replacing this condition:
if (testImages.get(index) != null) {
with
if (index < testImages.size() && testImages.get(index) != null) {

Related

Passing text and multiple images as an array using okhttp

I am using Okhttp lib to send data to server. I want to set text and images in RequestBody. For uploading multiple image to server using Okhttp i follow this link. I have implemented that type of code in my app in another activity class and its work fine. I have checked this question that how to pass array in RequestBody.
My arrayList format is like this
blockList
{
block0
{
description0 = First block
image0 = {image1, image2}
video0 = videolink
disp_order0 = 0
block0 = 0
}
block1
{
description1 = second block
image1 = {image1,image2,image2}
video1 = videolink
disp_order1 = 1
block1 = 1
}
.....
}
My Requirement :-
Now i want to send multiple images as an array in single parameter. When i send first block then parameter names are description0,image0[],video1,disp_order and block0 and image0[] will contain first block images as array and same for other.
API is working fine because when i test in postman then i receive the data in server side. You can see in below..
Here is my java function that set the data in RequestBody and make a call to send that data on sever.
ProgressDialog pd;
private OkHttpClient client;
private void saveCastBoxOnServer(String castBoxTitle, String selectedCastBoxId, String selectedCategoryId,
String userId, String action, ArrayList<CastBoxBlock> blockList)
{
try
{
client = new OkHttpClient.Builder()
.retryOnConnectionFailure(true)
.build();
ArrayList<CastBoxBlock> blockArrayList = blockList;
int blockSize = blockArrayList.size();
MultipartBody.Builder multipartBuilderNew = new MultipartBody.Builder().setType(MultipartBody.FORM);
for (int i = 0; i < blockSize; i++)
{
String description = blockArrayList.get(i).getBlockDescription();
String descriptionField = "description"+i;
multipartBuilderNew.addFormDataPart(descriptionField, description);
/**This is used for distribution of images and videos. After that set that
* Images and video in multipartBuilder.
**/
CastBoxBlock model = blockArrayList.get(i);
ArrayList<SelectedMediaModel> mediaModels = model.getSelectedMediaModelArrayList();
int mediaModelsSize = mediaModels.size();
String passingVideoUri = "";
String videoUri = "";
for (int j = 0; j < mediaModelsSize; j++)
{
String mediaType = mediaModels.get(j).getMediaType();
if (mediaType.equals(StringKeyConstant.mediaVideo))
{
videoUri = mediaModels.get(j).getMediaPath();
if (passingVideoUri.trim().length()==0){
passingVideoUri = videoUri;
}else{
passingVideoUri = passingVideoUri + "," + videoUri;
}
}
else if (mediaType.equals(StringKeyConstant.mediaImage))
{
String imagePath = mediaModels.get(j).getMediaPath();
File sourceFile = new File(imagePath);
/**Changes whether JPEG or PNG**/
final MediaType MEDIA_TYPE = MediaType.parse(
constant.getFileExt(imagePath).endsWith("png") ? "image/png" : "image/jpeg");
String imageName = System.currentTimeMillis() + j + "_block_img.jpg";
String imageField = "image"+i+"["+j+"]";
multipartBuilderNew.addFormDataPart(imageField,imageName,
RequestBody.create(MEDIA_TYPE, sourceFile));
}
}
/**This is used to set the {#videoUri} block of videos and send to sever**/
String videoField = "video"+i;
multipartBuilderNew.addFormDataPart(videoField, passingVideoUri);
/**This will set the {#display_order} in multipartBuilder**/
String displayOrderField = "disp_order"+i;
String displayOrder = blockArrayList.get(i).getBlockIndex();
multipartBuilderNew.addFormDataPart(displayOrderField, displayOrder);
/**This will set the {#block} value in multipartBuilder**/
String blockField = "block"+i;
String block = ""+i;
multipartBuilderNew.addFormDataPart(blockField, block);
}
pd = new ProgressDialog(activity);
pd.setCancelable(false);
pd.setMessage(getResources().getString(R.string.please_wait));
pd.show();
RequestBody formBody = multipartBuilderNew
.addFormDataPart("cast_title", castBoxTitle)
.addFormDataPart("user_id", userId)
.addFormDataPart("cast_box", selectedCastBoxId)
.addFormDataPart("category", selectedCategoryId)
.addFormDataPart("action", action)
.build();
Request request = new Request.Builder()
.url(ApiUtils.ADD_FETCH_USER_CAST)
.post(formBody)
.build();
client.newCall(request).enqueue(new Callback() {
#Override
public void onFailure(Call call, IOException e) {
Log.e(TAG, "Get Api credential fail."+call.toString());
e.printStackTrace();
}
#Override
public void onResponse(Call call, Response response) throws IOException
{
try
{
if (pd != null){
pd.cancel();
pd.dismiss();
pd = null;
}
String castSavedResponse = response.body().string();
Log.i(TAG, "castSavedResponse = " + castSavedResponse);
}
catch (Exception e){
Log.e(TAG, "***Error : onResponse() method");
e.printStackTrace();
}
}
});
}
catch (Exception e){
Log.e(TAG, "***Error : saveCastBoxOnServer()");
e.printStackTrace();
}
}
Thanks in advance. If any one will help it would be very appreciate.
When i check your code then i note one things. I am no sure but i think you change your code on image[] place. you can just change your code as below. I hope this will help you and you got the solution.
Use this code
String imageField = "image"+i+"[]";
Instead of
String imageField = "image"+i+"["+j+"]";
when i passed simple images as an array in Okhttp then i do code as above. For uploading multiple images on server using Okhttp, i also follow this link as you follow.

Printing issue with internet download manager

I want to print a document in my java web application using servlet and i clear report info to avoid download resume(SecurityContext.cleanReportInfo()).
everything is ok when you don't have IDM in your system but when IDM is running it sends 2 request instead 1 and everything goes wrong by redirecting to /403 like the code below .
PrintReportServlet
protected void service(HttpServletRequest request, HttpServletResponse response) throws ServletException, IOException {
ReportDto reportInfo = SecurityContext.getReportInfo();
if (reportInfo != null) {
ServletOutputStream outputStream = response.getOutputStream();
try {
generateHeaders(response, reportInfo);
generateContents(outputStream, reportInfo);
} catch (Exception e) {
LOGGER.log(Level.WARNING, "Error occurred during print report, nested error message: {0}", e.getMessage());
} finally {
outputStream.flush();
outputStream.close();
SecurityContext.cleanReportInfo();
}
} else {
response.sendRedirect(request.getContextPath() + "/403");
}
}
SecurityContext.java
public static void cleanReportInfo() {
SecurityContext.getCurrentSession().setAttribute("Report_Info", null);
}
errorreportController.js
$scope.printErrorList = function () {
errorReportService.printErrorList(function () {
window.open(contextPath + "/PrintReport");
});
};
ErrorReportController.java
#RequestMapping(value = "/printError", method = RequestMethod.GET)
#ResponseBody
public String printError() {
ReportDto reportDto = errorReportService.getReportInfo();
SecurityContext.setReportInfo(reportDto);
return success();
}
ErrorReportServiceImpl
public ReportDto getReportInfo() {
Map<String, Object> parameters = new HashMap<String, Object>();
parameters.put("jalaliDate", DateUtil.convertToJalali(new Date()).toStringBySlash());
parameters.put("alahImagePath", ReportUtil.getImagesPath() + File.separator + "alah.png");
parameters.put("sepahImagePath", ReportUtil.getImagesPath() + File.separator + "sepah.png");
parameters.put("reporter", SecurityContext.getCurrentUserName());
parameters.put("classify", "Normal");
parameters.put("userId",SecurityContext.getCurrentUserId());
ReportDto reportDto = new ReportDto(
ReportNames.ErrorReport, DataItems.Report_Type_PDF,
"error",
parameters);
return reportDto;
}

Images(base64) not uploading correctly

Unfortunately I'm facing some issues when I try to upload some images from an android device to a database.
The images are in a folder. This folder contains images as well as other stuff. I don't know the names of the images and I need to upload only the images(jpg). Before I upload the images I need to encode them with base64.
First I get the jpg files from the folder. Then I get the ID out of the image name. After that I encode it via base64:
Button upload = (Button) findViewById(R.id.upload);
upload.setOnClickListener(new View.OnClickListener() {
public void onClick(View arg0) {
String path = Environment.getExternalStorageDirectory().getPath();
File dir = new File(path);
File[] files = dir.listFiles();
for (int i = 0; i < files.length; ++i) {
if (files[i].getName().endsWith(".jpg")) {
pics = new File(String.valueOf(files[i]));
id = String.valueOf(files[i]);
String sub = id.substring(id.lastIndexOf("/") + 1);
int index = sub.indexOf("_");
String book;
if (index >= 0) {
book = sub.substring(0, index);
ID = book;
Log.e("ID", ID);
}
Bitmap imagex = BitmapFactory.decodeFile(pics.getAbsolutePath());
ByteArrayOutputStream baos = new ByteArrayOutputStream();
imagex.compress(Bitmap.CompressFormat.JPEG, 70, baos);
byte[] b = baos.toByteArray();
Image = Base64.encodeToString(b, Base64.DEFAULT);
try {
new HttpAsyncTask(ID,Image,Nummer).execute("https://....");
} catch (Exception e) {
Log.e("InputStream", e.getMessage());
}
Log.e("PICS", id);
}
}
}
});
public String POST(String url) {
InputStream inputStream;
try {
HttpClient httpclient = classxy.getNewHttpClient();
HttpPost httpPost = new HttpPost(url);
String json = "";
JSONObject jsonObject = new JSONObject();
jsonObject.put("bookId", ID);
jsonObject.put("imageString", Image);
jsonObject.put("imageNumber", Nummer);
json = jsonObject.toString();
StringEntity se = new StringEntity(json);
httpPost.setEntity(se);
httpPost.setHeader("Apikey", data);
httpPost.setHeader("Modul", "upload_image");
HttpResponse httpResponse = httpclient.execute(httpPost);
inputStream = httpResponse.getEntity().getContent();
if (inputStream != null)
result = classxy.convertInputStreamToString(inputStream);
else
result = "Fehler!";
} catch (Exception e) {
Log.e("InputStream", e.getLocalizedMessage());
}
int num = Integer.parseInt(Nummer);
num++;
Nummer = Integer.toString(num);
return result;
}
public class HttpAsyncTask extends AsyncTask<String, Void, String> {
private final Object ID, Image, Nummer;
public HttpAsyncTask(Object ID, Object Image, Object Nummer) {
this.ID = ID;
this.Image = Image;
this.Nummer = Nummer;
}
protected String doInBackground(String... urls) {
return POST(urls[0]);
}
protected void onPostExecute(String result) {
if (result.matches("(.*)false(.*)")) {
Toast.makeText(getApplicationContext(), "....", Toast.LENGTH_SHORT).show();
} else {
Toast.makeText(getApplicationContext(), "...", Toast.LENGTH_SHORT).show();
}
Log.e("RESPONSE", result);
}
}
It does encode the images via base64 and it does upload some of the images. Unfortunately it uploads only the first image or one image multiple times. It never uploads the correct amount of images in the correct order. I've been sitting on this problem for a while now and can't figure out what I'm doing wrong.
Can you tell me what I'm doing wrong?
Your program doesn't seem to be thread-safe at all.
Your fields ID, Image and Nummer are updated with every iteration of the for loop. Most likely the loop has already finished before POST runs for the first time. Your observation would support this assumption:
Unfortunately it uploads only the first image or one image multiple times.
You can observe this by logging every access to these fields. You'll find, that it's not alternating like you expect it to be.
Therefore you should implement everything without using these fields at all. Instead use local variables and pass these around. Using the Nummer field could be usefull if you want to use it for more than one upload. But it might be better to use an int directly:
upload.setOnClickListener(new View.OnClickListener() {
public void onClick(View arg0) {
String ID = "", Image;
int Nummer = 0;
[...]
for (int i = 0; i < files.length; ++i) {
if (files[i].getName().endsWith(".jpg")) {
[...]
try {
new HttpAsyncTask(ID,Image,Integer.toString(Nummer++)).execute("https://....");
} catch (Exception e) {
Log.e("InputStream", e.getMessage());
}
Log.e("PICS", id);
}
}
}
});
public String POST(String url, String ID, String Image, String Nummer) {
InputStream inputStream;
try {
[...]
} catch (Exception e) {
Log.e("InputStream", e.getLocalizedMessage());
}
//int num = Integer.parseInt(Nummer);
//num++;
//Nummer = Integer.toString(num);
return result;
}
public class HttpAsyncTask extends AsyncTask<String, Void, String> {
private final String ID, Image, Nummer;
public HttpAsyncTask(String ID, String Image, String Nummer) {
this.ID = ID;
this.Image = Image;
this.Nummer = Nummer;
}
protected String doInBackground(String... urls) {
return POST(urls[0], ID, Image, Nummer);
}
protected void onPostExecute(String result) {
[...]
}
}
In My suggestion Dont call Asynctask directly from for loop because there are no any monitor on we can set for which image selected.
So Go through below steps:
1) In for loop get all images ID,Name and number and store it to ArrayList
2) Check ArrayList first is empty or not
if not then get first position ID, Image and number
call new HttpAsyncTask(ID,Image,Integer.toString(Nummer++)).execute("https://....");
3) In HttpAsyncTask onPostExecute(String result) method
first remove first position data
then create
for loop (i=0;i<ArrayList.Size();i++) {
ID=ArrayList first position data ID
Image=ArrayList first position data IMAGE
number=ArrayList first position data number
Call new HttpAsyncTask(ID,Image,Integer.toString(Nummer++)).execute("https://....");
}
So here first Image send by then after second then after third up to your list not empty and every time different image selected.
Thats it...

Sending an XML Object via HTTP POST

We are students.
In our project,we want to send xml block,basically saml assertion,from one server to another server via http post method.
Can anyone help us out in sending the XML object from one servlet to another servlet where each servlet resides on two different computers in java.
/* here we are trying to send xml object(root) from one servlet to another servlet which resides on different pc... but dispatcher method isnt working in this case.*/
public class sp1serv extends HttpServlet
{
public void doPost(HttpServletRequest req,HttpServletResponse resp) throws ServletException,java.io.IOException
{
Connection c=null;
Statement s= null;
ResultSet rs = null;
String d=null;
int flag=0;
resp.setContentType("text/html");
PrintWriter out = resp.getWriter();
Response response=null;
XMLObject root=null;
HttpSession session1=req.getSession();
System.out.println(session1.getAttribute("sAccessLevel"));
System.out.println(session1.getAttribute("sUserId"));
String eid=session1.getAttribute("sUserId").toString();
String[] str1 = {"response","attr",session1.getAttribute("sAccessLevel").toString(), session1.getAttribute("sUserId").toString() };
String filename= eid.concat(".xml");
try {
response=SAMLProtocol.passResponse(str1);
root=SAMLSignature.passSignature(response,filename);
} catch (Exception e) {
// TODO Auto-generated catch block
e.printStackTrace();
}
req.setAttribute("SP1",root);
String abc="http://169.254.229.232:8080/sp_response_handler";
RequestDispatcher rd=getServletContext().getRequestDispatcher(abc);
rd.forward(req, resp);
break;
}
}
}
}}
/* this servlet is used for retrieving xml object(root) and parsing it..on another server.*/
public class sp1_response_handler extends HttpServlet {
private static final long serialVersionUID = 1L;
public sp1_response_handler() {
super();
// TODO Auto-generated constructor stub
}
protected void doPost(HttpServletRequest request, HttpServletResponse response) throws ServletException, IOException {
// TODO Auto-generated method stub
Response resp=null;
//XMLObject resp=null;
resp=(Response) request.getAttribute("SP1");
int result=0;
//SAMLSignature verification=null;
try {
result=SAMLSignature.verify(resp);
} catch (Exception e) {
// TODO Auto-generated catch block
e.printStackTrace();
}
if(result==1){
List attributeStatements = resp.getAssertions().get(0).getAttributeStatements();
for (int i = 0; i < attributeStatements.size(); i++)
{
List attributes = ((AttributeStatement) attributeStatements.get(i)).getAttributes();
for (int x = 0; x < attributes.size(); x++)
{
String strAttributeName = ((XMLObject) attributes.get(x)).getDOM().getAttribute("Name");
List<XMLObject> attributeValues = ((Attribute) attributes.get(x)).getAttributeValues();
for (int y = 0; y < attributeValues.size(); y++)
{
String strAttributeValue = attributeValues.get(y).getDOM().getTextContent();
System.out.println(strAttributeName + ": " + strAttributeValue);
}
}
}
response.sendRedirect("SP1.jsp");
}
else
{
System.out.println("NOT a Valid Signature");
}
}}
If you are using spring, you can use RestTemplate. From the docs:
String uri = "http://example.com/hotels/1/bookings";
PostMethod post = new PostMethod(uri);
// create booking request content
String request = post.setRequestEntity(new StringRequestEntity(request));
httpClient.executeMethod(post);
if (HttpStatus.SC_CREATED == post.getStatusCode()) {
Header location = post.getRequestHeader("Location");
if (location != null) {
System.out.println("Created new booking at :" + location.getValue());
}
}
Something like that should work (with the parameters being a Map<String,String>):
StringBuffer data = new StringBuffer();
if (parameters != null && parameters.size() > 0) {
for (Entry<String, String> e : parameters.entrySet()) {
if (data.length() > 0) {
data.append('&');
}
data.append(URLEncoder.encode(e.getKey(), "UTF-8")).append("=").append(URLEncoder.encode(e.getValue(), "UTF-8"));
}
}
String parametersAsString = data.toString();
// Send data
URL local_url = new URL(url);
URLConnection conn = local_url.openConnection();
conn.addRequestProperty("Content-Type", "text/xml; charset=utf-8");
conn.setDoOutput(true);
OutputStreamWriter wr = new OutputStreamWriter(conn.getOutputStream());
wr.write(parametersAsString);
wr.flush();
break;

posting data to a servlet using httpclient

I'm trying to post 2 fields, id and data, to a servlet using HttpClient.
The problem is that if the length of the data field is less than 1MB or so, the servlet will get what I posted. But if the length of the data field is larger than 1MB or so, the servlet will receive null for all fields. What am I missing here? Thanks.
Here's the sample data that I post to the servlet:
id=12312123123123
data=the content of a file that is base-64 encoded
Here's the method that I use to post data to the servlet.
private byte[] post(String aUrl,
Map<String,String> aParams,
String aCharsetEnc,
int aMaxWaitMs) throws Exception
{
PostMethod post = null;
try
{
HttpClient client = new HttpClient();
post = new PostMethod(aUrl);
post.setRequestHeader("Content-Type", "application/x-www-form-urlencoded; charset=" + aCharsetEnc);
for (String key : aParams.keySet())
{
post.addParameter(key, aParams.get(key));
}
final int code = client.executeMethod(post);
if (code == HttpStatus.SC_NO_CONTENT || code == HttpStatus.SC_NOT_FOUND)
{
return null;
}
else if (code != HttpStatus.SC_OK)
{
throw new HttpException("Error code " + code + " encountered.");
}
InputStream stream = post.getResponseBodyAsStream();
if (stream != null)
{
return BlobHelper.readBytes(stream);
}
return null;
}
finally
{
if (post != null)
{
post.releaseConnection();
}
}
}
Here's the method of the servlet.
public void doPost(HttpServletRequest aReq, HttpServletResponse aResp)
throws ServletException, IOException
{
setNoCache(aResp);
aResp.setContentType("text/plain");
try
{
final String id = aReq.getParameter(PARAM_ID);
final String dataStr = aReq.getParameter(PARAM_DATA);
if (log().isDebugEnabled())
{
log().debug("id=" + id);
log().debug("data=" + dataStr);
}
}
catch (Exception e)
{
}
}
Usually servlet containers have a maximum post size parameter.
For Tomcat you can follow the steps documented here(they should be similar for other appservers) -
Is there a max size for POST parameter content?

Categories