We Keep Coding

Scraping Google App Store for Meta Data

I am noob in Java trying to to build a scraper in a Java which could do the following things.
Ability to read data from a CSV file.
Use the URIs in that file and scrape the complete App info from the Google Playstore.
Export the scraped data and other meta data from the CSV file into an XML file
Can any one guide me in this how to go from here ?
Till now I have made the following three classes
main.java (This is the main method where I call other two classes)
import java.io.IOException;
public class main {
public static void main(String[] args) throws IOException {
ReadCVS obj = new ReadCVS();
obj.run();
AppInfo obj1 = new AppInfo();
obj1.readFile();
}
}
ReadCVS.java (This file reads the CSV file and give the output in a txt file)
import java.io.BufferedReader;
import java.io.File;
import java.io.FileNotFoundException;
import java.io.FileOutputStream;
import java.io.FileReader;
import java.io.IOException;
import java.io.PrintStream;
public class ReadCVS {
public void run() {
// Replace the file path to the appropriate path.
String csvFile = "\\Desktop\\https---play_google_com-store-apps-details-id=.csv";
BufferedReader br = null;
String line = "";
String cvsSplitBy = ";";
try {
File file = new File("\\Desktop\\output.txt");
FileOutputStream fos = new FileOutputStream(file);
PrintStream ps = new PrintStream(fos);
System.setOut(ps);
br = new BufferedReader(new FileReader(csvFile));
while ((line = br.readLine()) != null) {
// use comma as separator
String[] country = line.split(cvsSplitBy);
System.out.println("URL = " + country[0] + " "
);
}
} catch (FileNotFoundException e) {
e.printStackTrace();
} catch (IOException e) {
e.printStackTrace();
} finally {
if (br != null) {
try {
br.close();
} catch (IOException e) {
e.printStackTrace();
}
}
}
System.out.println("Done");
}
}
AppInfo.java (This file reads the input from the saved output.txt and tries to out put in the console. But it is not currently working)
import java.io.BufferedReader;
import java.io.FileReader;
import java.io.IOException;
public class AppInfo {
public void readFile(){
String fileName = "\\Desktop\\output.txt";
//read file into stream, try-with-resources
try (BufferedReader br = new BufferedReader(new FileReader(fileName))) {
String line;
while ((line = br.readLine()) != null) {
System.out.println(line);
}
} catch (IOException e) {
e.printStackTrace();
}
}
}
The problem is that whenever I try to run this code the program get hanged and does not terminate.
Can any one help me with my problem ?

Download a file from MediaFire with Java

As we all know, or might know, MediaFire does not allow direct download links, but you actually have to click the Download button to generate a random link that refers to the file. Is there a way to fetch that link and download it?

In despair of writing an auto-updating application, I have written a short Java application which allows the download of files from MediaFire. Here is the full code:
import java.io.BufferedInputStream;
import java.io.BufferedReader;
import java.io.FileOutputStream;
import java.io.IOException;
import java.io.InputStreamReader;
import java.net.URL;
import java.net.URLConnection;
public class mainwindow {
/**
* Launch the application.
*/
static mainwindow thisInstance;
public static void main(String[] args) {
new mainwindow();
}
public mainwindow()
{
otherthread();
}
public void otherthread()
{
navigate("http://www.mediafire.com/download/aqtmhwvb8yvqclu/SmartSharePC.jar","downloadedFromMediafire");
// navigate("http://www.mediafire.com/download/j7e4wh6hbdhdj84/Minecraft+1.5.2-+C.H.T.zip","mediafire");
}
private void navigate(String url,String sufix)
{
try
{
String downloadLink = fetchDownloadLink(getUrlSource(url));
saveUrl(downloadLink,sufix);
} catch (Exception e)
{
e.printStackTrace();
}
}
public void saveUrl(final String urlString,String sufix) throws Exception
{
System.out.println("Downloading...");
String filename = urlString.substring(urlString.lastIndexOf("/")+1, urlString.lastIndexOf("."))+"_"+sufix+urlString.substring(urlString.lastIndexOf("."), urlString.length());
BufferedInputStream in = null;
FileOutputStream fout = null;
try {
in = new BufferedInputStream(new URL(urlString).openStream());
fout = new FileOutputStream(filename);
final byte data[] = new byte[1024];
int count;
while ((count = in.read(data, 0, 1024)) != -1)
{
fout.write(data, 0, count);
}
} finally {
if (in != null) {
in.close();
}
if (fout != null) {
fout.close();
}
}
System.out.println("Success!");
}
private static String getUrlSource(String url) throws IOException
{
System.out.println("Connecting...");
URL yahoo = new URL(url);
URLConnection yc = yahoo.openConnection();
BufferedReader in = new BufferedReader(new InputStreamReader(
yc.getInputStream(), "UTF-8"));
String inputLine;
String total="";
while ((inputLine = in.readLine()) != null)
total+=inputLine;
in.close();
return total;
}
private static String fetchDownloadLink(String str)
{
System.out.println("Fetching download link");
try {
String regex = "(?=\\<)|(?<=\\>)";
String data[] = str.split(regex);
String found = "NOTFOUND";
for (String dat : data) {
if (dat.contains("DLP_mOnDownload(this)")) {
found = dat;
break;
}
}
String wentthru = found.substring(found.indexOf("href=\"") + 6);
wentthru = wentthru.substring(0, wentthru.indexOf("\""));
return wentthru;
} catch (Exception e)
{
e.printStackTrace();
return "ERROR";
}
}
}

Creating, writing and editing same text file in java

Let's say I have the following code:
import java.io.BufferedReader;
import java.io.BufferedWriter;
import java.io.File;
import java.io.FileReader;
import java.io.FileWriter;
import java.io.IOException;
public class EditFile {
public static void main(String[] args) {
try{
String verify, putData;
File file = new File("file.txt");
file.createNewFile();
FileWriter fw = new FileWriter(file);
BufferedWriter bw = new BufferedWriter(fw);
bw.write("Some text here for a reason");
bw.flush();
bw.close();
FileReader fr = new FileReader(file);
BufferedReader br = new BufferedReader(fr);
while( br.readLine() != null ){
verify = br.readLine();
if(verify != null){
putData = verify.replaceAll("here", "there");
bw.write(putData);
}
}
br.close();
}catch(IOException e){
e.printStackTrace();
}
}
}
All I wanted to do was to write something in a text file, in my case "Some text here for a reason". Then to read data from my file, and finally to change my text from my file from "Some text here for a reason" in "Some text there for a reason". I ran the code but all it happens is to write in my file "Some text here for a reason".
I tried to figure out what could be wrong in my code, but unfortunately it was in vain. Any advice or rewrite is highly appreciated from me.

Change your code to that:
import java.io.BufferedReader;
import java.io.BufferedWriter;
import java.io.File;
import java.io.FileReader;
import java.io.FileWriter;
import java.io.IOException;
public class EditFile {
public static void main(String[] args) {
try{
String verify, putData;
File file = new File("file.txt");
file.createNewFile();
FileWriter fw = new FileWriter(file);
BufferedWriter bw = new BufferedWriter(fw);
bw.write("Some text here for a reason");
bw.flush();
bw.close();
FileReader fr = new FileReader(file);
BufferedReader br = new BufferedReader(fr);
while( (verify=br.readLine()) != null ){ //***editted
//**deleted**verify = br.readLine();**
if(verify != null){ //***edited
putData = verify.replaceAll("here", "there");
bw.write(putData);
}
}
br.close();
}catch(IOException e){
e.printStackTrace();
}
}
}
The Problem is that you are calling br.readLine() twice which is provoking the application to read line1 and then line2 and in your case you have just one line which means that your program read it in the conditional form and when it comes to declaring it to the variable verify, it is stopping because you don't have anymore data to read your file.

I would do it this way:
import java.io.*;
public class EditFile {
public static void main(String[] args) {
try{
String verify, putData;
File file = new File("file.txt");
file.createNewFile();
FileWriter fw = new FileWriter(file);
BufferedWriter bw = new BufferedWriter(fw);
bw.write("Some text here for a reason");
bw.flush();
FileReader fr = new FileReader(file);
BufferedReader br = new BufferedReader(fr);
while( (verify=br.readLine()) != null )
{
if(verify != null)
{
putData = verify.replaceAll("here", "there");
bw.write(putData);
}
}
br.close();
bw.close();
}catch(IOException e){
e.printStackTrace();
}
}
}

use this code, I used it to remove logs and System.out statements in java file.
just change the matching and replacing string.
import java.io.BufferedReader;
import java.io.BufferedWriter;
import java.io.File;
import java.io.FileReader;
import java.io.FileWriter;
import java.util.ArrayList;
import java.util.List;
import java.util.Scanner;
public class FileReplace {
List<String> lines = new ArrayList<String>();
String line = null;
Scanner scan = null;
public void doIt() {
scan = new Scanner(System.in);
while (true) {
try {
System.out
.println("enter qualified file name ex.D:\\shiv\\shiv android all\\Main work space\\Welcomescreen1.java");
String path = scan.nextLine();
File f1 = new File(path);
FileReader fr = new FileReader(f1);
BufferedReader br = new BufferedReader(fr);
int i = 0;
while ((line = br.readLine()) != null) {
if (line.contains("System.out")) {
line = line.replace("System.out", "//");
} else if (line.contains("Log.")) {
line = line.replace("Log", "//");
}
lines.add(i, line);
i++;
}
fr.close();
br.close();
FileWriter fw = new FileWriter(f1);
BufferedWriter out = new BufferedWriter(fw);
for (int j = 0; j < lines.size(); j++) {
System.out.println(j + "." + lines.get(j));
out.append(lines.get(j));
out.newLine();
}
out.flush();
out.close();
System.out
.println("====================work done===================");
} catch (Exception ex) {
ex.printStackTrace();
}
}
}
public static void main(String args[]) {
FileReplace fr = new FileReplace();
fr.doIt();
}
}

import java.io.*;
public class TextFile
{
public static void main(String[] args)
{
try
{
String verify, putData;
File file = new File("G:\\Dairy.txt");
file.createNewFile();
FileWriter fw = new FileWriter(file);
BufferedWriter bw = new BufferedWriter(fw);
bw.write("I am Shah Khalid");
bw.flush();
FileReader fr = new FileReader(file);
BufferedReader br = new BufferedReader(fr);
while( (verify=br.readLine()) != null )
{
if(verify != null)
{
putData = verify.replaceAll("here", "there");
//bw.write(putData);
}
}
br.close();
bw.close();
}
catch(IOException e)
{
e.printStackTrace();
}
System.out.println("Shah");
}
}
There is no need to type bw.write(putData);, because it will just print the statement twice.
Whatever you want in a file, just give the correct path of the file and use the above code accordingly.

File file = new File("/tmp/my.txt");
FileWriter fw;
BufferedReader br;
BufferedWriter bw;
boolean no=false;
String line;
String data="";
String lessonPath="my new line";
try {
if(!file.exists()){
fw = new FileWriter(file);
bw = new BufferedWriter(fw);
bw.write(lessonPath);
bw.flush();
bw.close();
}else{
br = new BufferedReader(new FileReader(file));
while((line =br.readLine()) !=null){
if(!no){
data=line;
no=true;
}else{
data = data+"\n"+line;
}
}
bw = new BufferedWriter(new FileWriter(file));
bw.write(data+"\n"+lessonPath);
bw.flush();
bw.close();
}
} catch (Exception ex) {
ex.printStackTrace();
}

Why does the integer written to a file get read as a different value?

I've got a program where I need to generate an integer, write it to a text file and read it back the next time the program runs. After some anomalous behavior, I've stripped it down to setting an integer value, writing it to a file and reading it back for debugging.
totScore, is set to 25 and when I print to the console prior to writing to the file, I see a value of 25. However, when I read the file and print to the console I get three values...25, 13, and 10. Viewing the text file in notepad gives me a character not on the keyboard, so I suspect that the file is being stored in something other that int.
Why do I get different results from my write and read steps?
Is it not being written as an int? How are these values being stored in the file? Do I need to cast the read value as something else and convert it to an integer?
Consider:
import javax.swing.*;
import java.awt.*;
import java.awt.event.*;
import java.io.*;
import java.nio.file.*;
import java.nio.file.StandardOpenOption.*;
//
public class HedgeScore {
public static void main(String[] args) {
int totScore = 25;
OutputStream outStream = null; ///write
try {
System.out.println("totscore="+totScore);
BufferedWriter bw = new BufferedWriter(new FileWriter(new File("hedgescore.txt")));
bw.write(totScore);
bw.write(System.getProperty("line.separator"));
bw.flush();
bw.close();
}
catch(IOException f) {
System.out.println(f.getMessage());
}
try {
InputStream input = new FileInputStream("hedgescore.txt");
int data = input.read();
while(data != -1) {
System.out.println("data being read from file :"+ data);
data = input.read();
int prevScore = data;
}
input.close();
}
catch(IOException f) {
System.out.println(f.getMessage());
}
}
}

You're reading/writing Strings and raw data, but not being consistent. Why not instead read in Strings (using a Reader of some sort) and then convert to int by parsing the String? Either that or write out your data as bytes and read it in as bytes -- although that can get quite tricky if the file must deal with different types of data.
So either:
import java.io.*;
public class HedgeScore {
private static final String FILE_PATH = "hedgescore.txt";
public static void main(String[] args) {
int totScore = 25;
BufferedWriter bw = null;
try {
System.out.println("totscore=" + totScore);
bw = new BufferedWriter(new FileWriter(new File(
FILE_PATH)));
bw.write(totScore);
bw.write(System.getProperty("line.separator"));
bw.flush();
} catch (IOException f) {
System.out.println(f.getMessage());
} finally {
if (bw != null) {
try {
bw.close();
} catch (IOException e) {
e.printStackTrace();
}
}
}
InputStream input = null;
try {
input = new FileInputStream(FILE_PATH);
int data = 0;
while ((data = input.read()) != -1) {
System.out.println("data being read from file :" + data);
}
input.close();
} catch (IOException f) {
System.out.println(f.getMessage());
} finally {
if (input != null) {
try {
input.close();
} catch (IOException e) {
e.printStackTrace();
}
}
}
}
}
or:
import java.io.*;
public class HedgeScore2 {
private static final String FILE_PATH = "hedgescore.txt";
public static void main(String[] args) {
int totScore = 25;
PrintWriter pw = null;
try {
System.out.println("totscore=" + totScore);
pw = new PrintWriter(new FileWriter(new File(FILE_PATH)));
pw.write(String.valueOf(totScore));
pw.write(System.getProperty("line.separator"));
pw.flush();
} catch (IOException f) {
System.out.println(f.getMessage());
} finally {
if (pw != null) {
pw.close();
}
}
BufferedReader reader = null;
try {
reader = new BufferedReader(new FileReader(FILE_PATH));
String line = null;
while ((line = reader.readLine()) != null) {
System.out.println(line);
}
} catch (IOException f) {
System.out.println(f.getMessage());
} finally {
if (reader != null) {
try {
reader.close();
} catch (IOException e) {
e.printStackTrace();
}
}
}
}
}

read multiple data in multiple files

I have a list of files in the directory C:\Users\Mahady\Desktop\Java 31122011\src\register\
they are like this....
100100545.txt
100545454.txt etc etc
in each file, file data are like this line by line:
Bob
1234
4834
London
9852
1
My question is, how do i read each files one by one in the directory and for each files read all lines except line 3. i would then like to merge this data in word and create letters. thanks

Detailed Answer....
import java.io.BufferedReader;
import java.io.File;
import java.io.FileNotFoundException;
import java.io.FileReader;
import java.io.IOException;
public class FileRead {
public static void main(String[] args) {
FileReader fileReader = null;
BufferedReader bufferedReader = null;
try {
File folder = new File("C:/Users/Mahady/Desktop/Java 31122011/src/register/");
if (folder.isDirectory()) {
for (File file : folder.listFiles()) {
fileReader = new FileReader(file);
bufferedReader = new BufferedReader(fileReader);
String line = null;
int lineCount = 0;
while (null != (line = bufferedReader.readLine())) {
lineCount++;
if (3 != lineCount) {
System.out.println(line);
}
}
}
}
} catch (FileNotFoundException e) {
e.printStackTrace();
} catch (IOException e) {
e.printStackTrace();
} finally {
if (null != bufferedReader)
try {
bufferedReader.close();
} catch (IOException e) {
e.printStackTrace();
}
}
}
}
Hope this would help you.

Try this:
File dir = new File("C:\\Users\\Mahady\\Desktop\\Java 31122011\\src\\register\\");
for (string fn : dir.list()) {
FileInputStream fstream = new FileInputStream(fn);
DataInputStream in = new DataInputStream(fstream);
BufferedReader br = new BufferedReader(new InputStreamReader(in));
String strLine;
while ((strLine = br.readLine()) != null) {
System.out.println (strLine);
}
in.close();
}
Obviously, you will need to add exception handling code around this skeletal implementation.