We Keep Coding

Data corrupting on program close in jxl

Im using jxl to store data entered in by the user into an excel spreadsheet. I want to be able to close my program while its running and open up the excel spreadsheet and view the data. Whenever I do this, I get a message that says that it is in a different format than specified in the file extension. Here is my code.
import java.io.File;
import java.io.IOException;
import java.util.Scanner;
import jxl.*;
import jxl.write.*;
import jxl.Workbook;
import jxl.read.biff.BiffException;
import jxl.write.Label;
import jxl.write.Number;
import jxl.write.WritableSheet;
import jxl.write.WritableWorkbook;
import jxl.write.WriteException;
import jxl.write.biff.RowsExceededException;
public class ReadWriteTest
{
public static void main(String[] args) throws BiffException, IOException, RowsExceededException, WriteException
{
Scanner scan = new Scanner(System.in);
int i = 1;
Integer iAlliance = -1;
File test = new File("C:\\test.xls");
WritableWorkbook wb = Workbook.createWorkbook(test);
while(iAlliance != 0 && i != 5)
{
WritableSheet sheet = wb.createSheet("Sheet" + i, i);
iAlliance = scan.nextInt();
WriteExcel.addNumber(sheet, 0, 0, iAlliance);
iAlliance = scan.nextInt();
WriteExcel.addNumber(sheet, 1, 0, iAlliance);
i++;
}
wb.write();
wb.close();
}
}

"close my program while its running"
Your code needs to handle that. Right now, it just gets killed at any point (probably in that loop), and nothing after that point happens. The WriteableWorkbook never gets write() or close() called, and the file is probably corrupted.
So see this, try adding some simple output at various places, like "got to line number XXX". Run it, close it, and see how far it got.
You'll probably need to wrap the bulk of your code in a try-catch-finally. The loop is inside the try, the catch handles the lightning-out-of-the-blue program closing from that pesky human at the keyboard, and the finally closes up the workbook file properly.

Jsoup reddit scraper 429 error

So I'm trying to use jsoup to scrape Reddit for images, but when I scrape certain subreddits such as /r/wallpaper, I get a 429 error and am wondering how to fix this. Totally understand that this code is horrible and this is a pretty noob question, but I'm completely new to this. Anyways:
import java.io.IOException;
import org.jsoup.Jsoup;
import org.jsoup.nodes.Document;
import org.jsoup.nodes.Element;
import org.jsoup.select.Elements;
import java.io.BufferedReader;
import java.io.BufferedWriter;
import java.io.File;
import java.io.FileWriter;
import java.io.IOException;
import java.io.InputStreamReader;
import java.net.MalformedURLException;
import java.net.URL;
import java.net.URLConnection;
import java.io.*;
import java.net.URL;
import java.util.logging.Level;
import java.util.logging.Logger;
import java.io.*;
import java.util.logging.Level;
import java.util.logging.Logger;
import org.jsoup.Jsoup;
import org.jsoup.nodes.Attributes;
import org.jsoup.nodes.Document;
import org.jsoup.nodes.Element;
import org.jsoup.select.Elements;
import java.io.IOException;
import java.net.URL;
import java.util.Scanner;
public class javascraper{
public static void main (String[]args) throws MalformedURLException
{
Scanner scan = new Scanner (System.in);
System.out.println("Where do you want to store the files?");
String folderpath = scan.next();
System.out.println("What subreddit do you want to scrape?");
String subreddit = scan.next();
subreddit = ("http://reddit.com/r/" + subreddit);
new File(folderpath + "/" + subreddit).mkdir();
//test
try{
//gets http protocol
Document doc = Jsoup.connect(subreddit).timeout(0).get();
//get page title
String title = doc.title();
System.out.println("title : " + title);
//get all links
Elements links = doc.select("a[href]");
for(Element link : links){
//get value from href attribute
String checkLink = link.attr("href");
Elements images = doc.select("img[src~=(?i)\\.(png|jpe?g|gif)]");
if (imgCheck(checkLink)){ // checks to see if img link j
System.out.println("link : " + link.attr("href"));
downloadImages(checkLink, folderpath);
}
}
}
catch (IOException e){
e.printStackTrace();
}
}
public static boolean imgCheck(String http){
String png = ".png";
String jpg = ".jpg";
String jpeg = "jpeg"; // no period so checker will only check last four characaters
String gif = ".gif";
int length = http.length();
if (http.contains(png)|| http.contains("gfycat") || http.contains(jpg)|| http.contains(jpeg) || http.contains(gif)){
return true;
}
else{
return false;
}
}
private static void downloadImages(String src, String folderpath) throws IOException{
String folder = null;
//Exctract the name of the image from the src attribute
int indexname = src.lastIndexOf("/");
if (indexname == src.length()) {
src = src.substring(1, indexname);
}
indexname = src.lastIndexOf("/");
String name = src.substring(indexname, src.length());
System.out.println(name);
//Open a URL Stream
URL url = new URL(src);
InputStream in = url.openStream();
OutputStream out = new BufferedOutputStream(new FileOutputStream( folderpath+ name));
for (int b; (b = in.read()) != -1;) {
out.write(b);
}
out.close();
in.close();
}
}

Your issue is caused by the fact that your scraper is violating reddit's API rules. Error 429 means "Too many requests" – you're requesting too many pages too fast.
You can make one request every 2 seconds, and you also need to set a proper user agent (they format they recommend is <platform>:<app ID>:<version string> (by /u/<reddit username>)). The way it currently looks, your code is running too fast and doesn't specify one, so it's going to be severely rate-limited.
To fix it, first off, add this to the start of your class, before the main method:
public static final String USER_AGENT = "<PUT YOUR USER AGENT HERE>";
(Make sure to specify an actual user agent).
Then, change this (in downloadImages)
URL url = new URL(src);
InputStream in = url.openStream();
to this:
URLConnection connection = (new URL(src)).openConnection();
Thread.sleep(2000); //Delay to comply with rate limiting
connection.setRequestProperty("User-Agent", USER_AGENT);
InputStream in = connection.getInputStream();
You'll also want to change this (in main)
Document doc = Jsoup.connect(subreddit).timeout(0).get();
to this:
Document doc = Jsoup.connect(subreddit).userAgent(USER_AGENT).timeout(0).get();
Then your code should stop running into that error.
Note that using reddit's API (IE, /r/subreddit.json instead of /r/subreddit) would probably make this project easier, but it isn't required and your current code will work.

As you can look up at Wikipedia the 429 status code tells you that you have too many requests:
The user has sent too many requests in a given amount of time. Intended for use with rate limiting schemes.
A solution would be to slow down your scraper. There are some options how to do this, one would be to use sleep.

SlopeOneRecommender not working

I am following the book Apache Mahout Cookbook by piero giacomelli. Now when i download the maven sources using netbeans as IDE, i guess the sources are from mahout version 1.0 and not 0.8 as it showing an error in SlopeOneRecommender import alone.
Here is the complete code -
package com.packtpub.mahout.cookbook.chapter01;
import java.io.BufferedReader;
import java.io.BufferedWriter;
import java.io.File;
import java.io.FileNotFoundException;
import java.io.FileReader;
import java.io.FileWriter;
import java.io.IOException;
import java.util.List;
import org.apache.commons.cli2.OptionException;
import org.apache.mahout.cf.taste.common.TasteException;
import org.apache.mahout.cf.taste.impl.common.LongPrimitiveIterator;
import org.apache.mahout.cf.taste.impl.model.file.FileDataModel;
import org.apache.mahout.cf.taste.impl.recommender.CachingRecommender;
import org.apache.mahout.cf.taste.impl.recommender.slopeone.SlopeOneRecommender;
import org.apache.mahout.cf.taste.model.DataModel;
import org.apache.mahout.cf.taste.recommender.RecommendedItem;
public class App {
static final String inputFile = "/home/hadoop/ml-1m/ratings.dat";
static final String outputFile = "/home/hadoop/ml-1m/ratings.csv";
public static void main( String[] args ) throws IOException, TasteException, OptionException
{
CreateCsvRatingsFile();
// create data source (model) - from the csv file
File ratingsFile = new File(outputFile);
DataModel model = new FileDataModel(ratingsFile);
// create a simple recommender on our data
CachingRecommender cachingRecommender = new CachingRecommender(new SlopeOneRecommender(model));
// for all users
for (LongPrimitiveIterator it = model.getUserIDs(); it.hasNext();){
long userId = it.nextLong();
// get the recommendations for the user
List<RecommendedItem> recommendations = cachingRecommender.recommend(userId, 10);
// if empty write something
if (recommendations.size() == 0){
System.out.print("User ");
System.out.print(userId);
System.out.println(": no recommendations");
}
// print the list of recommendations for each
for (RecommendedItem recommendedItem : recommendations) {
System.out.print("User ");
System.out.print(userId);
System.out.print(": ");
System.out.println(recommendedItem);
}
}
}
private static void CreateCsvRatingsFile() throws FileNotFoundException, IOException {
BufferedReader br = new BufferedReader(new FileReader(inputFile));
BufferedWriter bw = new BufferedWriter(new FileWriter(outputFile));
String line = null;
String line2write = null;
String[] temp;
int i = 0;
while (
(line = br.readLine()) != null
&& i < 1000
){
i++;
temp = line.split("::");
line2write = temp[0] + "," + temp[1];
bw.write(line2write);
bw.newLine();
bw.flush();
}
br.close();
bw.close();
}
}
The error is being shown only on import org.apache.mahout.cf.taste.impl.recommender.slopeone.SlopeOneRecommender;
and hence on the line where i create an object using this. Error being shown is package does not exist.
Please help. Is it because i am using a newer version of mahout? I am even uncertain if I am using version 0.8 or a higher version as i followed all the links given in the book.

The SlopeOneRecommender was removed from mahout since v0.8. If you want to use it, you can switch to version such as 0.7.
<dependency>
<groupId>org.apache.mahout</groupId>
<artifactId>mahout-core</artifactId>
<version>0.7</version>
</dependency>
See http://permalink.gmane.org/gmane.comp.apache.mahout.user/20282

Exactly,
The SlopeOneRecommender was removed from mahout since v0.8. So either you get back to the version 0.7. Or if your purpose is only to try mahout u can try with other recommenders, such as :ItemAverageRecommender.

Find the second duplicate word

import java.io.BufferedReader;
import java.io.File;
import java.io.FileReader;
import java.io.IOException;
public class Example {
public static void main(String[] args) throws IOException
{
File fis=new File("D:/Testcode/Test.txt");
BufferedReader br;
String input;
String var = null;
if(fis.isAbsolute())
{
br=new BufferedReader(new FileReader(fis.getAbsolutePath()));
while ((input=br.readLine())!=null) {
var=input;
}
}
//String var="Duminy to Warner, OUT, Duminy gets a wicket again. He has been breaking...
if(var!=null)
{
String splitstr[]=var.split(",");
if(splitstr[0].contains("to"))
{
String ss=splitstr[0];
String a[]=ss.split("\\s+");
int value=splitstr[0].indexOf("to");
System.out.println("Subject:"+splitstr[0].substring(0,value));
System.out.println("Object:"+splitstr[0].substring(value+2));
System.out.println("Event:"+splitstr[1]);
int count=var.indexOf(splitstr[2]);
System.out.println("Narrated Information:"+var.substring(count));
}
}
}
}
The above program shown the following output:
Subject:Duminy
Object: Warner
Event: OUT
Narrated Information: Duminy gets a wicket again. He has been breaking....
my question is, the text may contain, For example: "Dumto to Warner, OUT, Duminy gets a wicket again. He has been breaking..." means, the above program wouldn't show output like above.. how to identity the text after the space for checking the condition

Instead of:
if(splitstr[0].contains("to")
Change it to:
if(splitstr[0].contains(" to ")
It should then work fine IMO.

How to direct std input from file in Eclipse

How we can take input from the file in the Eclipse?
Just like we direct the I/O from the file from the command line.
java MyProgram < input.txt >output.txt
I am unable to direct the input.
but output directing is easy.
Just go->Run->Run->Configurations->Common

Why don't you use File instead of redirection?
Your program will have a fileName as input and then write the result in a file.

If you need necessarily use the default in you can do something like this:
System.setIn(new FileInputStream("testFile.txt"));
a sample of how it works follows:
import java.io.BufferedReader;
import java.io.FileInputStream;
import java.io.FileOutputStream;
import java.io.IOException;
import java.io.InputStreamReader;
public class TestSystemIn {
public static void main(final String[] args) throws IOException {
// prepare test
FileOutputStream fos = new FileOutputStream("testFile.txt");
fos.write("testToken".getBytes());
// configure env
System.setIn(new FileInputStream("testFile.txt"));
// perform read test
BufferedReader br = new BufferedReader(new InputStreamReader(System.in));
System.out.println("read: " + br.readLine());
}
}
The output could be done in the same way using:
System.setOut(new PrintStream("testFile.txt"));