We Keep Coding

Copy Json Flat file from local to HDFS

package com.Main;
import java.io.BufferedInputStream;
import java.io.FileInputStream;
import java.io.IOException;
import java.io.InputStream;
import java.io.OutputStream;
import java.net.URI;
import org.apache.hadoop.conf.Configuration;
import org.apache.hadoop.fs.FileSystem;
import org.apache.hadoop.fs.Path;
import org.apache.hadoop.io.IOUtils;
public class Main {
public static void main(String[] args) throws IOException {
//Source file in the local file system
String localSrc = args[0];
//Destination file in HDFS
String dst = args[1];
//Input stream for the file in local file system to be written to HDFS
InputStream in = new BufferedInputStream(new FileInputStream(localSrc));
//Get configimport org.apache.commons.configuration.Configuration;uration of Hadoop system
Configuration conf = new Configuration();
System.out.println("Connecting to -- "+conf.get("fs.defaultFS"));
//Destination file in HDFS
FileSystem fs = FileSystem.get(URI.create(dst), conf);
OutputStream out = fs.create(new Path(dst));
//Copy file from local to HDFS
IOUtils.copyBytes(in, out, 4096, true);
System.out.println(dst + " copied to HDFS");
}
}
AM getting following error message "Exception in thread "main" java.lang.ArrayIndexOutOfBoundsException: 0
at com.Main.Main.main(Main.java:22)"
I have Json file in my local , have to move that in HDFS
Ex:
{"Del":"Ef77xvP","time":1509073785106},
{"Del":"2YXsF7r","time":1509073795109}

Specify command line arguments to your program. You code snippet expects first argument to be source and next arguments to be destination.
For more details refer to What is "String args[]"? parameter in main method Java

Sphinx4 dictionary file not found

I am using Sphinx4 for custome home automatization software and I am stuck on using my custom dict file, for some reason it cant find the file even though I am sure I am using the correct path. This is my code :
package pccomone;
import java.io.File;
import java.io.FileInputStream;
import java.io.InputStream;
import edu.cmu.sphinx.api.Configuration;
import edu.cmu.sphinx.api.LiveSpeechRecognizer;
import edu.cmu.sphinx.api.SpeechResult;
import edu.cmu.sphinx.api.StreamSpeechRecognizer;
public class Main {
private static final String BODICPATH = "PCCom/src/resource/bodict.dict";
public static void main(String args[]) throws Exception{
Configuration configuration = new Configuration();
configuration.setAcousticModelPath("resource:/edu/cmu/sphinx/models/en-us/en-us");
configuration.setDictionaryPath(BODICPATH);
//configuration.setDictionaryPath("resource:/edu/cmu/sphinx/models/en-us/cmudict-en-us.dict"); use only for large commands!!!!!!!!!
configuration.setLanguageModelPath("resource:/edu/cmu/sphinx/models/en-us/en-us.lm.bin");
LiveSpeechRecognizer recognizer = new LiveSpeechRecognizer(configuration);
recognizer.startRecognition(true);
SpeechResult result;
while((result = recognizer.getResult()) != null){
String command = result.getHypothesis();
if(command.equalsIgnoreCase("stop")){
recognizer.stopRecognition();
System.exit(0);
}
System.out.println(command);
}
}
}
I didn't come far since I can't get it to use my custom dict file and the included dict file is way to large for the software to work fast and accurate.
This is the Error,
Exception in thread "main" java.lang.RuntimeException: Allocation of
search manager resources failed
at edu.cmu.sphinx.decoder.search.WordPruningBreadthFirstSearchManager.allocate(WordPruningBreadthFirstSearchManager.java:247)
at edu.cmu.sphinx.decoder.AbstractDecoder.allocate(AbstractDecoder.java:103)
at edu.cmu.sphinx.recognizer.Recognizer.allocate(Recognizer.java:164)
at edu.cmu.sphinx.api.LiveSpeechRecognizer.startRecognition(LiveSpeechRecognizer.java:47)
at pccomone.Main.main(Main.java:26)
Caused by: java.io.FileNotFoundException: PCCom\src\resource\bodict.dict (The system cannot find the path
specified)
at java.io.FileInputStream.open0(Native Method)
at java.io.FileInputStream.open(Unknown Source)
at java.io.FileInputStream.(Unknown Source)
at java.io.FileInputStream.(Unknown Source)
at sun.net.www.protocol.file.FileURLConnection.connect(Unknown Source)
at sun.net.www.protocol.file.FileURLConnection.getInputStream(Unknown
Source)
at java.net.URL.openStream(Unknown Source)
at edu.cmu.sphinx.linguist.dictionary.TextDictionary.allocate(TextDictionary.java:180)
at edu.cmu.sphinx.linguist.lextree.LexTreeLinguist.allocate(LexTreeLinguist.java:332)
at edu.cmu.sphinx.decoder.search.WordPruningBreadthFirstSearchManager.allocate(WordPruningBreadthFirstSearchManager.java:243)
... 4 more

Java - Get data from website doesn't work (403 Error)

I'm trying to make a program that get data from here but an error appear (403 error)
import java.io.BufferedReader;
import java.io.IOException;
import java.io.InputStream;
import java.io.InputStreamReader;
import java.net.MalformedURLException;
import java.net.URL;
public class Test {
public static void main(String[] args) throws IOException {
URL urlObject;
String codigo;
try{
urlObject=new URL("http://www.pccomponentes.com/intel_core_i5_6600_3_3ghz_box.html");
InputStreamReader isr = new InputStreamReader(urlObject.openStream());
BufferedReader br=new BufferedReader(isr);
while((codigo=br.readLine())!=null)
System.out.println(codigo);
br.close();
}
catch(MalformedURLException e){
e.printStackTrace();
}
catch(IOException e){
e.printStackTrace();
}
}
}
When I run the program this error appear:
java.io.IOException: Server returned HTTP response code: 403 for URL: http://www.pccomponentes.com/intel_core_i5_6600_3_3ghz_box.html
at sun.net.www.protocol.http.HttpURLConnection.getInputStream0(Unknown Source)
at sun.net.www.protocol.http.HttpURLConnection.getInputStream(Unknown Source)
at java.net.URL.openStream(Unknown Source)
at Test.Test.main(Test.java:17)
The purpose of the program it's get the price of the product and print it with a System.out.println, how can I do that?

I have just tested with curl it works, but if I set the User-Agent used by Java by default I get this 403 HTTP error. It seems that the web master of this website doesn't like Java :-)
To work around this, simply set another User-Agent by doing this:
urlObject=new URL("http://www.pccomponentes.com/intel_core_i5_6600_3_3ghz_box.html");
URLConnection c = urlObject.openConnection();
c.setRequestProperty("User-Agent", "<put a the user agent of your choice here>");
InputStreamReader isr = new InputStreamReader(c.getInputStream());
If you don't know which User-Agent to use, use the one of your browser that you can get from here

Create and write a file into hdfs from my local machine

I have two systems connected in the network. One is hdfs running. I want to create a file and write data from my another machine.
package myorg;
import java.io.*;
import java.util.*;
import java.net.*;
import org.apache.hadoop.fs.*;
import org.apache.hadoop.conf.*;
import org.apache.hadoop.io.*;
import org.apache.hadoop.mapred.*;
import org.apache.hadoop.util.*;
public class Write1{
public static void main (String [] args) throws Exception{
try{
System.out.println("Starting...");
Path pt=new Path("hdfs://10.236.173.95:8020/user/jfor/out/gwmdfd");
FileSystem fs = FileSystem.get(new Configuration());
BufferedWriter br=new BufferedWriter(new OutputStreamWriter(fs.create(pt,true)));
// TO append data to a file, use fs.append(Path f)
String line;
line="Disha Dishu Daasha dfasdasdawqeqwe";
System.out.println(line);
br.write(line);
br.close();
}catch(Exception e){
System.out.println("File not found");
}
}
}
I compiled it using
javac -classpath hadoop-0.20.1-dev-core.jar -d Write1/ Write1.java
Created a jar using
jar -cvf Write1.jar -C Write1/ .
Run command
hadoop jar Write1.jar myorg.Write1
If i run this, i am getting
starting...
File not found
What could be the reason? If i run this program, in my hadoop machine, it works fine [I replaced ip with localhost].
Error is at BufferedWriter line. It says "File Not found". what does it mean? I used fs.creat. Then it should create if it doesn't exist. Isn't?
java.lang.IllegalArgumentException: Wrong FS: hdfs://10.72.40.68:8020/user/jfor/..... expected localhost:8020
So i modified the following line
FileSystem fs = FileSystem.get(new URI("hdfs://<ip>:8020"),new Configuration());
It says Connection refused. What could be the reason

About detecting proxy setting using rome api

The below provides an error when in lan behind a proxy ,but works properly outside lan/proxy.
Please let me know how I can rectify it
I used the code to detect proxy setting and it gave
Detecting Windows/IE proxy setting using Java
I am getting: proxy hostname : DIRECT No Proxy Does this mean I am not behind a Proxy Server?
I'm trying to use java rome-fetcher to acquire rss feeds for processing. Everything works fine when I have direct internet access.
However, I need to be able to run my application behind a proxy server.
The below provides an error when in lan ,but works properly outside lan
Exception in thread "main" java.net.ConnectException: Connection timed out: connect
at java.net.PlainSocketImpl.socketConnect(Native Method)
at java.net.PlainSocketImpl.doConnect(Unknown Source)
import java.util.Properties;
import java.net.*;
import java.io.*;
import java.io.FileWriter;
import java.io.Writer;
import java.net.HttpURLConnection;
import java.net.URL;
import java.util.Iterator;
import java.util.List;
import com.sun.syndication.feed.synd.SyndEntry;
import com.sun.syndication.feed.synd.SyndFeed;
import com.sun.syndication.io.SyndFeedInput;
import com.sun.syndication.io.SyndFeedOutput;
import com.sun.syndication.io.XmlReader;
public class RomeLibraryExample {
#SuppressWarnings("unchecked")
public static void main(String[] args) throws Exception {
URL url = new URL("http://rss.cnn.com/rss/cnn_topstories.rss");
//System.setProperty("http.proxyHost", "DIRECT");
// System.setProperty("http.proxyPort", "8080");
HttpURLConnection httpcon = (HttpURLConnection)url.openConnection(Proxy.NO_PROXY);
// Reading the feed
SyndFeedInput input = new SyndFeedInput();
SyndFeed feed = input.build(new XmlReader(httpcon));
List<SyndEntry> entries = feed.getEntries();
Iterator<SyndEntry> itEntries = entries.iterator();
while (itEntries.hasNext()) {
SyndEntry entry = itEntries.next();
System.out.println("Title: " + entry.getTitle());
System.out.println("Link: " + entry.getLink());
System.out.println("Author: " + entry.getAuthor());
System.out.println("Publish Date: " + entry.getPublishedDate());
System.out.println("Description: " + entry.getDescription().getValue());
System.out.println();
}
}
}