The below provides an error when in lan behind a proxy ,but works properly outside lan/proxy.
Please let me know how I can rectify it
I used the code to detect proxy setting and it gave
Detecting Windows/IE proxy setting using Java
I am getting: proxy hostname : DIRECT No Proxy Does this mean I am not behind a Proxy Server?
I'm trying to use java rome-fetcher to acquire rss feeds for processing. Everything works fine when I have direct internet access.
However, I need to be able to run my application behind a proxy server.
The below provides an error when in lan ,but works properly outside lan
Exception in thread "main" java.net.ConnectException: Connection timed out: connect
at java.net.PlainSocketImpl.socketConnect(Native Method)
at java.net.PlainSocketImpl.doConnect(Unknown Source)
import java.util.Properties;
import java.net.*;
import java.io.*;
import java.io.FileWriter;
import java.io.Writer;
import java.net.HttpURLConnection;
import java.net.URL;
import java.util.Iterator;
import java.util.List;
import com.sun.syndication.feed.synd.SyndEntry;
import com.sun.syndication.feed.synd.SyndFeed;
import com.sun.syndication.io.SyndFeedInput;
import com.sun.syndication.io.SyndFeedOutput;
import com.sun.syndication.io.XmlReader;
public class RomeLibraryExample {
#SuppressWarnings("unchecked")
public static void main(String[] args) throws Exception {
URL url = new URL("http://rss.cnn.com/rss/cnn_topstories.rss");
//System.setProperty("http.proxyHost", "DIRECT");
// System.setProperty("http.proxyPort", "8080");
HttpURLConnection httpcon = (HttpURLConnection)url.openConnection(Proxy.NO_PROXY);
// Reading the feed
SyndFeedInput input = new SyndFeedInput();
SyndFeed feed = input.build(new XmlReader(httpcon));
List<SyndEntry> entries = feed.getEntries();
Iterator<SyndEntry> itEntries = entries.iterator();
while (itEntries.hasNext()) {
SyndEntry entry = itEntries.next();
System.out.println("Title: " + entry.getTitle());
System.out.println("Link: " + entry.getLink());
System.out.println("Author: " + entry.getAuthor());
System.out.println("Publish Date: " + entry.getPublishedDate());
System.out.println("Description: " + entry.getDescription().getValue());
System.out.println();
}
}
}
Related
I'm trying to just write some data from an API (google stocks/finance API) to my AWS Firehose stream. I already downloaded and installed the AWS plugin on Eclipse, setup my Firehose stream on AWS, and everything seems to be setup correctly. Am encountering some problems, though. The following line seems to be deprecated...I tried different variations from Amazon's SDK, but I can't seem to get the correct code.
AmazonKinesisFirehoseClient firehoseClient = new
AmazonKinesisFirehoseClient(credentials);
Next, I'm getting some errors with the following. The specific error is, "The method setRecord(Record) is undefined for the type PutRecordRequest," even though I took it directly from Amazon's API reference.
request.setRecord(record);
firehoseClient.putRecord(request);
Also getting an error on the second line above: "The method putRecord(com.amazonaws.services.kinesisfirehose.model.PutRecordRequest) in the type AmazonKinesisFirehoseClient is not applicable for the arguments (com.amazonaws.services.kinesis.model.PutRecordRequest)"
package com.amazonaws.samples;
import java.io.BufferedReader;
import java.io.InputStreamReader;
import java.net.HttpURLConnection;
import java.net.URL;
import java.nio.ByteBuffer;
import org.apache.http.client.CredentialsProvider;
import com.amazonaws.*;
import com.amazonaws.AmazonClientException;
import com.amazonaws.auth.AWSCredentials;
import com.amazonaws.auth.AWSCredentialsProvider;
import com.amazonaws.auth.profile.ProfileCredentialsProvider;
import com.amazonaws.client.builder.AwsClientBuilder;
import com.amazonaws.services.dynamodbv2.AmazonDynamoDBClient;
import com.amazonaws.services.kinesis.AmazonKinesis;
import com.amazonaws.services.kinesis.AmazonKinesisClient;
import com.amazonaws.services.kinesis.AmazonKinesisClientBuilder;
import com.amazonaws.services.kinesis.clientlibrary.interfaces.IRecordProcessorFactory;
import com.amazonaws.services.kinesis.clientlibrary.lib.worker.InitialPositionInStream;
import com.amazonaws.services.kinesis.clientlibrary.lib.worker.KinesisClientLibConfiguration;
import com.amazonaws.services.kinesis.clientlibrary.lib.worker.Worker;
import com.amazonaws.services.kinesis.model.PutRecordRequest;
import com.amazonaws.services.kinesis.model.ResourceNotFoundException;
import com.amazonaws.services.kinesisfirehose.AmazonKinesisFirehoseClient;
import com.amazonaws.services.kinesisfirehose.model.PutRecordBatchRequest;
import com.amazonaws.services.kinesisfirehose.model.Record;
public class FirehoseExample {
public static void main(String[] args) {
AWSCredentials credentials = null;
try {
credentials = new ProfileCredentialsProvider().getCredentials();
}
catch (Exception e) {
throw new AmazonClientException("Cannot load the credentials from the credential profiles file. "
+ "Please make sure that your credentials file is at the correct "
+ "location (/Users/elybenari/.aws/credentials), and is in valid format.", e);
}
AmazonKinesisFirehoseClient firehoseClient = new AmazonKinesisFirehoseClient(credentials);
PutRecordRequest request = new PutRecordRequest();
request.setStreamName("project-stream");
Record record = new Record();
for (int i = 0; i < 20*60; i++){
try {
URL url = new URL("https://www.google.com/finance/info?q=NASDAQ:AMZN");
HttpURLConnection conn = (HttpURLConnection) url.openConnection();
BufferedReader reader = new BufferedReader(new InputStreamReader(conn.getInputStream()));
StringBuilder response = new StringBuilder();
String line;
while ((line = reader.readLine()) != null) {
response.append(line);
}
reader.close();
System.out.println(response.toString().replace("\n", "").replaceAll(" ", ""));
System.out.println("****\n");
ByteBuffer buffer = ByteBuffer.wrap(response.toString().replace("\n", "").replaceAll(" ", "").getBytes());
record.setData(buff);
request.setRecord(record);
firehoseClient.putRecord(request);
Thread.sleep(2000);
}
catch(Exception e){
e.printStackTrace();
}
}
}
}
The problem is that you've included some classes from Kinesis, not Kinesis Firehose, Java package. For e.g., you've used:
import com.amazonaws.services.kinesis.model.PutRecordRequest;
Whereas, you should've used:
import com.amazonaws.services.kinesisfirehose.model.PutRecordRequest;
Kinesis, Kinesis Firehose and Kinesis Analytics are different services, even though they fall under one umbrella of streaming services on AWS. Consequently, they have different package namespaces in the Java SDK. If you start from the official documentation here, you'll reach the correct Java SDK reference here.
EDIT: To answer your other question: yes, the following is deprecated:
AmazonKinesisFirehoseClient firehoseClient = new AmazonKinesisFirehoseClient(credentials);
You should instead use the following:
AmazonKinesisFirehoseClient firehoseClient = AmazonKinesisFirehoseClientBuilder.standard().withCredentials(new AWSStaticCredentialsProvider(awsCredentials)).build();
Refer to the official documentation here on how to correctly initialize AmazonKinesisFirehoseClient.
I am trying to downloada picture from a certain url, but cant do so because I somehow have to give the right userclient to the website.I am sure the problem is that I cant give the user client while using the Url class, because the page can be accesed via browser. I tried using proxy and Urlconnection but couldnt get it to work. Please share your toughts on the matter!
My code is the following:
import java.awt.image.BufferedImage;
import java.io.File;
import java.io.IOException;
import java.io.InputStream;
import java.net.MalformedURLException;
import java.net.Proxy;
import java.net.SocketAddress;
import java.net.URL;
import java.net.URLConnection;
import javax.imageio.ImageIO;
import org.opencv.core.Core;
import org.opencv.core.Mat;
import org.opencv.highgui.Highgui;
public class KepLetolto {
public static void main(String[] args) throws IOException {
System.loadLibrary(Core.NATIVE_LIBRARY_NAME);
File file = new File("tempjpeg");
SocketAddress address = new java.net.InetSocketAddress("xyz.com", 8080);
// Create an HTTP Proxy using the above SocketAddress.
Proxy proxy = new Proxy(Proxy.Type.HTTP, address);
URL url_kep =new URL("http://www.theouthousers.com/images/templates/thumbnails/128058/bayfinger_size3.png");
ImageIO.write(ImageIO.read(url_kep), "jpeg", file);
Mat uj = Highgui.imread("temp.jpeg" ,Highgui.CV_LOAD_IMAGE_COLOR);
}
}
Instead of using ImageIO.read(URL), which limits you to the default behavior of the URL's underlying URLConnection, use ImageIO.read(InputStream).
This allows you to use any HTTP client library - including the basic HttpURLConnection, which you can get from (HttpURLConnection)url_kep.openConnection(). Using that, you can set headers such as User-Agent, if that's the header required by the site, or other headers such as Referer which are sometimes used to prevent deep-linking.
Once you set up all the headers and any other request options, you can get an InputStream from the client object, and pass that to ImageIO.
This Solution Worked For Me:
URLConnection openConnection = new URL("YOUR_IMAGE_URL").openConnection();
openConnection.addRequestProperty("User-Agent", "YOUR USER AGENT");
InputStream is = openConnection.getInputStream();
BufferedImage saveImage = ImageIO.read(is);
ImageIO.write(saveImage, "png", new File("PATH\\TO\\IMAGE\\FILE.PNG"));
I have create a java program that will connect to unix server1 (with user name pwd),now i need to connect another server (with username password) and execute command to push data from server 1 to server 2 using java program.
Server1 connection is working fine and i can execute some basic commands in it.
Code below
import java.io.File;
import java.io.FileInputStream;
import java.io.FileNotFoundException;
import java.io.FileOutputStream;
import java.io.IOException;
import java.io.InputStream;
import java.text.DateFormat;
import java.text.ParseException;
import java.text.SimpleDateFormat;
import java.util.ArrayList;
import java.util.Date;
import java.util.HashMap;
import java.util.Properties;
import net.neoremind.sshxcute.core.ConnBean;
import net.neoremind.sshxcute.core.Result;
import net.neoremind.sshxcute.core.SSHExec;
import net.neoremind.sshxcute.exception.TaskExecFailException;
import net.neoremind.sshxcute.task.CustomTask;
import net.neoremind.sshxcute.task.impl.ExecCommand;
public class UnixConnect {
// String hostName, String logFile, String userName, String password
static String[] host_names = null;
static String[] user_names = null;
static String[] pwd_text = null;
public void execCommand() throws TaskExecFailException {
ConnBean cb1 = new ConnBean("0.000.00.000", "***", "****");
SSHExec ssh1 = SSHExec.getInstance(cb1);
ssh1.connect();
String[] cmd = {"cd /apps/a/b/c/logs","tail -1 aLoadJob.log"};
CustomTask tasks = new ExecCommand(cmd);
Result res1 = ssh1.exec(tasks);
if (res1.isSuccess) {
/*System.out.println(res1.sysout)*/;
} else {
System.out.println("Return code: " + res1.sysout);
System.out.println("error message: " + res1.error_msg);
}
ssh1.disconnect();
}
/**
* #param args
*/
public static void main(String[] args) throws TaskExecFailException {
UnixConnect ob = new UnixConnect();
ob.execCommand();
}
}
In both Server i need to give username and password
This is a bit strange use case but I would do the following:
Implement bash script on server1 that would do all the job of transfering data to server2. Of course server1 has to be able to connecto server2
Use you program to invoke the bash script on server1.
I am attempting to connect to Intuit Quickbooks with OAuth, using their JAVA API V3, hosted on AppEngine (SDK 1.8.4). To get started, I'm just running their sample code from their instructions here: http://ippdocs.intuit.com/0025_QuickBooksAPI/0055_DevKits/0201_IPP_Java_DevKit_3.0
It's working well on TomCat, but when I adapt the code to work on AppEngine, I get this error:
OauthHelper.java Extract, the exception is raised at the last line:
import java.io.BufferedReader;
import java.io.IOException;
import java.io.InputStream;
import java.io.InputStreamReader;
import java.io.UnsupportedEncodingException;
import java.net.HttpURLConnection;
import java.net.MalformedURLException;
import java.net.URL;
import java.net.URLEncoder;
import java.util.HashMap;
import java.util.Map;
import oauth.signpost.OAuthConsumer;
import oauth.signpost.OAuthProvider;
import oauth.signpost.basic.DefaultOAuthConsumer;
import oauth.signpost.basic.DefaultOAuthProvider;
import oauth.signpost.exception.OAuthCommunicationException;
import oauth.signpost.exception.OAuthExpectationFailedException;
import oauth.signpost.exception.OAuthMessageSignerException;
import oauth.signpost.exception.OAuthNotAuthorizedException;
import oauth.signpost.http.HttpParameters;
[...]
URL url;
url = new URL(signedRequestTokenUrl);
HttpURLConnection httpconnection = (HttpURLConnection) url
httpconnection.setRequestMethod("GET");
httpconnection.setRequestProperty("Content-type", "application/xml");
httpconnection.setRequestProperty("Content-Length", "0");
if (httpconnection != null) {
BufferedReader rd = new BufferedReader(new InputStreamReader(
httpconnection.getInputStream()));
[...]
Causing this Error:
javax.net.ssl.SSLHandshakeException: Could not verify SSL certificate for URL: https://oauth.intuit.com/oauth/v1/get_request_token?oauth_signature=VzStL8UcIoDrgKdcU7jJAWaux5Y%3D&oauth_callback=http%3A%2F%2Flocalhost%3A8888%2Faccesstoken.htm&oauth_consumer_key=qyprdulFn7zfTw5ewpZhkPxSo4q27X&oauth_version=1.0&oauth_signature_method=HMAC-SHA1&oauth_timestamp=1379761817&oauth_nonce=5706619579888946790
at com.google.appengine.api.urlfetch.URLFetchServiceImpl.convertApplicationException(URLFetchServiceImpl.java:144)
at com.google.appengine.api.urlfetch.URLFetchServiceImpl.fetch(URLFetchServiceImpl.java:43)
at com.google.apphosting.utils.security.urlfetch.URLFetchServiceStreamHandler$Connection.fetchResponse(URLFetchServiceStreamHandler.java:417)
at com.google.apphosting.utils.security.urlfetch.URLFetchServiceStreamHandler$Connection.getInputStream(URLFetchServiceStreamHandler.java:296)
at com.intuit.utils.OauthHelper.getRequestTokenSignPost(OauthHelper.java:167)
The url is as follows:
https://oauth.intuit.com/oauth/v1/get_request_token?oauth_signature=xxxx&oauth_callback=http%3A%2F%2Flocalhost%3A8888%2Faccesstoken.htm&oauth_consumer_key=xxxx&oauth_version=1.0&oauth_signature_method=HMAC-SHA1&oauth_timestamp=1379768671&oauth_nonce=-78989043
The exact same piece of code works perfectly on TomCat 7.
Signpost
I also tried to replace DefaultOAuthConsumer & DefaultOAuthProvider by the following:
oauth.signpost.commonshttp.CommonsHttpOAuthProvider;
oauth.signpost.commonshttp.CommonsHttpOAuthConsumer;
, but it generates exactly the same exception.
Any ideas?
And right after I say that I tried something that to me makes little sense, but seems to work, try this:
URLFetchService fetcher = URLFetchServiceFactory.getURLFetchService();
FetchOptions lFetchOptions = FetchOptions.Builder.validateCertificate();
HTTPRequest request = new HTTPRequest(url, HTTPMethod.GET, lFetchOptions);
HTTPResponse response = fetcher.fetch(request);
For me it was the FetchOptions.Builder and explicitly setting "validateCertificate()"; See if this solve it for you.
How do I determine whether a web proxy IP is of type HTTP or SOCKS4/5 with java?
Thank you.
As mentioned in the comments from my other answer, if you know the IP address of a proxy server and want to detect what type it is, you could try each proxy type in Java until one works.
import java.io.IOException;
import java.net.InetSocketAddress;
import java.net.Proxy;
import java.net.SocketException;
import java.net.URL;
import java.net.URLConnection;
import java.util.Arrays;
import java.util.List;
public class ProxyTest
{
public static void main(String... args)
throws IOException
{
InetSocketAddress proxyAddress = new InetSocketAddress("myproxyaddress", 1234);
Proxy.Type proxyType = detectProxyType(proxyAddress);
System.out.println(proxyAddress + " is a " + proxyType + " proxy.");
}
public static Proxy.Type detectProxyType(InetSocketAddress proxyAddress)
throws IOException
{
URL url = new URL("http://www.google.com");
List<Proxy.Type> proxyTypesToTry = Arrays.asList(Proxy.Type.SOCKS, Proxy.Type.HTTP);
for (Proxy.Type proxyType : proxyTypesToTry)
{
Proxy proxy = new Proxy(proxyType, proxyAddress);
//Try with SOCKS
URLConnection connection = null;
try
{
connection = url.openConnection(proxy);
//Can modify timeouts if default timeout is taking too long
//connection.setConnectTimeout(1000);
//connection.setReadTimeout(1000);
connection.getContent();
//If we get here we made a successful connection
return(proxyType);
}
catch (SocketException e) //or possibly more generic IOException?
{
//Proxy connection failed
}
}
//No proxies worked if we get here
return(null);
}
}
In this code, it first tries to connect to www.google.com using the proxy at myproxyaddress with SOCKS, and if that fails it will try using it as an HTTP proxy, returning the method that worked, or null if none worked.
If you want to determine the type of proxy being used from Java, you can use ProxySelector and Proxy.
e.g.
import java.net.Proxy;
import java.net.ProxySelector;
import java.net.URI;
import java.util.List;
public class ProxyTest
{
public static void main(String... args)
{
System.setProperty("java.net.useSystemProxies", "true");
List<Proxy> proxyList = ProxySelector.getDefault().select(URI.create("http://www.google.com"));
if (!proxyList.isEmpty())
{
Proxy proxy = proxyList.get(0);
switch (proxy.type())
{
case DIRECT:
System.out.println("Direct connection - no proxy.");
break;
case HTTP:
System.out.println("HTTP proxy: " + proxy.address());
break;
case SOCKS:
System.out.println("SOCKS proxy: " + proxy.address());
break;
}
}
}
}