Java - Get data from website doesn't work (403 Error) - java

I'm trying to make a program that get data from here but an error appear (403 error)
import java.io.BufferedReader;
import java.io.IOException;
import java.io.InputStream;
import java.io.InputStreamReader;
import java.net.MalformedURLException;
import java.net.URL;
public class Test {
public static void main(String[] args) throws IOException {
URL urlObject;
String codigo;
try{
urlObject=new URL("http://www.pccomponentes.com/intel_core_i5_6600_3_3ghz_box.html");
InputStreamReader isr = new InputStreamReader(urlObject.openStream());
BufferedReader br=new BufferedReader(isr);
while((codigo=br.readLine())!=null)
System.out.println(codigo);
br.close();
}
catch(MalformedURLException e){
e.printStackTrace();
}
catch(IOException e){
e.printStackTrace();
}
}
}
When I run the program this error appear:
java.io.IOException: Server returned HTTP response code: 403 for URL: http://www.pccomponentes.com/intel_core_i5_6600_3_3ghz_box.html
at sun.net.www.protocol.http.HttpURLConnection.getInputStream0(Unknown Source)
at sun.net.www.protocol.http.HttpURLConnection.getInputStream(Unknown Source)
at java.net.URL.openStream(Unknown Source)
at Test.Test.main(Test.java:17)
The purpose of the program it's get the price of the product and print it with a System.out.println, how can I do that?

I have just tested with curl it works, but if I set the User-Agent used by Java by default I get this 403 HTTP error. It seems that the web master of this website doesn't like Java :-)
To work around this, simply set another User-Agent by doing this:
urlObject=new URL("http://www.pccomponentes.com/intel_core_i5_6600_3_3ghz_box.html");
URLConnection c = urlObject.openConnection();
c.setRequestProperty("User-Agent", "<put a the user agent of your choice here>");
InputStreamReader isr = new InputStreamReader(c.getInputStream());
If you don't know which User-Agent to use, use the one of your browser that you can get from here

Related

GSON library error with Lotus Notes Java Agent - java.lang.NoClassDefFoundError: com.google.gson.JsonObject

I am trying to use GSON library to convert Java object to JSON in a Java Agent in my Lotus Notes application. I have added GSON jar file to Project->Java Build Path. But when I run the agent, I am getting error - "Exception in thread "AgentThread: JavaAgent" java.lang.NoClassDefFoundError: com.google.gson.JsonObject".
Basically what I am trying to achieve is to get some JSON from external API and then save it in Lotus Notes database. I am able to send HTTP request, but what I am getting in return is a Java object. I want to convert it to JSON.
Here's the class file to,
import java.io.BufferedReader;
import java.io.IOException;
import java.io.InputStream;
import java.io.InputStreamReader;
import java.net.MalformedURLException;
import java.net.URL;
import java.net.URLConnection;
import org.apache.commons.validator.routines.UrlValidator;
import com.google.gson.JsonObject;
import com.google.gson.JsonParser;
public class CustRestConsumer {
/**
* Method for receiving HTTP JSON GET request against a RESTful URL data source.
*
* #param myUrlStr the URL of the REST endpoint
* #return JsonObject containing the data from the REST response.
* #throws IOException
* #throws MalformedURLException
* #throws ParseException
*/
public static JsonObject GetMyRestData( String myUrlStr ) throws IOException, MalformedURLException {
JsonObject myRestData = new JsonObject();
try{
UrlValidator defaultValidator = new UrlValidator();
if(defaultValidator.isValid(myUrlStr)){
URL myUrl = new URL(myUrlStr);
URLConnection urlCon = myUrl.openConnection();
urlCon.setConnectTimeout(5000);
InputStream is = urlCon.getInputStream();
InputStreamReader isR = new InputStreamReader(is);
BufferedReader reader = new BufferedReader(isR);
StringBuffer buffer = new StringBuffer();
String line = "";
while( (line = reader.readLine()) != null ){
buffer.append(line);
}
reader.close();
JsonParser parser = new JsonParser();
myRestData = (JsonObject) parser.parse(buffer.toString());
return myRestData;
}else{
myRestData.addProperty("error", "URL failed validation by Apache Commmons URL Validator");
return myRestData;
}
}catch( MalformedURLException e ){
e.printStackTrace();
myRestData.addProperty("error", e.toString());
return myRestData;
}catch( IOException e ){
e.printStackTrace();
myRestData.addProperty("error", e.toString());
return myRestData;
}
}
}
I am calling the function here,
import com.google.gson.JsonObject;
import lotus.domino.*;
public class JavaAgent extends AgentBase {
public void NotesMain() {
try {
Session session = getSession();
AgentContext agentContext = session.getAgentContext();
// (Your code goes here)
Database db = agentContext.getCurrentDatabase();
String url = "https://jsonplaceholder.typicode.com/todos/1";
System.out.println("Reached Here - 1");
JsonObject myStuff = CustRestConsumer.GetMyRestData(url);
System.out.println("Reached Here - 2");
System.out.println(myStuff);
Document newNotesDoc = db.createDocument();
newNotesDoc.replaceItemValue("Form", "IBMForm");
// newNotesDoc.replaceItemValue("WebPageUS", dto.title);
newNotesDoc.computeWithForm(true, false);
newNotesDoc.save(true, true);
db.recycle();
} catch(Exception e) {
e.printStackTrace();
}
}
}
This is the Java debug console output,
Reached Here - 1
Exception in thread "AgentThread: JavaAgent" java.lang.NoClassDefFoundError: com.google.gson.JsonObject
at CustRestConsumer.GetMyRestData(Unknown Source)
at JavaAgent.NotesMain(Unknown Source)
at lotus.domino.AgentBase.runNotes(Unknown Source)
at lotus.domino.NotesThread.run(Unknown Source)
Caused by: java.lang.ClassNotFoundException: com.google.gson.JsonObject
at lotus.domino.AgentLoader.loadClass(Unknown Source)
at java.lang.ClassLoader.loadClass(ClassLoader.java:852)
I am new to Java agents, so any help will be much appreciated.
Thanks.
Project->Java Build Path is for using JAR files in XPages. I believe the jar file can be added directly to the agent or a script library, but there are memory leak issues with detaching them, so it's not recommended. The recommended approach is to add them to the jvm\lib\ext of the server and, presumably, also the client that is trying to compile the code.
There are other options for scheduling Java code in Domino, some of which are covered here https://www.intec.co.uk/tag/xots-microservice-scheduler-tutorial/, but there are other imaginative approaches possible.

Java (JRE 1.8.0_141) - Error 405 for GET request

I am using Java JRE 1.8.0_141 and I am trying to access a specific URL and store the HTML into a String so that I can manipulate the data later in the code, but I keep getting error 405 whenever I call getInputStream().
The code seems to work with other URLs without problems. The trouble URL is:
http://www.streeteasy.com/for-rent/nyc/status:open%7Cprice:1750-2900%7Carea:104,116,119,143,141%7Camenities:pool?page=2&refined_search=true
Here is Eclipse 4.6.3's specific error:
<terminated, exit value: 1>C:\Program Files\Java\jre1.8.0_141\bin\javaw.exe (Aug 6, 2017, 10:53:37 PM)
Exception in thread "main" java.lang.RuntimeException: java.io.IOException: Server returned HTTP response code: 405 for URL: http://www.streeteasy.com/for-rent/nyc/status:open%7Cprice:1750-2900%7Carea:104,116,119,143,141%7Camenities:pool?page=2&refined_search=true
at RunMe.getHTMLFromURL(RunMe.java:52)
at RunMe.main(RunMe.java:18)
Caused by: java.io.IOException: Server returned HTTP response code: 405 for URL: http://www.streeteasy.com/for-rent/nyc/status:open%7Cprice:1750-2900%7Carea:104,116,119,143,141%7Camenities:pool?page=2&refined_search=true
at sun.net.www.protocol.http.HttpURLConnection.getInputStream0(Unknown Source)
at sun.net.www.protocol.http.HttpURLConnection.getInputStream(Unknown Source)
at RunMe.getHTMLFromURL(RunMe.java:36)
... 1 more
My RunMe.java code is below:
import java.io.BufferedReader;
import java.io.File;
import java.io.FileReader;
import java.io.IOException;
import java.io.InputStreamReader;
import java.net.URL;
import java.net.URLConnection;
import java.util.LinkedList;
public class RunMe {
public static void main(String[] args) throws IOException {
// TODO Auto-generated method stub
System.out.println(getHTMLFromURL("http://www.streeteasy.com/for-rent/nyc/status:open%7Cprice:1750-2900%7Carea:104,116,119,143,141%7Camenities:pool?page=2&refined_search=true"));
}
public static String getHTMLFromURL(String url){
try{
URL urlObj = new URL(url);
URLConnection con = urlObj.openConnection();
con.setDoOutput(false);
con.connect();
BufferedReader in = new BufferedReader(new InputStreamReader(con.getInputStream()));
// CODE FAILS HERE ^
StringBuilder response = new StringBuilder();
String inputLine;
String newLine = System.getProperty("line.separator");
while ((inputLine = in.readLine()) != null){
response.append(inputLine + newLine);
}
in.close();
return response.toString();
}
catch (Exception e){
throw new RuntimeException(e);
}
}
}
Any idea how I can pull the HTML from this URL if not via this method? Thanks in advance!
I executed a curl command against the URL and it seems like the site is trying to run JavaScript for rendering the page.
curl -v -L -H "User-Agent: Mozilla/5.0" -H "Accept: text/html" "http://www.streeteasy.com/for-rent/nyc/status:open%7Cprice:1750-2900%7Carea:104,116,119,143,141%7Camenities:pool?page=2"
> GET /for-rent/nyc/status:open%7Cprice:1750-2900%7Carea:104,116,119,143,141%7Camenities:pool?page=2 HTTP/1.1
> Host: www.streeteasy.com
> User-Agent: Mozilla/5.0
> Accept: text/html
>
< HTTP/1.1 405 Not Allowed
// elided
<h1>Pardon Our Interruption...</h1>
<p>As you were browsing <strong>www.streeteasy.com</strong> something about your browser made us think you were a bot. There are a few reasons this might happen:</p>
<ul>
<li>You're a power user moving through this website with super-human speed.</li>
<li>You've disabled JavaScript in your web browser.</li>
<li>A third-party browser plugin, such as Ghostery or NoScript, is preventing JavaScript from running. Additional information is available in this <a title='Third party browser plugins that block javascript' href='http://ds.tl/help-third-party-plugins' target='_blank'>support article</a>.</li>
</ul>
<p>After completing the CAPTCHA below, you will immediately regain access to www.streeteasy.com.</p>
You may be outta luck here unless you can programmatically fill in the captcha.
Edit:
The problem is clearly cookies as indicated in the discussion below.

JAVA apps and ECLIPSE can't connect to internet

Ok so I wrote a piece of code testing ability of my java to connect to internet. It is supposed to fetch html from www.google.com and display the contents in a JFrame's JTextArea object.
Here's the code, so you can have clear picture:
import java.awt.Color;
import java.awt.Dimension;
import java.io.BufferedReader;
import java.io.IOException;
import java.io.InputStream;
import java.io.InputStreamReader;
import java.net.URL;
import javax.swing.JFrame;
import javax.swing.JTextArea;
import org.jsoup.Jsoup;
import org.jsoup.nodes.Document;
public class JSoupFetchTest extends JFrame{
String userAgent = "Mozilla/5.0 (Windows NT 6.1; WOW64; rv:37.0) Gecko/20100101 Firefox/37.0";
boolean jsoupcond = true;
String address = "http://www.google.com";
JTextArea text;
public JSoupFetchTest(){
text = new JTextArea();
text.setPreferredSize(new Dimension(500, 500));
text.setBackground(Color.BLACK);
text.setForeground(Color.WHITE);
text.setVisible(true);
text.setLineWrap(true);
this.add(text);
this.setDefaultCloseOperation(JFrame.EXIT_ON_CLOSE);
this.setVisible(true);
this.pack();
gogo();
}
private void gogo() {
if(jsoupcond){
text.setText(text.getText() +"\nstart...");
try {
text.setText(text.getText() +"\nConnecting to " +address+ "...");
Document doc = Jsoup.connect(address).userAgent(userAgent).get();
text.setText(text.getText() +"\nConverting page document into text");
String s = doc.toString();
text.setText(text.getText() +"\nText: \n" +s);
System.out.println();
} catch (Exception e) {
text.setText(text.getText() +"\n" +e.toString());
e.printStackTrace();
}
text.setText(text.getText() +"\nEnd.");
}
String html = downloadHtml(address);
text.setText(text.getText() +"\nDownloading HTML...");
text.setText(text.getText() +"\nHTML:");
text.setText(text.getText() +"\n" +html);
}
private String downloadHtml(String path) {
text.setText(text.getText() +"\ndownloadHtml entry point...");
InputStream is = null;
try {
text.setText(text.getText() +"\ntry block entered...");
String result = "";
String line;
URL url = new URL(path);
text.setText(text.getText() +"\nabout to open url stream...");
is = url.openStream(); // throws an IOException
text.setText(text.getText() +"\nurl stream opened...");
BufferedReader br = new BufferedReader(new InputStreamReader(is));
text.setText(text.getText() +"\nstarting to read lines...");
while ((line = br.readLine()) != null) {
result += line;
}
text.setText(text.getText() +"\nreading lines finished...");
return result;
} catch (IOException ioe) {
ioe.printStackTrace();
} finally {
try {
if (is != null) is.close();
} catch (IOException ioe) { }
}
return "";
}
public static void main(String[] args) {
new JSoupFetchTest();
}
}
I should also add that:
1. My eclipse (cause that's what I'm using) can't connect to marketplace nor can't fetch updates.
2. Eclipse's web browser works fine.
3. My system's browser (Mozilla Firefox) connects fine
4. I exported JSoupFetchTest into a runnable jar and tried to run it from system's level, with no effect
5. I am running Windows 7 Professional MSDN version
6. I contacted eclipse support and they concluded it is not eclipse's fault and suggested that I'm behind a proxy.
7. I contacted my ISP to see if I indeed am and they said I am not.
8. I changed my JAVA's network settings so now it connects "directly". Before the setting was "use browser settings" and it didn't work either.
9. My eclipse's Window -> Preferences -> General -> Network Connections active provider is set to "Native", I also tried "Direct"
10. Method downloadHtml(String path) stops at "is = url.openStream();" and goes on forever...
The exception I get from JSoup is:
java.net.SocketTimeoutException: Read timed out
at java.net.SocketInputStream.socketRead0(Native Method)
at java.net.SocketInputStream.read(SocketInputStream.java:150)
at java.net.SocketInputStream.read(SocketInputStream.java:121)
at java.io.BufferedInputStream.fill(BufferedInputStream.java:246)
at java.io.BufferedInputStream.read1(BufferedInputStream.java:286)
at java.io.BufferedInputStream.read(BufferedInputStream.java:345)
at sun.net.www.http.HttpClient.parseHTTPHeader(HttpClient.java:703)
at sun.net.www.http.HttpClient.parseHTTP(HttpClient.java:647)
at sun.net.www.protocol.http.HttpURLConnection.getInputStream0(HttpURLConnection.java:1534)
at sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1439)
at java.net.HttpURLConnection.getResponseCode(HttpURLConnection.java:480)
at org.jsoup.helper.HttpConnection$Response.execute(HttpConnection.java:453)
at org.jsoup.helper.HttpConnection$Response.execute(HttpConnection.java:434)
at org.jsoup.helper.HttpConnection.execute(HttpConnection.java:181)
at org.jsoup.helper.HttpConnection.get(HttpConnection.java:170)
at JSoupFetchTest.gogo(JSoupFetchTest.java:42)
at JSoupFetchTest.<init>(JSoupFetchTest.java:32)
at JSoupFetchTest.main(JSoupFetchTest.java:92)
I also tried to set JSoup.connect's timeout to infinity. Then it goes on forever.
Before you guys say that my question is a duplicate, or delegate me to other, external possible solutions to my problem, believe me - either the question is mine or I was there - I browse internet in search for solution for weeks now and I feel like pulling my hair out...
Please help if you can cause it prevents me from installing stuff in my eclipse and from developing anything else than stand alone apps...
You need a socket number after the URL -- "http:/www.google.com:80" works. JSoup likely uses defaults for that, but opening the URL as a stream in Java does not.
The following program works for me. So Java and JSoup are working. It has to be some sort of local configuration problem with your network. Check your firewall, routers, gateway, and Java permissions. Do a clean rebuild of your project. Etc. Comment out lines until it does work and then put the lines back one at a time until you find the problem. Etc.
package stuff;
import java.io.BufferedReader;
import java.io.InputStream;
import java.io.InputStreamReader;
import java.io.Reader;
import java.net.URL;
import java.net.URLConnection;
import org.jsoup.Jsoup;
import org.jsoup.nodes.Document;
public class SocketTest
{
public static void main( String[] args ) throws Exception
{
URL url = new URL( "http://www.google.com" );
URLConnection sock = url.openConnection();
InputStream ins = sock.getInputStream();
BufferedReader reader = new BufferedReader( new InputStreamReader(ins, "UTF-8" ) );
for( String line; (line = reader.readLine()) != null; ) {
System.out.println( line );
}
ins.close();
Document doc = Jsoup.connect( "http://www.google.com" ).get();
System.out.println( doc.toString() );
String userAgent = "Mozilla/5.0 (Windows NT 6.1; WOW64; rv:37.0) Gecko/20100101 Firefox/37.0";
Document doc2 = Jsoup.connect( "http://www.google.com" ).userAgent(userAgent).get();
System.out.println( doc2.toString() );
}
}

java Connection reset error

I am quite a newbie to Java. Please excuse me if you find this as a very basic question.There are many answers available already in stack overflow about this and I went through almost all the possible helps i can get in Stack overflow and also in some other forums. Unfortunately none of them helped me.
I have client/server program in which the client send a string to server and server just attaches another string to the string sent by client and sends it back to the client.
Server program looks like this.
import java.io.DataInputStream;
import java.io.DataOutputStream;
import java.io.InputStream;
import java.io.OutputStream;
import java.net.ServerSocket;
import java.net.Socket;
public class server {
public static void main(String[] args) {
try
{
ServerSocket server = new ServerSocket(7300);
Socket s = server.accept();
DataInputStream inp = new DataInputStream(s.getInputStream());
DataOutputStream out = new DataOutputStream(s.getOutputStream());
String str =inp.readUTF();
str = str+" buddy!";
out.writeUTF(str);
}
catch(Exception e)
{
e.printStackTrace();
}
}
}
Client looks like This.
import java.io.DataInputStream;
import java.io.DataOutputStream;
import java.net.*;
public class client {
public static void main(String[] args) {
try
{
Socket s = new Socket("192.168.1.3",7300);
DataInputStream inp = new DataInputStream(s.getInputStream());
DataOutputStream out = new DataOutputStream(s.getOutputStream());
out.writeUTF("hi");
System.out.println(inp.readUTF());
Thread.sleep(2000);
out.writeUTF("hello");
System.out.println(inp.readUTF());
}
catch(Exception e)
{
e.printStackTrace();
}
}
}
Everything works fine while client writes "hi" and when client starts sending "hello" i am getting Connection reset error. I am not getting what mistake am i doing please help me in resolving this.
The output with the error i am getting looks like this.
hi buddy!
java.net.SocketException: Connection reset by peer: socket write error
at java.net.SocketOutputStream.socketWrite0(Native Method)
at java.net.SocketOutputStream.socketWrite(Unknown Source)
at java.net.SocketOutputStream.write(Unknown Source)
at java.io.DataOutputStream.write(Unknown Source)
at java.io.DataOutputStream.writeUTF(Unknown Source)
at java.io.DataOutputStream.writeUTF(Unknown Source)
at sokry.client.main(client.java:18)
In your server example, readUTF is only called once on the DataInputStream, even though the client wrote to the DataOutputStream twice. Thus, simply adding
str = inp.readUTF();
str = str + " buddy!";
out.writeUTF(str);
to your server example, after the last out.writeUTF(str), will solve your problem.
do comment on following line of your client.java file and try.it will work
Thread.sleep(2000);
`//out.writeUTF("hello");;
//System.out.println(inp.readUTF());
because when you are sending "hi" from client to server and server gives reply then it finished it work and it stop connection but in client.java you sending another request to server but server is at rest.
you should start server until client finish it work..
hope it will wait

Cannot access azure blobs through rest api

I was able to create a Container in Storage Account and upload a blob to it through the Client Side Code.
I was able to make the blob available for Public access as well , such that when I hit the following query from my browser, I am able to see the image which I uploaded.
https://MYACCOUNT.blob.core.windows.net/MYCONTAINER/MYBLOB
I now have a requirement to use the rest service to retrieve the contents of the blob. I wrote down the following java code.
package main;
import java.io.BufferedReader;
import java.io.IOException;
import java.io.InputStream;
import java.io.InputStreamReader;
import java.net.HttpURLConnection;
import java.net.URL;
import java.security.MessageDigest;
import java.security.NoSuchAlgorithmException;
import java.text.SimpleDateFormat;
import java.util.Date;
import java.util.TimeZone;
public class GetBlob {
public static void main(String[] args) {
String url="https://MYACCOUNT.blob.core.windows.net/MYCONTAINER/MYBLOB";
try {
System.out.println("RUNNIGN");
HttpURLConnection connection = (HttpURLConnection) new URL(url).openConnection();
connection.setRequestProperty("Authorization", createQuery());
connection.setRequestProperty("x-ms-version", "2009-09-19");
InputStream response = connection.getInputStream();
System.out.println("SUCCESSS");
String line;
BufferedReader reader = new BufferedReader(new InputStreamReader(response));
while ((line = reader.readLine()) != null) {
System.out.println(line);
}
} catch (IOException e) {
e.printStackTrace();
}
}
public static String createQuery()
{
String dateFormat="EEE, dd MMM yyyy hh:mm:ss zzz";
SimpleDateFormat dateFormatGmt = new SimpleDateFormat(dateFormat);
dateFormatGmt.setTimeZone(TimeZone.getTimeZone("UTC"));
String date=dateFormatGmt.format(new Date());
String Signature="GET\n\n\n\n\n\n\n\n\n\n\n\n" +
"x-ms-date:" +date+
"\nx-ms-version:2009-09-19" ;
// I do not know CANOCALIZED RESOURCE
//WHAT ARE THEY??
// +"\n/myaccount/myaccount/mycontainer\ncomp:metadata\nrestype:container\ntimeout:20";
String SharedKey="SharedKey";
String AccountName="MYACCOUNT";
String encryptedSignature=(encrypt(Signature));
String auth=""+SharedKey+" "+AccountName+":"+encryptedSignature;
return auth;
}
public static String encrypt(String clearTextPassword) {
try {
MessageDigest md = MessageDigest.getInstance("SHA-256");
md.update(clearTextPassword.getBytes());
return new sun.misc.BASE64Encoder().encode(md.digest());
} catch (NoSuchAlgorithmException e) {
}
return "";
}
}
However , I get the following error when I run this main class...
RUNNIGN
java.io.IOException: Server returned HTTP response code: 403 for URL: https://klabs.blob.core.windows.net/delete/Blob_1
at sun.net.www.protocol.http.HttpURLConnection.getInputStream(Unknown Source)
at sun.net.www.protocol.https.HttpsURLConnectionImpl.getInputStream(Unknown Source)
at main.MainClass.main(MainClass.java:61)
Question1: Why this error, did I miss any header/parameter?
Question2: Do I need to add headers in the first place, because I am able to hit the request from the browser without any issues.
Question3: Can it be an SSL issue? What is the concept of certificates, and how and where to add them? Do I really need them? Will I need them later, when I do bigger operations on my blob storage(I want to manage a thousand blobs)?
Will be thankful for any reference as well, within Azure and otherwise that could help me understand better.
:D
AFTER A FEW DAYS
Below is my new code for PutBlob I azure. I believe I have fully resolved all header and parameter issues and my request is perfect. However I am still getting the same 403. I do not know what the issue is. Azure is proving to be pretty difficult.
A thing to note is that the containers name is delete, and I want to create a blob inside it, say newBlob. I tried to initialize the urlPath in the code below with both "delete" and "delete/newBlob".
Does not work..
package main;
import java.io.DataInputStream;
import java.io.DataOutputStream;
import java.io.File;
import java.io.IOException;
import java.io.UnsupportedEncodingException;
import java.net.HttpURLConnection;
import java.net.URISyntaxException;
import java.net.URL;
import java.security.InvalidKeyException;
import java.security.NoSuchAlgorithmException;
import java.text.SimpleDateFormat;
import java.util.Calendar;
import java.util.TimeZone;
import javax.crypto.Mac;
import javax.crypto.spec.SecretKeySpec;
import com.sun.org.apache.xml.internal.security.exceptions.Base64DecodingException;
import com.sun.org.apache.xml.internal.security.utils.Base64;
public class Internet {
static String key="password";
static String account="klabs";
private static Base64 base64 ;
private static String createAuthorizationHeader(String canonicalizedString) throws InvalidKeyException, Base64DecodingException, NoSuchAlgorithmException, IllegalStateException, UnsupportedEncodingException {
Mac mac = Mac.getInstance("HmacSHA256");
mac.init(new SecretKeySpec(base64.decode(key), "HmacSHA256"));
String authKey = new String(base64.encode(mac.doFinal(canonicalizedString.getBytes("UTF-8"))));
String authStr = "SharedKey " + account + ":" + authKey;
return authStr;
}
public static void main(String[] args) {
System.out.println("INTERNET");
String key="password";
String account="klabs";
long blobLength="Dipanshu Verma wrote this".getBytes().length;
File f = new File("C:\\Users\\Dipanshu\\Desktop\\abc.txt");
String requestMethod = "PUT";
String urlPath = "delete";
String storageServiceVersion = "2009-09-19";
SimpleDateFormat fmt = new SimpleDateFormat("EEE, dd MMM yyyy HH:mm:sss");
fmt.setTimeZone(TimeZone.getTimeZone("UTC"));
String date = fmt.format(Calendar.getInstance().getTime()) + " UTC";
String blobType = "BlockBlob";
String canonicalizedHeaders = "x-ms-blob-type:"+blobType+"\nx-ms-date:"+date+"\nx-ms-version:"+storageServiceVersion;
String canonicalizedResource = "/"+account+"/"+urlPath;
String stringToSign = requestMethod+"\n\n\n"+blobLength+"\n\n\n\n\n\n\n\n\n"+canonicalizedHeaders+"\n"+canonicalizedResource;
try {
String authorizationHeader = createAuthorizationHeader(stringToSign);
URL myUrl = new URL("https://klabs.blob.core.windows.net/" + urlPath);
HttpURLConnection connection=(HttpURLConnection)myUrl.openConnection();
connection.setRequestProperty("x-ms-blob-type", blobType);
connection.setRequestProperty("Content-Length", String.valueOf(blobLength));
connection.setRequestProperty("x-ms-date", date);
connection.setRequestProperty("x-ms-version", storageServiceVersion);
connection.setRequestProperty("Authorization", authorizationHeader);
connection.setDoOutput(true);
connection.setRequestMethod("POST");
System.out.println(String.valueOf(blobLength));
System.out.println(date);
System.out.println(storageServiceVersion);
System.out.println(stringToSign);
System.out.println(authorizationHeader);
System.out.println(connection.getDoOutput());
DataOutputStream outStream = new DataOutputStream(connection.getOutputStream());
// Send request
outStream.writeBytes("Dipanshu Verma wrote this");
outStream.flush();
outStream.close();
DataInputStream inStream = new DataInputStream(connection.getInputStream());
System.out.println("BULLA");
String buffer;
while((buffer = inStream.readLine()) != null) {
System.out.println(buffer);
}
// Close I/O streams
inStream.close();
outStream.close();
} catch (InvalidKeyException | Base64DecodingException | NoSuchAlgorithmException | IllegalStateException | UnsupportedEncodingException e) {
// TODO Auto-generated catch block
e.printStackTrace();
} catch (IOException e) {
// TODO Auto-generated catch block
e.printStackTrace();
}
}
}
I know only a proper code reviewer might be able to help me, please do it if you can.
Thanks
Question1: Why this error, did I miss any header/parameter?
Most likely you're getting this error is because of incorrect signature. Please refer to MSDN documentation for creating correct signature: http://msdn.microsoft.com/en-us/library/azure/dd179428.aspx. Unless your signature is correct you'll not be able to perform operations using REST API.
Question2: Do I need to add headers in the first place, because I am
able to hit the request from the browser without any issues.
In your current scenario, because you can access the blob directly (which in turn means the container in which the blob exist has Public or Blob ACL) you don't really need to use REST API. You can simply make a HTTP request using Java and read the response stream which will have blob contents. You would need to go down this route if the container ACL is Private because in this case your requests need to be authenticated and the code above creates an authenticated request.
Question3: Can it be an SSL issue? What is the concept of
certificates, and how and where to add them? Do I really need them?
Will I need them later, when I do bigger operations on my blob
storage(I want to manage a thousand blobs)?
No, it is not an SSL issue. Its an issue with incorrect signature.
Finally found the mistake!!
In the code above , I was using a String "password" as key for my SHA2
base64.decode(key)
It should have been the key associated with my account with AZURE.
Silly One!! Took me 2 weeks to find.

Categories