Curl Object to URL Generation - java

I have webservice that I am trying to call:
The following Curl command works for that
curl -F fa=c.apiupload -F sessiontoken=EA3237F922644115A0F7DB75D0AE388F -F destfolderid=52482BD488DB4AD6887C5C7BF47BD6FC -F filedata=#/Users/cpinera/tmp/panda2.jpg -F zip_extract=1 -F metadata=1 -F meta_img_description="This is a very nice panda" -F meta_img_keywords="panda,happy panda" http://domain.com/razuna/raz1/dam/index.cfm
but the equivalent URL that I generates as my HttpURLConnection for accessing RESTFul API does not :
http://domain.com/razuna/raz1/dam/index.cfm?fa=c.apiupload&sessiontoken=F46D2226463C4ADE866819AACD7D2F5E&filedata=C:\JobInterview\BatchUpload\auth.xml&destfolderid=52482BD488DB4AD6887C5C7BF47BD6FC&zip_extract=1
I get this response for the REST Request:
The content was :: <?xml version="1.0" encoding="UTF-8"?><Response><responsecode>1</responsecode><message>Upload failed This was not an uploaded form type</message></Response>
Here is the Java Code
Method that uses queries the URL:
public static String doQuery(String loginUrl) throws IOException{
URL url = new URL(loginUrl);
HttpURLConnection conn =
(HttpURLConnection) url.openConnection();
conn.setRequestMethod("POST");
if (conn.getResponseCode() != 200) {
throw new IOException(conn.getResponseMessage());
}
InputStream is = conn.getInputStream();
//Buffer the result into a string
BufferedReader rd = new BufferedReader(
new InputStreamReader(is));
StringBuilder sb = new StringBuilder();
String line;
while ((line = rd.readLine()) != null) {
line = StringEscapeUtils.unescapeHtml4(line);
sb.append(line);
}
rd.close();
conn.disconnect();
System.out.println("The content was :: " + sb.toString());
return sb.toString();
}
Method that should upload:
public static void testUpload(String seesionToken, String file ) throws IOException{
String upload = "http://domain.com:8080/razuna/raz1/dam/index.cfm?fa=c.apiupload&sessiontoken="+seesionToken+"&filedata="+file+"&destfolderid=52482BD488DB4AD6887C5C7BF47BD6FC&zip_extract=1" ;
System.out.println(upload);
Authenticate.doQuery(upload);
}
After adding in doQuery:
conn.setRequestProperty("Content-Type", "multipart/form-data");
And changing RESTFul URL to:
http://domain.com:8080/razuna/raz1/dam/index.cfm?fa=c.apiupload&sessiontoken="+seesionToken+"&filedata="+files.toURI()+"&destfolderid=52482BD488DB4AD6887C5C7BF47BD6FC&zip_extract=1"
It seems to detect that URL is trying to send a File but still cannot upload file:
The content was :: <?xml version="1.0" encoding="UTF-8"?><Response><responsecode>1</responsecode><message>Upload failed There was no appropriate FILE found in the upload</message></Response>


Your curl command is uploading the file #/Users/cpinera/tmp/panda2.jpg. Using the URL by itself will not, it will simply access the page, hence your error that 'this was not an uploaded form'.


When you run curl with -F parameters, it sends an HTTP POST request with the specified name-value pairs passed in the body of the HTTP request. When you access a URL using query string parameters, an HTTP GET is performed. It seems likely that the server handles those two request methods differently.
Update, now that you've posted some Java code:
The values in the query string portion of the URL are used to convey information to the web server that will be handling the request. Of course, this remote server is not going to know what C:/path/to/your/file is, because this is a path to a file on your local machine. The proper way to send this file data is to open the file in Java, and then read it and write it to the HttpUrlConnection's output stream.
For example, see the code snippet under "Posting Content" on the Android documentation's discussion of HttpUrlConnection: http://developer.android.com/reference/java/net/HttpURLConnection.html

Related

Generate gcs signed URL to upload large files

I have been trying to upload large files by generating a signed URL. Here is the documentation I was following to generate the signed URL: https://cloud.google.com/storage/docs/access-control/signing-urls-with-helpers#code-samples
It was working fine for files in 100's of MB's but as soon as the file size went up to 1 GB, the curl command started timing out even after increasing the expiration time. I tried looking at the answer here: https://stackoverflow.com/a/63789297/7466551, but I am still unable to get the URL working to upload the URL.
I am using this command to upload the file:
curl -X POST -H 'x-goog-resumable: start' --upload-file file-name 'pre_signed_google_url'. I am adding the 'x-goog-resumable: start' header as I am having "x-goog-resumable", "start" header as a part of my code to generate the URL.
Can someone please let me know if I need to do any additional thing to generate the URL to upload large files?

Answering my own question as I had to use two separate sources to reach a solution.
On top of the java code here: https://stackoverflow.com/a/63789297/7466551, I referred to the medium article here: https://medium.com/google-cloud/google-cloud-storage-signedurl-resumable-upload-with-curl-74f99e41f0a2
So you need the following additional lines of code on top of the StackOverflow answer to get a signed URL for resumable uploads:
// Open a HTTP connection and add header
URL obj = new URL(url.toString());
HttpURLConnection connection = (HttpURLConnection) obj.openConnection();
connection.setRequestMethod("POST");
connection.setRequestProperty("x-goog-resumable", "start");
connection.setDoOutput(true);
// Connect to the URL and post headers
DataOutputStream writer = new DataOutputStream(
connection.getOutputStream());
writer.writeBytes("");
writer.flush();
writer.close();
// Checking the responseCode to
if (connection.getResponseCode() == connection.HTTP_CREATED) {
connection.disconnect();
System.out.println("Location: " + connection.getHeaderField("Location"));
}
else {
// Throw error
BufferedReader reader = new BufferedReader(new InputStreamReader(connection.getInputStream()));
String inputLine;
StringBuffer response = new StringBuffer();
while ((inputLine = reader.readLine()) != null) {
response.append(inputLine);
}
reader.close();
String errorMessage = response.toString();
connection.disconnect();
throw new IOException(errorMessage);
}

POST file (as binary stream) using java.net.HttpURLConnection as a param file=<file binary stream>

I am trying to upload (POST) a file to an endpoint using java.net.HttpURLConnection but I keep getting http code 400 (bad request).
I refered to Send File And Parameters To Server With HttpURLConnection in android API 23
but problem is that I need to send this file as request body param (file=).
Note: The files will be of small size only (4-5mb) so I am reading it entirely in memory.
Corresponding curl request is:
curl -X POST "API" -H "Content-Type: multipart/form-data" -F "file="
Excerpts of Code that I am using:
Proxy webproxy = new Proxy(Proxy.Type.HTTP, new InetSocketAddress("
<proxy host>", <proxy_port>));
HttpURLConnection http_conn = (HttpURLConnection)
url.openConnection(webproxy);
String authorization = getAuthorization(access_token);
http_conn.setRequestMethod("POST");
http_conn.setRequestProperty("Accept-Charset", "UTF-8");
http_conn.setRequestProperty("Authorization", authorization);
http_conn.setRequestProperty("Connection", "Keep-Alive");
http_conn.setRequestProperty("Content-Type", "multipart/form-data);
http_conn.setDoOutput(true);
http_conn.setDoInput(true);
DataOutputStream outputStream;
outputStream = new DataOutputStream(http_conn.getOutputStream());
File file_obj = new File(this.file);
byte[] allBytes = new byte[(int) file_obj.length()];
FileInputStream fileInputStream = new FileInputStream(file_obj);
outputStream.write("file=".getBytes("UTF-8")); <---Trying to add file param here
fileInputStream.read(allBytes);
outputStream.write(allBytes);
Post that I just read response using below piece of code (works fine for different GET requests):
InputStream inputStream = http_conn.getInputStream();
BufferedReader bufferedReader = new BufferedReader(new
InputStreamReader(inputStream));
String line = "";
while ((line = bufferedReader.readLine()) != null) {
data = data + line;
}
Note: I use java rarely an am not very familiar with it so please be descriptive in your response.

When looking at your curl command line, it shows that the file needs to be send as a multipart/form-data request. This is actually a complex way of formatting your data when it is requires.
An example of the format you need to send is:
Headers:
Content-Type: multipart/form-data; boundary=AaB03x
Body:
--AaB03x
Content-Disposition: form-data; name="files"; filename="file1.txt"
Content-Type: text/plain
... contents of file1.txt ...
--AaB03x--
At the moment, your code is sending the file as a POST/GET formatted request, and this doesn't work as the backend isn't expecting that.
To solve this problem, we need to format the source files into the format required by the backend, and once you know that the "boundary" header option is just a randomly generated value, it becomes more easy to send the request.
String boundary = "MY_AWESOME_BOUNDARY"
http_conn.setRequestProperty("Content-Type", "multipart/form-data; boundary=" + boundary);
try(DataOutputStream outputStream = new DataOutputStream(http_conn.getOutputStream())) {
File file_obj = new File(this.file);
// Write form-data header
outputStream.write(("--" + boundary + "\r\n").getBytes("UTF-8"));
outputStream.write(("Content-Disposition: form-data; name=\"file\"; filename=\"file1.txt\"\r\n").getBytes("UTF-8"));
outputStream.write(("Content-Type: text/plain\r\n").getBytes("UTF-8"));
outputStream.write(("\r\n").getBytes("UTF-8"));
// Write form-data body
Files.copy(file_obj.toPath(), outputStream)
// Write form-data "end"
outputStream.write(("--" + boundary + "--\r\n").getBytes("UTF-8"));
}
// Read backend response here
try(InputStream inputStream = http_conn.getInputStream()) {
BufferedReader bufferedReader = new BufferedReader(new
InputStreamReader(inputStream));
StringBuilder lines = new StringBuilder(); // StringBuilder is faster for concatination than appending strings
while ((line = bufferedReader.readLine()) != null) {
lines.append(line);
}
System.out.println(lines);
}
Note that I used "try-with-resource" blocks, these blocks make sure that any external resources are closed and disposed when you are done using them, generally the open resource limit of the OS is very low, compared to the amount of memory your program has, so what happens is that your program could give weird errors that only happens after some time of running or when the user executes certain actions inside your application

The above didnt worked for me so I switched to different package (okhttp3), here is what worked for me:
File file_obj = new File(this.file);
String authorization = "my authorization string";
Proxy webproxy = new Proxy(Proxy.Type.HTTP, new
InetSocketAddress("proxy", <port>));
OkHttpClient client = new OkHttpClient.Builder().proxy(webproxy).build();
RequestBody requestBody = new MultipartBody.Builder().setType(MultipartBody.FORM).addFormDataPart("file", "filename",
RequestBody.create(MediaType.parse("application/octet-stream"), file_obj)).build();
Request request = new Request.Builder().header("Authorization", authorization).url(this.url).post(requestBody).build();
try (Response response = client.newCall(request).execute()){
if(!response.isSuccessful()) return "NA";
return (response.body().string());
}

replicate wget command in java

i have this command:
wget -O prova.csv --header="prova-user: guest" --header="prova-passwd: guest"
"http://www.....................80&albedo=0.2&horizon=1"
i want to do a batch scheduled in Java but I can not connect. When I try to take the imputstream return me this error:
ERROR message -8: Unregistered IP address
This is my piece of code:
URL myURL = new URL(url);
HttpURLConnection myURLConnection = (HttpURLConnection)myURL.openConnection();
String userCredentials = "guest:guest";
String basicAuth = "Basic " + new String(new Base64().encode(userCredentials.getBytes()));
myURLConnection.setRequestProperty ("Authorization", basicAuth);
myURLConnection.setRequestMethod("POST");
myURLConnection.setRequestProperty("Content-Language", "en-US");
myURLConnection.setUseCaches(false);
myURLConnection.setDoInput(true);
myURLConnection.setDoOutput(true);
// Show page.
BufferedReader reader = null;
try {
reader = new BufferedReader(new InputStreamReader(myURLConnection.getInputStream(), "UTF-8"));
for (String line; ((line = reader.readLine()) != null);) {
System.out.println(line);
}
} finally {
if (reader != null) try { reader.close(); } catch (IOException ignore) {}
}
is it possible? and how can I do it?
Thanks in advance

You had provided 2 completely different commands.
The first is a wget that send in HTTP headers a sort of authentication infos, and GET a result.
The second is a java program that perform an HTTP request in POST with basic authentication.
If the first command is working, than you should forget about the basic authentication and set the proper HTTP headers as you did in the wget command.
I don't know why you try a POST, if the wget looks as a normal GET request.
Just use a GET request in java too.
And it should work.
About the error, I suppose is the server that sent you such error message.
So it could be as you haven't correctly authenticated.
But it is a strange error, I'm expecting such kind of error if the server have a white list of IP addresses allowed to connect.
Are you running the wget and the java code on the same server?

google URL change

When I call this in java :
URL url = new URL("http://www.google.com/finance/getprices?q=MSFT");
URLConnection goog = url.openConnection();
BufferedReader in = new BufferedReader(new InputStreamReader(goog.getInputStream()));
I get this as exception :
java.io.IOException: Server returned HTTP response code: 503 for URL: http://www.google.com/sorry/?continue=http://www.google.com/finance/getprices%3Fq%3MSFTO%26
I don't have converted URL in my function because its generated automatically when my URL is called, My original URL is string after "continue=", how can I get it back from this URL?
EDIT :
Because I am calling this page again and again it generates this URL http://www.google.com/sorry/?continue=http://www.google.com/finance/getprices%3Fq%3MSFTO%26 and it says :
Our systems have detected unusual traffic from your computer network. This page checks to see if it's really you sending the requests, and not a robot.
If I copy paste URL after continue= it gives me actual content of page.

503 is a response code which tells service is down or unavailabe
wiki link
According to wikipedia
503 Service Unavailable
The server is currently unavailable (because it is overloaded or down for maintenance).[2] Generally, this is a temporary state.
I have tried your code and this is woriking fine (giving no exception)
URL url = new URL("http://www.google.com/finance/getprices?q=MSFT");
URLConnection goog = url.openConnection();
BufferedReader in = new BufferedReader(new InputStreamReader(goog.getInputStream()));
String temp;
while((temp = in.readLine())!= null){
System.out.println(temp);
}
}catch(Exception ex){
ex.printStackTrace();
}
and its return me
EXCHANGE%3DNASDAQ
MARKET_OPEN_MINUTE=570
MARKET_CLOSE_MINUTE=960
INTERVAL=86400
COLUMNS=DATE,CLOSE,HIGH,LOW,OPEN,VOLUME
DATA=
TIMEZONE_OFFSET=-240
a1370289600,35.59,35.63,34.83,34.92,51256272
1,34.99,35.74,34.771,35.62,65538438
2,34.78,34.89,34.43,34.6,46032657
3,34.96,35.11,34.49,34.84,37627133
4,35.67,35.78,35.06,35.25,40762249
7,35.47,35.65,35.14,35.51,35995223
8,34.84,35.18,34.68,35.05,39350316
9,35,35.27,34.85,35.14,37373032
10,34.715,35.02,34.59,34.99,45654803
11,34.4,34.6901,34.25,34.55,53116371
14,35,35.16,34.63,34.69,49672492
15,34.98,35.17,34.895,34.97,28622929
16,34.59,35.09,34.59,34.96,30820208
17,33.49,34.33,33.37,34.26,54496758
18,33.265,33.73,33.05,33.66,85338395
21,33.715,34.2,32.57,32.94,56113708
22,33.67,34.38,33.46,34.08,44073348
23,34.35,34.48,33.8875,34.12,48667834
24,34.62,34.78,34.5,34.52,28993542
25,34.545,34.79,34.34,34.38,65548196
28,34.36,34.99,34.33,34.75,31064000
So probably problem is that service is not available temporarily in your area.

Send a POST to a HTTPS - Logic Issue

Anyone knows why i get this error. I am trying to send a POST request, and here's the error message that i get.
Server response:
Error while dispatching hrxml [ Server was unable to process request. --> Procedure or function 'sp__LogMessage' expects parameter '#pi_ClientID', which was not supplied. at System.Web.Services.Protocols.SoapHttpClientProtocol.ReadResponse(SoapClientMessage message, WebResponse response, Stream responseStream, Boolean asyncCall)
at System.Web.Services.Protocols.SoapHttpClientProtocol.Invoke(String methodName, Object[] parameters)
at DispatchService.HRISMessageRouter.MessageRouter.Route(String HRXML)
at DispatchService.DispatchMessage.Dispatch(String HRXML)]
My Code :
URL link = new URL("https://example.com/example.asp");
HttpsURLConnection com = (HttpsURLConnection) link.openConnection();
String l;
con.setRequestMethod("POST");
con.setDoInput(true);
con.setDoOutput(true);
con.setRequestProperty("name", "rrrrr");
con.setRequestProperty("pwd", "ffff");
OutputStream os = con.getOutputStream();
os.flush();
InputStream is = con.getInputStream();
BufferedReader rd = new BufferedReader(new InputStreamReader(is));
StringBuffer r = new StringBuffer();
while((l = rd.readLine()) != null) {
r.append(l);
r.append('\r');
}
rd.close();
System.out.println("out "+ r.toString());
I have tried debugging the code etc, but still unable to find a possible reason to why this hapend . Can anyone help me to figure out the reason and a possible solution for this issue ?

The error message is:
Server was unable to process request. --> Procedure or function 'sp__LogMessage' expects parameter '#pi_ClientID', which was not supplied.
This looks like a SQL Stored Procedure on the server. Check to make sure that a client ID is being supplied to it.

Judging from the stacktrace:
Your application is talking to a SOAP service
The SOAP service is expecting an (XML encoded) request which includes a parameter that doesn't exist.
But this doesn't mesh with what you client code is doing. In fact, you are sending a POST request with parameters, and that will turn into a request body which is probably encoded as application/x-www-form-urlencoded ... not XML. That ain't going to work.

Categories