We Keep Coding

Generate gcs signed URL to upload large files

I have been trying to upload large files by generating a signed URL. Here is the documentation I was following to generate the signed URL: https://cloud.google.com/storage/docs/access-control/signing-urls-with-helpers#code-samples
It was working fine for files in 100's of MB's but as soon as the file size went up to 1 GB, the curl command started timing out even after increasing the expiration time. I tried looking at the answer here: https://stackoverflow.com/a/63789297/7466551, but I am still unable to get the URL working to upload the URL.
I am using this command to upload the file:
curl -X POST -H 'x-goog-resumable: start' --upload-file file-name 'pre_signed_google_url'. I am adding the 'x-goog-resumable: start' header as I am having "x-goog-resumable", "start" header as a part of my code to generate the URL.
Can someone please let me know if I need to do any additional thing to generate the URL to upload large files?

Answering my own question as I had to use two separate sources to reach a solution.
On top of the java code here: https://stackoverflow.com/a/63789297/7466551, I referred to the medium article here: https://medium.com/google-cloud/google-cloud-storage-signedurl-resumable-upload-with-curl-74f99e41f0a2
So you need the following additional lines of code on top of the StackOverflow answer to get a signed URL for resumable uploads:
// Open a HTTP connection and add header
URL obj = new URL(url.toString());
HttpURLConnection connection = (HttpURLConnection) obj.openConnection();
connection.setRequestMethod("POST");
connection.setRequestProperty("x-goog-resumable", "start");
connection.setDoOutput(true);
// Connect to the URL and post headers
DataOutputStream writer = new DataOutputStream(
connection.getOutputStream());
writer.writeBytes("");
writer.flush();
writer.close();
// Checking the responseCode to
if (connection.getResponseCode() == connection.HTTP_CREATED) {
connection.disconnect();
System.out.println("Location: " + connection.getHeaderField("Location"));
}
else {
// Throw error
BufferedReader reader = new BufferedReader(new InputStreamReader(connection.getInputStream()));
String inputLine;
StringBuffer response = new StringBuffer();
while ((inputLine = reader.readLine()) != null) {
response.append(inputLine);
}
reader.close();
String errorMessage = response.toString();
connection.disconnect();
throw new IOException(errorMessage);
}

How to fix dialogflow authentication issue

I have a dialogflow project that I'm trying to access from Java with a rest call.
It is giving me an authentication issue.
I have followed all online instructions (and many forum suggestions) to no avail.
I have tried generating the key json, as per the instructions here:
https://dialogflow.com/docs/reference/v2-auth-setup
and setting my environment variable as described, but nothing seems to work.
I have checked my projectID, and am running off the same machine with the environment variable, and have double, triple and quadruple checked it's name and location, but I still get the following error:
java.net.HttpRetryException: cannot retry due to server authentication, in streaming mode
Here is my code (though it's a REST call, so I don't know if it's so relevant):
String url = https://dialogflow.googleapis.com/v2/projects/MYPROJECT/agent/sessions/SESSION_NUM:detectIntent
URL url = new URL(full_url);
String inText = "Hello World";
String outText = "";
HttpURLConnection con = (HttpURLConnection) url.openConnection();
try {
con.setDoOutput(true);
con.setRequestMethod("POST");
// set body of http post
Map<String,String> arguments = new HashMap<>();
JSONObject inTextJsn = new JSONObject();
inTextJsn.append("text",inText);
inTextJsn.append("languageCode","en");
JSONObject fieldJsn = new JSONObject();
fieldJsn.append("text", inTextJsn);
arguments.put("queryInput", fieldJsn.toString());
StringJoiner sj = new StringJoiner("&");
for(Map.Entry<String,String> entry : arguments.entrySet())
sj.add(URLEncoder.encode(entry.getKey(), "UTF-8") + "="
+ URLEncoder.encode(entry.getValue(), "UTF-8"));
// post http post as bytes
byte[] bytes_out = sj.toString().getBytes(StandardCharsets.UTF_8);
con.setFixedLengthStreamingMode(bytes_out.length);
con.connect();
try (OutputStream os = con.getOutputStream()) {
os.write(bytes_out);
}
BufferedReader reader = new BufferedReader(new InputStreamReader(con.getInputStream(),
"UTF-8"));
// read all lines to a string
String line;
String response = "";
while ((line = reader.readLine()) != null) {
response += line;
}
JSONObject responseJsn = new JSONObject(response);
outText = responseJsn.get("fulfillmentText").toString();
} catch (Exception e) {
System.err.println(e);
} finally {
con.disconnect();
}
return restResponse;
The gist of the code is to simply send a message ("Hello World!") to my dialogflow, and get back my agent's response (the code may have bugs - it's a bit hard to test when I can't get passed this authentication issue, so please help with the authentication, not code bugs).
Thanks all!

The directions at that page assume you're going to use the gcloud program to generate a currently valid bearer token, which is then sent along with the HTTP headers. That page illustrates
Your code doesn't seem to be generating an Authorization HTTP header at all, which is why you're getting the error you do.
Since you're using Java, you should look at the google-auth-library-java library, which will give you the tools to generate the token you need to provide in the Authorization header.
You may also wish to check out the google-cloud-java library. This contains Java classes to directly perform operations against Dialogflow instead of coding the REST/HTTP calls yourself. (However, it is still at an Alpha level for Dialogflow, so may not be stable or forwards compatible.)

Ameritrade API POST Request JSON Response

I haven't coded in JAVA for years, and am trying to put an algorithm together to automatically make trades based on certain conditions.
I'm hoping to use the Ameritrade API
I've tried sending a cURL message in command prompt and I do indeed get a response back from the server 'Invalid Key'. I'd like to see the 'Invalid Key' response come back in Java as this will prove that I can send POST and receive JSON objects back into Java. From there I will work at authenticating but one step at a time!
Here's the curl message sent in command prompt that works, try it yourself by copying and pasting::
curl -X POST -H "Content-Type: application/x-www-form-urlencoded" -d "grant_type=authorization_code&refresh_token=&access_type=offline&code=&client_id=&redirect_uri=" " https://api.tdameritrade.com/v1/oauth2/token
The first thing I'd like to do is be able to send this curl message in JAVA and receive the JSON response back in JAVA
This is what I have for code so far, but I get a 500 error, which makes me think its something with the way im sending the message to the server?
public void trytoAuthenticate() {
HttpURLConnection connection = null;
//
//this is the curl message in command prompt you can send to receive JSON response back
//curl -X POST --header "Content-Type: application/x-www-form-urlencoded" -d
//"grant_type=authorization_code&
//refresh_token=&
//access_type=offline&
//code=&
//client_id=&
//redirect_uri=" "https://api.tdameritrade.com/v1/oauth2/token"
try {
//Create connection
URL url = new URL("https://api.tdameritrade.com/v1/oauth2/token");
String urlParameters = "grant_type=" + URLEncoder.encode("authorization_code", "UTF-8") +
"&refresh_token=" + URLEncoder.encode("", "UTF-8") +
"&access_type=" + URLEncoder.encode("", "UTF-8") +
"&code=" + URLEncoder.encode("", "UTF-8") +
"&client_id=" + URLEncoder.encode("", "UTF-8") +
"&redirect_uri=" + URLEncoder.encode("", "UTF-8");
connection = (HttpURLConnection) url.openConnection();
connection.setRequestMethod("POST"); //-X
connection.setRequestProperty("Content-Type", "application/x-www-form-urlencoded"); //-H
connection.setRequestProperty("Content-Length",
Integer.toString(urlParameters.getBytes().length));
connection.setRequestProperty("Content-Language", "en-US");
connection.setUseCaches(false);
connection.setDoOutput(true);//connection will be output
connection.setDoInput(true);//connection will be input
//Send request
DataOutputStream wr = new DataOutputStream (connection.getOutputStream());
wr.writeBytes(urlParameters);
System.out.println(urlParameters); //added for testing
wr.close();
//Get Response
DataInputStream is = new DataInputStream (connection.getInputStream());
BufferedReader rd = new BufferedReader(new InputStreamReader(is));
rd.readLine();
//StringBuffer response = new StringBuffer(); // or StringBuffer/StringBuilder if Java version 5+
//String line;
//while ((line = rd.readLine()) != null) {
// response.append(line);
// response.append('\r');
//}
rd.close();
//System.out.println(response.toString());
//return response.toString();
} catch (Exception e) {
e.printStackTrace();
//return null;
} finally {
if (connection != null) {
connection.disconnect();
}
}
}
}

A few things:
You need four parameters: grant_type, access_type, redirect_url and code.
You should URLDecode the authorization code you got from the browser login you probably just performed (as per their instructions)
Remove empty parameters, leave only what I mentioned above.
The redirect URL must match EXACTLY the redirect URL you added when you created your APP in the console.
If this is an app (looks like it), you probably have to set the access_type to "offline". Again see their documentation. Depends on your application.
grant_type should be "authorization_code", as that's what you want.

Curl Object to URL Generation

I have webservice that I am trying to call:
The following Curl command works for that
curl -F fa=c.apiupload -F sessiontoken=EA3237F922644115A0F7DB75D0AE388F -F destfolderid=52482BD488DB4AD6887C5C7BF47BD6FC -F filedata=#/Users/cpinera/tmp/panda2.jpg -F zip_extract=1 -F metadata=1 -F meta_img_description="This is a very nice panda" -F meta_img_keywords="panda,happy panda" http://domain.com/razuna/raz1/dam/index.cfm
but the equivalent URL that I generates as my HttpURLConnection for accessing RESTFul API does not :
http://domain.com/razuna/raz1/dam/index.cfm?fa=c.apiupload&sessiontoken=F46D2226463C4ADE866819AACD7D2F5E&filedata=C:\JobInterview\BatchUpload\auth.xml&destfolderid=52482BD488DB4AD6887C5C7BF47BD6FC&zip_extract=1
I get this response for the REST Request:
The content was :: <?xml version="1.0" encoding="UTF-8"?><Response><responsecode>1</responsecode><message>Upload failed This was not an uploaded form type</message></Response>
Here is the Java Code
Method that uses queries the URL:
public static String doQuery(String loginUrl) throws IOException{
URL url = new URL(loginUrl);
HttpURLConnection conn =
(HttpURLConnection) url.openConnection();
conn.setRequestMethod("POST");
if (conn.getResponseCode() != 200) {
throw new IOException(conn.getResponseMessage());
}
InputStream is = conn.getInputStream();
//Buffer the result into a string
BufferedReader rd = new BufferedReader(
new InputStreamReader(is));
StringBuilder sb = new StringBuilder();
String line;
while ((line = rd.readLine()) != null) {
line = StringEscapeUtils.unescapeHtml4(line);
sb.append(line);
}
rd.close();
conn.disconnect();
System.out.println("The content was :: " + sb.toString());
return sb.toString();
}
Method that should upload:
public static void testUpload(String seesionToken, String file ) throws IOException{
String upload = "http://domain.com:8080/razuna/raz1/dam/index.cfm?fa=c.apiupload&sessiontoken="+seesionToken+"&filedata="+file+"&destfolderid=52482BD488DB4AD6887C5C7BF47BD6FC&zip_extract=1" ;
System.out.println(upload);
Authenticate.doQuery(upload);
}
After adding in doQuery:
conn.setRequestProperty("Content-Type", "multipart/form-data");
And changing RESTFul URL to:
http://domain.com:8080/razuna/raz1/dam/index.cfm?fa=c.apiupload&sessiontoken="+seesionToken+"&filedata="+files.toURI()+"&destfolderid=52482BD488DB4AD6887C5C7BF47BD6FC&zip_extract=1"
It seems to detect that URL is trying to send a File but still cannot upload file:
The content was :: <?xml version="1.0" encoding="UTF-8"?><Response><responsecode>1</responsecode><message>Upload failed There was no appropriate FILE found in the upload</message></Response>

Your curl command is uploading the file #/Users/cpinera/tmp/panda2.jpg. Using the URL by itself will not, it will simply access the page, hence your error that 'this was not an uploaded form'.

When you run curl with -F parameters, it sends an HTTP POST request with the specified name-value pairs passed in the body of the HTTP request. When you access a URL using query string parameters, an HTTP GET is performed. It seems likely that the server handles those two request methods differently.
Update, now that you've posted some Java code:
The values in the query string portion of the URL are used to convey information to the web server that will be handling the request. Of course, this remote server is not going to know what C:/path/to/your/file is, because this is a path to a file on your local machine. The proper way to send this file data is to open the file in Java, and then read it and write it to the HttpUrlConnection's output stream.
For example, see the code snippet under "Posting Content" on the Android documentation's discussion of HttpUrlConnection: http://developer.android.com/reference/java/net/HttpURLConnection.html

send http request to linux server

I have to send an HTTP request to our C programme which is running on a Linux machine. How can I send an HTTP request in Java to our server which is in C and running on a Linux machine?

public void sendPostRequest() {
//Build parameter string
String data = "width=50&height=100";
try {
// Send the request
URL url = new URL("http://www.somesite.com");
URLConnection conn = url.openConnection();
conn.setDoOutput(true);
OutputStreamWriter writer = new OutputStreamWriter(conn.getOutputStream());
//write parameters
writer.write(data);
writer.flush();
// Get the response
StringBuffer answer = new StringBuffer();
BufferedReader reader = new BufferedReader(new InputStreamReader(conn.getInputStream()));
String line;
while ((line = reader.readLine()) != null) {
answer.append(line);
}
writer.close();
reader.close();
//Output the response
System.out.println(answer.toString());
} catch (MalformedURLException ex) {
ex.printStackTrace();
} catch (IOException ex) {
ex.printStackTrace();
}
}
The above example is for sending a POST request using a URL.

If you're asking how to send an HTTP request in Java to a web server written in C, you can use the URLConnection class.

try {
// Construct data
String data = URLEncoder.encode("key1", "UTF-8") + "=" + URLEncoder.encode("value1", "UTF-8");
data += "&" + URLEncoder.encode("key2", "UTF-8") + "=" + URLEncoder.encode("value2", "UTF-8");
// Send data
URL url = new URL("http://hostname:80/cgi");
URLConnection conn = url.openConnection();
conn.setDoOutput(true);
OutputStreamWriter wr = new OutputStreamWriter(conn.getOutputStream());
wr.write(data);
wr.flush();
// Get the response
BufferedReader rd = new BufferedReader(new InputStreamReader(conn.getInputStream()));
String line;
while ((line = rd.readLine()) != null) {
// Process line...
}
wr.close();
rd.close(); } catch (Exception e) { }
The above example is for sending a POST request using a URL.
Also take a look at Sun Tutorial on reading/Writing from/to a URLConnection. The other option is to use Apache HTTPComponents which has examples for the HttpCore and HttpClient module.
If you are looking into implementing the web Server, you will have to handle the Http request yourselves which involves a thread pool, parsing the requests, generating HTML, security, multiple sessions, etc or follow the easy route by using off-the-shelf web server like Apache and seeing which all high-level languages like Perl, Ruby can be used for developing the web application.
For implementing your own Http server, please take a look at Micro-Httpd or tinyHttpd
You may also want to look at Adding Web Interface -C++ application which has a sample code.

From the way your question is worded.. I think you need to know some basic stuff before you can start. Try try googling for a simple guide to how web servers work.
Once you have the basic idea, there are a couple of options for a C programmer:
1) You want your C program to be running continuously, waiting for a request from your Java.
In this case, you will have to code your C program to open a Socket and Listen for connections. See http://www.linuxhowtos.org/C_C++/socket.htm for example.
OR
2) You have a web Server on your server which will run your C program each time a particular request is made? In this case, you will have to code your C as a CGI program. See http://www.cs.tut.fi/~jkorpela/forms/cgic.html for example.
Hint: (2) is much easier!