Currently we are working on a requirement, where we need to create sub directories in azure data bricks using java.
From Databricks UI, we are able to create subdirectory. Whether we can create subdirectory from Java code.
folders created from databricks UI
file inside created folder
Using curl command, we are able to create subdirectory from terminal. But, when we use ProcessBuilder in Java code it is not creating subdirectory.
Curl command for reference :
curl -n -X POST -H 'Authorization: Bearer ' -H 'Content-Type: application/json' -d '{ "path":"/Users/.com/testfolder4" }' https://*.azuredatabricks.net/api/2.0/workspace/mkdirs
Related
I'm trying to connect to one of my containers inside my cluster and execute multiple curl commands programmatically in Java.
To solve this I've written the following code:
String[] executeResetServerCachesBatFileCommand = {"cmd.exe", "/C", "start", "resetServerCaches.bat"};
ProcessBuilder cacheResetBuilder = new ProcessBuilder(executeResetServerCachesBatFileCommand);
cacheResetBuilder.directory(new File("C:/Program Files/PuTTY/"));
Process cacheReset = cacheResetBuilder.start();
The goal is to run a .bat file which contains multiple commands line by line.
The commands inside the .bat file are as follows:
kubectl exec --stdin --tty <podname> --namespace=<namespace> -- /bin/bash (connect to container)
curl -X DELETE -H "Authorization: Basic <auth> <url>
curl -X DELETE -H "Authorization: Basic <auth> <url>
curl -X DELETE -H "Authorization: Basic <auth> <url>
Don't wonder, I haven't written down the original values.
The bat file gets executed, so that is working. The problem is that after the first command in the .bat file (connect to container, which works) the rest of the commands are not executed or skipped.
I've put commands just as ipconfig or dir on the first line and then connect to the container to debug the problem. I found out that ipconfig and dir get executed just as the connecting part. So the problem is that the kubectl exec command take some seconds to finish and the rest is somehow skipped or bugged out.
I also tried working with "&" and "&&" to wait for the first command but that didn't work either.
I've went to many stackoverflow topics and googled but I couldn't make it.
Maybe someone can help me.
Best regards,
Julian
How do I write the following APIGEE related curl command in Java
curl -X POST -H "Content-Type: application/x-www-form-urlencoded" \
https://{org-name}-test.apigee.net/oauth/client_credential/accesstoken?
grant_type=client_credentials \
-d 'client_id={consumer-key}&client_secret={consumer-secret}'
Using curl is not allowed in our company so want to execute the command from Java. Any help would be appreciated.
Note - To some extent this URL was helpful How do i do the following curl command in Java
We are having java code that runs curl command to fetch the some result.
We have built a jar file and the jar file executes fine
Now, when we try to dokerize the java program (using jar) and run the application in docker we get this error:
errorjava.io.IOException: Cannot run program "curl": error=2, No such file or directory
at java.lang.ProcessBuilder.start(ProcessBuilder.java:1048)
at com.ps.api.common.CoreAPI_Spec.executeCoreAPI(CoreAPI_Spec.java:295)
at com.ps.api.common.CoreAPI_Spec.getAccessTokens(CoreAPI_Spec.java:319)
Dockerfile used :
FROM ubuntu:16.04
MAINTAINER niro;
# Install prerequisites
RUN apt-get update && apt-get install -y \
curl
FROM java:8-jdk-alpine
# Set the working directory to /app
WORKDIR /Users/******/Desktop/CoreAPI_Jar
# Copy the current directory contents into the container at /app
ADD *******_Automation-0.0.1-SNAPSHOT-jar-with-dependencies.jar ******_Automation-0.0.1-SNAPSHOT-jar-with-dependencies.jar
# Run app.py when the container launches
CMD ["java", "-jar", "******-0.0.1-SNAPSHOT-jar-with-dependencies.jar"]
The Java base image you are using is Alpine Linux one and curl package also needs to be downloaded from there. Here is Dockerfile I have used for Production deployments.
FROM openjdk:8-jre-alpine
RUN apk add --update \
curl \
&& rm -rf /var/cache/apk/*
Update 05/2019
As of Alpine Linux 3.3 there exists a new --no-cache option for apk. It allows users to install packages with an index that is updated and used on-the-fly and not cached locally:
FROM openjdk:8-jre-alpine
RUN apk --no-cache add curl
This avoids the need to use --update and remove /var/cache/apk/* when done installing packages.
Reference -
https://github.com/gliderlabs/docker-alpine/blob/master/docs/usage.md and Thank you #Daniel for the comment.
Your example dockerfile contains multiple FROM statements. This is valid but as the documentation says each FROM clears the state from previous instructions. And so the fresh installed curl is wiped after the second FROM.
Most languages have readily available HTTP clients these days; you should almost never be calling out to curl from a program in a language more sophisticated than a shell script. java.net.URLConnection has been a part of Java since Java 1.0 and (without knowing why you're trying to shell out for this) it's almost definitely the right tool here.
Assuming you control the executeCoreAPI method from your backtrace, you should change it to use the built-in Java HTTP client, and just delete all of the Dockerfile parts that try to install curl.
I am having trouble using the session-based speech recognition interface. Specifically, I am trying to split a longer audio stream into multiple chunks, upload them one at a time, and receive the complete parsed text at the end (as opposed to streaming the chunked audio from a single source).
IBM Watson's offers both stateless and stateful interfaces to speech recognition. The more common stateless protocol accepts a (chunked) audio stream and returns the parsed content on completion. The session-based approach allows the client to establish a persistent session, upload the audio as multiple chunks using multi-part, and query for the results, which can be very useful for processing long streams or processing microphone input.
I was able to find some tutorials and discussions but none of the examples seem to work (likely out of date, as the interface is evolving rapidly).
Here's a representative sample. The following POST will create a session:
curl -X POST -u "user:password" -H "Content-Type: application/json" \
https://stream.watsonplatform.net/speech-to-text/api/v1/sessions -verbose -d ""
Then, the next one should submit a portion of the audio data to recognize service, using the endpoints provided by the previous command:
curl -k -X POST -u "user:password" \
-H "content-type: audio/flac" --data-binary #temp.2.flac -H "Transfer-encoding: chunked" \
--cookie "SESSIONID=65097570295a0eccd15fd6dba326487416634371; Secure" \
https://stream.watsonplatform.net/speech-to-text/api/v1/sessions/65097570295a0eccd15fd6dba3264874/recognize -verbose
Finally, this command should return the results:
curl -k -X GET -u "user:password" \
--cookie "SESSIONID=65097570295a0eccd15fd6dba326487416634371; Secure" \
https://stream.watsonplatform.net/speech-to-text/api/v1/sessions/65097570295a0eccd15fd6dba3264874/observe_result -verbose
The first command completes without any issues, returning HTTP 201 Created status, as well as reasonably looking endpoints, which are used (together with the SESSIONID cookie) for subsequent calls.:
"recognize": "https://stream.watsonplatform.net/speech-to-text/api/v1/sessions/65097570295a0eccd15fd6dba3264874/recognize",
"recognizeWS": "wss://stream.watsonplatform.net/speech-to-text/api/v1/sessions/65097570295a0eccd15fd6dba3264874/recognize",
"observe_result": "https://stream.watsonplatform.net/speech-to-text/api/v1/sessions/65097570295a0eccd15fd6dba3264874/observe_result",
"session_id": "65097570295a0eccd15fd6dba3264874",
"new_session_uri": "https://stream.watsonplatform.net/speech-to-text/api/v1/sessions/65097570295a0eccd15fd6dba3264874"
However, both the 2nd and 3rd command fail with HTTP code of 404 "Session does not exist." error.
Any curl or Java pointers or examples would be greatly appreciated.
Robert,
I was just made aware of this post; sorry for the delay. I'm not sure how you're issuing the commands, but the issue may be that the session timed out before the subsequent calls. If the default 30-second session timeout expires before the subsequent calls, the service returns a 404 with the indicated message. It could also be an issue with how you're providing the cookie, as the previous user indicates. But I've experienced the session timeout issue, which could also be the culprit.
I wrote a Gist that uses curl commands to recognize a PCM file. In your case you just need to change the audio format and point to your file.
See https://gist.github.com/germanattanasio/ae26dc0144f229ad913a
When dealing with cookies it's always good to save them in a file and then use that file in the subsequent request.
For example
curl -X POST -u "user:password" -H "Content-Type: application/json" \
https://stream.watsonplatform.net/speech-to-text/api/v1/sessions \
-verbose -d ""
could be writen as:
curl -X POST -b cookies.txt -c cookies.txt -u $USERNAME:$PASSWORD \
"https://stream.watsonplatform.net/speech-to-text/api/v1/sessions" \
-d ""
The result will be the same and cookies.txt will have the SESSIONID.
Then you can use:
curl -X POST -b cookies.txt -c cookies.txt -u $USERNAME:$PASSWORD \
"https://stream.watsonplatform.net/speech-to-text/api/v1/sessions/$SESSION_ID/recognize?continuous=true" \
--header "Content-Type: audio/flac" --header "Transfer-Encoding: chunked" \
--data-binary #temp.2.flac
Make sure $SESSION_ID is updated with the value you get in the first curl command.
I have a curl syntax in .sh file. I need to run the curl sytnax or curl command in Java replicating the same syntax, but I am facing problem in replicating the same.
$AUTH_OPTION="--basic -u testuser:testpwd"
$HTTP_METHOD=POST
$FILE_OPTION="-d #$INPUT_FILE"
$CONTENT_TYPE="application/xml"
$ACCEPT_TYPE="application/xml"
echo curl -o response.txt -w %{http_code} -k -v $AUTH_OPTION -X $HTTP_METHOD $FILE_OPTION -H \"Content-Type: $CONTENT_TYPE\" -H \"Accept: $ACCEPT_TYPE\"
I have the corresponding Java code as:
StringBuffer curlCmd=new StringBuffer();
curlCmd.append("curl -o response.txt");
curlCmd.append(WHITE_SPACE);
curlCmd.append("-w %{http_code}");
curlCmd.append("-k -v -u testuser:testpwd");
curlCmd.append(WHITE_SPACE);
curlCmd.append("-X POST");
curlCmd.append(WHITE_SPACE);
curlCmd.append("-d #/test/xyz/xml" );
curlCmd.append(WHITE_SPACE);
curlCmd.append("-H"+"Content-type: application/xml");
curlCmd.append(WHITE_SPACE);
curlCmd.append("-H"+" Accept: application/xml");
curlCmd.append(WHITE_SPACE);
This does not seems to work: its not simulating the same behaviour of .sh curl syntax. Can any one help me to sort out this issue?
output
curl -o response.txt -w %{http_code} -k -v -u testuser:testpwd -X POST -d #/path/xyz.xml -H "Content-Type: application/xml" -H "Accept: application/xml"
the problem is xml is not getting accessed properly
I think there are a few possible problems, but one that catches my eye is that you are missing quotes around Content-Type: $CONTENT_TYPE and Accept: $ACCEPT_TYPE, for example:
"-H \"Content-type: application/xml\""
A second error is you have written -d #/test/xyz/xml but it should be:
-d #/test/xyz.xml
If it still doesn't work, can you post the output of both the sh script and your StringBuffer so we can more easily see where the differences are?
If you're using environment variables then you need to make sure that:
they are exported
you execute curl via a shell (e.g. /bin/bash)
The exporting means that the variables are exposed to child processes. The shell execution will expand these prior to calling your executable.
So your invocation will look like:
sh curl ....
It would help to see how you're invoking Process.exec(). One common gotcha is that you need to consume the stdout/stderr of the process concurrently, otherwise your sh/curl process may block waiting for your parent process to consume the output. See here for more details.
You should probably replace %{http_code} with something else on line 4 of the Java code. Environment variables will not be interpolated by Java.
Also, take a look at the Runtime#exec method. This lets you execute commands without having to worry about escaping quotes and such.
It's probably a good idea to make sure that your command runs without problems (such as the server not accepting the posted content) before trying to debug the invocation from java. It's far easier to deal with one problem at a time.