I am trying to run spark/java application on kubernetese (via minikube) using spark-operator. I am getting a bit confused on what should I place in the Dockerfile so that it could be built in the image format and execute via spark-operator ?
Sample spark-operator.yaml :
apiVersion: sparkoperator.k8s.io/v1beta2
kind: SparkApplication
metadata:
name: my-spark-app
namespace: default
spec:
type: Java
mode: cluster
image: docker/repo/my-spark-app-image
mainApplicationFile: local:///opt/app/my-spark-app.jar
As mentioned above, the spark operator yaml only requires the jar and the image location. So, do I need to mention just below in my Dockerfile ? Is there any sample Dockerfile available which I can refer ?
Dockerfile:
FROM openjdk11:alpine-jre
COPY target/*.jar /opt/app/csp_auxdb_refresh.jar
COPY src/main/resources/* opt/app
In the Dockerfile you have provided, nor Spark, nor other dependencies are installed.
To quickly get started, use gcr.io/spark-operator/spark:v3.1.1 as the base for your image, i.e. change the FROM statement to FROM gcr.io/spark-operator/spark:v3.1.1 and build again.
There is a great guide on how to get started with the spark-operator in their Github repo (here).
Related
I've got a Spring Boot application that'd I'd like to automatically generate traces for using the OpenTelemetry Java agent, and subsequently upload those traces to Google Cloud Trace.
I've added the following code to the entry point of my application for sending traces:
OpenTelemetrySdk.builder()
.setTracerProvider(
SdkTracerProvider.builder()
.addSpanProcessor(
SimpleSpanProcessor.create(TraceExporter.createWithDefaultConfiguration())
)
.build()
)
.buildAndRegisterGlobal();
...and I'm running my application with the following system properties:
-javaagent:path/to/opentelemetry-javaagent-all.jar \
-jar myapp.jar
...but I don't know how to connect the two.
Is there some agent configuration I can apply? Something like:
-Dotel.traces.exporter=google_cloud_trace
I ended up resolving this as follows:
Clone the GoogleCloudPlatform /
opentelemetry-operations-java repo
git clone
git#github.com:GoogleCloudPlatform/opentelemetry-operations-java.git
Build the exporter-auto project
./gradlew clean :exporter-auto:shadowJar
Copy the jar produced in exporter-auto/build/libs to my target project
Run the application with the following arguments:
-javaagent:path/to/opentelemetry-javaagent-all.jar
-Dotel.javaagent.experimental.extensions=[artifact-from-step-3].jar
-Dotel.traces.exporter=google_cloud_trace
-Dotel.metrics.exporter=none
-jar myapp.jar
Note: This setup does not require any explicit code changes in the target code base.
I have read some answers about similar topics but I was not satisfied with any of them.
We are deploying some code into AWS Lambda with a jar file which contains a version of the code like name-of-my-app-14.jar, the 14 is the Jenkins build number.
The problem I have is that I don't have a way to understand which version of the jar is currently deployed in AWS and it would be a nice to have.
This is the cloudformation fragment I have to create the lambda:
MyLambdaFunction:
Type: AWS::Serverless::Function
Properties:
CodeUri: name-of-my-app.jar
FunctionName: "my-function-name"
Handler: "com.package.something.myapp.HandlerClass::handleRequest"
MemorySize: 256
Role: "arn:aws:iam::1234567890:role/some-role"
Runtime: "java8"
Timeout: 60
Environment:
Variables:
SOME_VARIABLE: "value"
To deploy we download the jar with the version we want to deploy from our artifact repository, saving it as it is specified in the above template and we run:
aws cloudformation package --template-file myapp-stack.yaml --output-template-file tmp.yaml --s3-bucket my.bucket
aws cloudformation deploy --region my-region --template-file tmp.yaml --stack-name prod-myappstackname --capabilities CAPABILITY_IAM --parameter-overrides Environment=prod --no-fail-on-empty-changeset
I am trying to build a native image of a micronaut (v1.0.4) application.
This application uses Consul as service discovery.
I've created the app using --features option:
$ mn create-app my-app --features discovery-consul --features graal-native-image --build maven
The application works perfectly on my local machine, but when I try to build a docker container with the native image I get an error:
$ ./docker-build.sh
error: No instances are allowed in the image heap for a class that is initialized or reinitialized at image runtime:
sun.security.provider.NativePRNG
Detailed message:
Error: No instances are allowed in the image heap for a class that is initialized or reinitialized at image runtime: sun.security.provider.NativePRNG
Trace: object java.security.SecureRandom
method com.sun.jndi.dns.DnsClient.query(DnsName, int, int, boolean, boolean)
Call path from entry point to com.sun.jndi.dns.DnsClient.query(DnsName, int, int, boolean, boolean):
at com.sun.jndi.dns.DnsClient.query(DnsClient.java:178)
at com.sun.jndi.dns.Resolver.query(Resolver.java:81)
at com.sun.jndi.dns.DnsContext.c_getAttributes(DnsContext.java:434)
at com.sun.jndi.toolkit.ctx.ComponentDirContext.p_getAttributes(ComponentDirContext.java:235)
at com.sun.jndi.toolkit.ctx.PartialCompositeDirContext.getAttributes(PartialCompositeDirContext.java:141)
at com.sun.jndi.toolkit.ctx.PartialCompositeDirContext.getAttributes(PartialCompositeDirContext.java:129)
at javax.naming.directory.InitialDirContext.getAttributes(InitialDirContext.java:142)
at io.micronaut.discovery.client.DnsResolver.getCNamesFromTxtRecord(DnsResolver.java:59)
at io.micronaut.discovery.client.EndpointUtil.getEC2DiscoveryUrlsFromZone(EndpointUtil.java:197)
at io.micronaut.discovery.client.EndpointUtil.getServiceUrlsFromDNS(EndpointUtil.java:141)
If I remove Consul integration, it works without any problem.
I could not find anything useful on the official documentation:
Microservices as GraalVM native images
Consul Support
Does anyone know where the problem is?
After going over several issues and posts, I ended up finding the answer.
To remove this failure, just add this class com.sun.jndi.dns.DnsClient to the list of classes under the option --delay-class-initialization-to-runtime when you create the native image in you Dockerfile:
Dockerfile
RUN native-image --no-server \
...
--delay-class-initialization-to-runtime=...,com.sun.jndi.dns.DnsClient \
-H:-UseServiceLoaderFeature \
--allow-incomplete-classpath \
-H:Name=model-quotes \
-H:Class=model.quotes.Application
...
After doing that, everything works ok and the docker image is generated successfully.
It should be a good idea to add this class in Dockerfile generated by default. It is a bit annoying to generate a new project using Micronaut CLI and find that native images does not work without changing anything.
I'm trying to do continuous integration using VSTS for Java rest api.
I've added following lines for building Docker image:
FROM openjdk:8
ADD $(build.artifactstagingdirectory)/docker-spring.jar docker-spring.jar
EXPOSE 8085
ENTRYPOINT ["java","-jar","docker-spring.jar"]
I'm getting following error:
ADD failed: stat /var/lib/docker/tmp/docker-builder946930284/SpringRest/target/docker-spring.jar: no such file or directory
/usr/local/bin/docker failed with return code: 1
I could see lot few references, still not able to correlate the issue exactly.
This happens when you haven't supplied the path right. Make sure that your variable that has the path has the . for current directory, or / and the full path to the resources:
$(build.artifactstagingdirectory) = ./correct/relative/path/docker-spring.jar
or
$(build.artifactstagingdirectory) = /correct/full/path/path/docker-spring.jar<
Looking at your output I'd assume you're using "path/to/docker-spring.jar" which doesn't work.
EDIT: In your particular case you want to change the following line in the docker file:
What you have:
COPY /SpringRest/target/docker-spring.jar /usr/local/app/docker-spring.jar
What it should be:
COPY ./target/docker-spring.jar /usr/local/app/docker-spring.jar
Notice the little dot. This way it should work.
I want to test my lambda functions locally with Serverless Application Model (SAM)
In the AWS docs they write :
SAM Local leverages the docker-lambda Docker images to run your code in a sandbox that simulates the Lambda execution environment.
I pulled the docker image on my computer. I could successfully run a simple Hello World Lambda Function.
Command to run Lambda function locally:
$ docker run -v "$PWD/target/classes":/var/task lambci/lambda:java8 com.amazonaws.lambda.demo.LambdaFunctionHandler
results:
"Hello from Lambda!"
Code of Lambda function automatically generated with Eclipse Toolkit:
package com.amazonaws.lambda.demo;
import com.amazonaws.services.lambda.runtime.Context;
import com.amazonaws.services.lambda.runtime.RequestHandler;
public class LambdaFunctionHandler implements RequestHandler<Object, String> {
#Override
public String handleRequest(Object input, Context context) {
context.getLogger().log("Input: " + input);
// TODO: implement your handler
return "Hello from Lambda!"
}
}
This is my progress till yet. What i couldnt do is to use sam local which uses the docker-lambda image.(Maybe i should not have to download it manually?).
I installed sam local on my windows:
npm install -g aws-sam-local
created a template.yaml config sam file.
AWSTemplateFormatVersion: 2010-09-09
Transform: AWS::Serverless-2016-10-31
Resources:
ExampleJavaFunction:
Type: AWS::Serverless::Function
Properties:
Handler: com.amazonaws.lambda.demo.LambdaFunctionHandler
CodeUri: ./target/demo-1.0.0-shaded.jar
Runtime: java8
the name for CodeUri: i choosed after i build my shaded jar file with:
mvn compile shade:shade
after this i should run to run my lambda function:
$ echo '{ "some": "input" }' | sam local invoke
NOW i have this ERROR:
2017/12/05 14:56:36 Successfully parsed template.yaml
2017/12/05 14:56:36 Running AWS SAM projects locally requires Docker. Have you got it installed?
2017/12/05 14:56:36 error during connect: Get http://%2F%2F.%2Fpipe%2Fdocker_engine/_ping: open //./pipe/docker_engine: The system cannot find the file specified. In the default daemon configuration on Windows, the docker client must be run elevated to connect. This error may also indicate that the docker daemon is not running.
What is my mistake to use SAM Local with Java? Can it be that its not working because my computer has not Hyper-V and iam using dockertoolbox?..
here you can see Advanced sam docs with compiled languages like java.
It was a bug in Sam local.. fixed with new
update
If you still have a problem in windows then try this :
COMPOSE_CONVERT_WINDOWS_PATHS=1
this should help if your Path is wrong. / \