How to add Custom Processor to StreamSets - java

I have a StreamSets container in docker compose and jar file, which are created according with tutorial - https://github.com/streamsets/tutorials/blob/master/tutorial-processor/readme.md
https://github.com/streamsets/tutorials/tree/master/tutorial-processor/sampleprocessor
The source was added correctly.
Extract the tarball to the SDC user-libs directory, restart SDC, and
you should see the sample stages in the stage library:
To add Custom Processor to pipeline I copied jar file to user-labs And restart docker
docker exec -it docker-hadoop_streamsets_1 bash
sudo cp -a /data/lib/. /opt/streamsets-datacollector-3.9.0/user-libs
docker restart docker-hadoop_streamsets_1
But after restarting I don't see new destination. Also I tried to add with web interface and read this article - https://streamsets.com/documentation/datacollector/latest/help/datacollector/UserGuide/Installation/AddtionalStageLibs.html#concept_evs_xkm_s5
bash-4.4$ ls -R /opt/streamsets-datacollector-3.9.0/user-libs/
/opt/streamsets-datacollector-3.9.0/user-libs/:
com.example-1.0-SNAPSHOT com.example-1.0-SNAPSHOT.jar com.example-1.0-SNAPSHOT~ sampleprocessor sampleprocessor-1.0-SNAPSHOT sampleprocessor-1.0-SNAPSHOT.jar
/opt/streamsets-datacollector-3.9.0/user-libs/com.example-1.0-SNAPSHOT~:
CredentialStores.json Interceptors.json PipelineStages.json com default.png
Delegates.json LineagePublishers.json Services.json data-collector-library-bundle.properties
ElDefinitions.json META-INF StageDefList.json datacollector-resource-bundles.json
/opt/streamsets-datacollector-3.9.0/user-libs/com.example-1.0-SNAPSHOT~/META-INF:
MANIFEST.MF maven
/opt/streamsets-datacollector-3.9.0/user-libs/com.example-1.0-SNAPSHOT~/META-INF/maven:
com.example
/opt/streamsets-datacollector-3.9.0/user-libs/com.example-1.0-SNAPSHOT~/META-INF/maven/com.example:
com.example
/opt/streamsets-datacollector-3.9.0/user-libs/com.example-1.0-SNAPSHOT~/META-INF/maven/com.example/com.example:
pom.properties pom.xml
/opt/streamsets-datacollector-3.9.0/user-libs/com.example-1.0-SNAPSHOT~/com:
example
/opt/streamsets-datacollector-3.9.0/user-libs/com.example-1.0-SNAPSHOT~/com/example:
stage
/opt/streamsets-datacollector-3.9.0/user-libs/com.example-1.0-SNAPSHOT~/com/example/stage:
destination lib origin processor
/opt/streamsets-datacollector-3.9.0/user-libs/com.example-1.0-SNAPSHOT~/com/example/stage/destination:
sample
/opt/streamsets-datacollector-3.9.0/user-libs/com.example-1.0-SNAPSHOT~/com/example/stage/destination/sample:
Groups.class SampleDTarget.class SampleTarget$1.class SampleTarget.class
/opt/streamsets-datacollector-3.9.0/user-libs/com.example-1.0-SNAPSHOT~/com/example/stage/lib:
sample
/opt/streamsets-datacollector-3.9.0/user-libs/com.example-1.0-SNAPSHOT~/com/example/stage/lib/sample:
Errors.class
/opt/streamsets-datacollector-3.9.0/user-libs/com.example-1.0-SNAPSHOT~/com/example/stage/origin:
sample
/opt/streamsets-datacollector-3.9.0/user-libs/com.example-1.0-SNAPSHOT~/com/example/stage/origin/sample:
Groups.class SampleDSource.class SampleSource.class
/opt/streamsets-datacollector-3.9.0/user-libs/com.example-1.0-SNAPSHOT~/com/example/stage/processor:
sample
/opt/streamsets-datacollector-3.9.0/user-libs/com.example-1.0-SNAPSHOT~/com/example/stage/processor/sample:
Groups.class SampleDProcessor.class SampleProcessor.class
/opt/streamsets-datacollector-3.9.0/user-libs/sampleprocessor:
ElDefinitions.json PipelineStages.json data-collector-library-bundle.properties default.png
META-INF com datacollector-resource-bundles.json
/opt/streamsets-datacollector-3.9.0/user-libs/sampleprocessor/META-INF:
MANIFEST.MF maven
/opt/streamsets-datacollector-3.9.0/user-libs/sampleprocessor/META-INF/maven:
com.example
/opt/streamsets-datacollector-3.9.0/user-libs/sampleprocessor/META-INF/maven/com.example:
sampleprocessor

All you need is the com.example-1.0-SNAPSHOT.jar file, and it should be located in /opt/streamsets-datacollector-3.9.0/user-libs/sampleprocessor/lib.

Related

How to access Java dependency on command line?

I have a moderately old, small Java application which has an option to read and export PDF files using the Apache PDFBox library (hereunder, "pdfbox-app.jar"). All the files, including this resource, are stored in a single flat folder.
This works fine when called from a JAR file:
D:\Prog\!GitHub\Arena>java -jar Athena.jar NPCGenerator -p
OED NPC Generator
-----------------
Writing Gwenllian-ElfFtr1Wiz1.pdf
It similarly works fine when run from my IDE (jGrasp).
But it fails when called from the command line, outside of its JAR:
D:\Prog\!GitHub\Arena>java NPCGenerator -p
OED NPC Generator
-----------------
Writing Eoin-HalflingFtr1.pdf
Exception in thread "main" java.lang.NoClassDefFoundError: org/apache/pdfbox/pdmodel/PDDoc
ument
at CharacterPDF.writePDF(CharacterPDF.java:49)
at NPCGenerator.printToPDF(NPCGenerator.java:294)
at NPCGenerator.makeAllNPCs(NPCGenerator.java:270)
at NPCGenerator.main(NPCGenerator.java:308)
Caused by: java.lang.ClassNotFoundException: org.apache.pdfbox.pdmodel.PDDocument
at java.base/jdk.internal.loader.BuiltinClassLoader.loadClass(BuiltinClassLoader.j
ava:641)
at java.base/jdk.internal.loader.ClassLoaders$AppClassLoader.loadClass(ClassLoader
s.java:188)
at java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:520)
... 4 more
What should I be doing to run this on the command line outside of its own JAR?
You should put the pdfbox jar, and any other dependencies, on the classpath:
java -classpath .;pdfbox-app.jar NPCGenerator -p
Without that, Java doesn't know where to look for org/apache/pdfbox classes. It looks for .class files relative to the default classpath (which is just ., the current directory), but does not look inside jars.

snapscraft snap.yaml for java apps using swing ui on ubuntu

I have a basic java app using java.awt Swing UI and I want to create a snap for it so when it's installed there is a launcher available and my UI Launches. I use gradle and the jar task to create a jar which works fine.
Naturally I have in a class that when called loads my app just fine:
package com.foo
class Bar() {
static void main(String... args) { launchUI() }
}
I created a snap folder at the root of the project and a snap.yaml where i followed the instructions on https://snapcraft.io/docs/java-applications so i have a snap yaml that produces a snap file which also installs fine:
name: deepthought
base: core18
version: '0.0.7'
summary: ""
icon: gui/foo.png
description: |
Computes the ultimate answer for life the universe and everything
grade: devel # must be 'stable' to release into candidate/stable channels
confinement: devmode # use 'strict' once you have the right plugs and slots
# This doesn't work
#apps:
# htmldoc:
# command: desktop-launch $SNAP/bin/foo.sh
## desktop: share/applications/htmldoc.desktop
# plugs: [home, network, x11]
parts:
foopart:
plugin: gradle
source: https://github.com/bsautner/foo.git
source-type: git
gradle-options: [] # suppress running of tests and run the war task
gradle-output-dir: build/libs
I've spent quite some time trying to figure out:
If i create a shell script to run a java -jar foo.jar command it ends up in the /snap directory but it's not on the users path so they can't get to it
I've tried creating launchers but always get an error that my launcher can't be found, if i put it in my root folder as /bin/launch.sh snap can't find it and if i put it in the snap/bin/ folder i also get errors to not put things in the snap folder
When I do install my snap, i don't see where it puts my jar i want to execute, so i can't write a script that does that
I'd really appreciate if anyone can share a working snap.yaml for a java program with a launcher and if there is any mention of a path to something that you note where those files are in relation to the path of the /snap/snap.yaml file
ok figured it out - the docs don't say this but the files are relative to the root of the project so even though the yaml says this is the launch
apps:
cmd3:
command: usr/bin/foo.sh
foo.sh should be in the root of the project and the orginize section here moves it to the bin dir
foo:
plugin: gradle
source-type: local
source: .
build-packages:
- openjdk-11-jdk
stage-packages:
- openjdk-11-jdk
- x11-utils
organize:
${SNAPCRAFT_PART_BUILD}/jg-snap: usr/bin/foo.sh
The jar is in the /snap/foo/current/usr/jar directory

Javapackager - NoSuchFileException while copying libraries JDK 9

When I use the javapacker command, I get a NoSuchFileException when it should copy the libraries to the app folder. I'm running the following command:
javapackager -deploy -nosign -v -native image -name test
-appclass test.Test
-srcdir "E:\projects\test\target"
-srcfiles test-1.0.jar;lib
-outdir "E:\projects\test\target\dist"
Anything seams to work fine, but i get the NoSuchFileException inside the output and the libraries inside the lib folder are missing (did not get copied).
Running [D:\Java64\jdk-9.0.4\bin\java.exe, -version]
Creating app bundle: test in E:\projects\test\target\dist
"Adding modules: [java.rmi, java.sql, javafx.web, ... long list ...
outputDir = E:\projects\test\target\dist\test\runtime
modulePath = [D:\Java64\jdk-9.0.4\jmods]
addModules = [java.rmi, java.sql, javafx.web, ... long list ...
limitModules = []
excludeFileList = .*\.diz
stripNativeCommands = true
userArguments = {}
Using default package resource [icon] (add package/windows/test.ico to the class path to customize)
Using default package resource [Template for creating executable properties file.] (add package/windows\test.properties to the class path to customize)
Running [D:\Java64\jdk-9.0.4\bin\javapackager.exe, --icon-swap, C:\Users\XXX\AppData\Local\Temp\fxbundler15945263774290817340\windows\test.ico, E:\projects\test\target\dist\test\test.exe]
Icon File Name: C:\Users\XXX\AppData\Local\Temp\fxbundler15945263774290817340\windows\test.ico
Executable File Name: E:\projects\test\target\dist\test\test.exe
Running [D:\Java64\jdk-9.0.4\bin\javapackager.exe, --version-swap, C:\Users\XXX\AppData\Local\Temp\fxbundler15945263774290817340\windows\test.properties, E:\projects\test\target\dist\test\test.exe]
Resource File Name: C:\Users\XXX\AppData\Local\Temp\fxbundler15945263774290817340\windows\test.properties
Executable File Name: E:\projects\test\target\dist\test\test.exe
Exception: java.nio.file.NoSuchFileException: E:\projects\test\target\lib\animal-sniffer-annotations-1.14.jar -> E:\projects\test\target\dist\test\app\lib\animal-sniffer-annotations-1.14.jar
Config files are saved to C:\Users\XXX\AppData\Local\Temp\fxbundler15945263774290817340\windows. Use them to customize package.
Result application bundle: E:\projects\test\target\dist
I don't know what the following line of the output is trying to tell me. The file clearly exists inside my lib folder and the javapackager did find it itself by looking inside the lib folder, that I provided with the -srcfiles test-1.0.jar;lib argument.
Exception: java.nio.file.NoSuchFileException: E:\projects\test\target\lib\animal-sniffer-annotations-1.14.jar -> E:\projects\test\target\dist\test\app\lib\animal-sniffer-annotations-1.14.jar
Any idea why javapacker complains that the file would not exist?
As per https://docs.oracle.com/javase/9/tools/javapackager.htm#JSWOR719,
-srcfiles should be provided with a list of files. lib is a directory. I would try using -srcdir.
Also, if you get
Exception: java.lang.Exception: Error: Modules are not allowed in srcfiles
it means you have to use --module-path, and then set the main module using -m <module name>

How do I bundle a Custom C Shared Library with Java Application in Cloud Foundry?

I have previously been deploying a java application successfully in Cloud Foundry. However, now the developers have added a shared library compiled from C. I have added the .so to the .zip file (I'm using universal packager and deploying to Cloud Foundry by zip file) and ensured the .so is executable. However, when I deploy I get an error:
2016-12-23T14:39:01.013+00:00 [STG/0] [OUT] -----> Java Buildpack Version: eba4df6 | git://github.com/cloudfoundry/java-buildpack.git#eba4df6
2016-12-23T14:39:01.017+00:00 [STG/0] [ERR] [Buildpack] ERROR Compile failed with exception #<RuntimeError: No container can run this application. Please ensure that you’ve pushed a valid JVM artifact or artifacts using the -p command line argument or path manifest entry. Information about valid JVM artifacts can be found at https://github.com/cloudfoundry/java-buildpack#additional-documentation.
2016-12-23T14:39:01.017+00:00 [STG/0] [ERR] No container can run this application. Please ensure that you’ve pushed a valid JVM artifact or artifacts using the -p command line argument or path manifest entry. Information about valid JVM artifacts can be found at https://github.com/cloudfoundry/java-buildpack#additional-documentation.
2016-12-23T14:39:01.022+00:00 [STG/0] [ERR] Failed to compile droplet
2016-12-23T14:39:01.024+00:00 [STG/0] [OUT] Exit status 223
So my understanding is that I need to created a Custom Buildpack from the Java Buildpack. However, I am getting stuck with the code at this point:
https://github.com/ONSdigital/java-buildpack/blob/master/bin/compile#L28
# Adding crf tagger library
status "Adding crf tagger support"
cp $bp_dir/parsers/src/main/resources/libbackend.so $build_dir/vendor/
# update the PATH
status "Building runtime environment"
mkdir -p $build_dir/.profile.d
echo "export PATH=\"$HOME/bin:\$HOME/vendor/:\$PATH\";" > $build_dir/.profile.d/crftagger.sh
echo "export LD_LIBRARY_PATH=\"\$HOME/vendor/\";" >> $build_dir/.profile.d/crftagger.sh
From what I can gather I only need to modify the compile stage. What do I need to do to copy over the .so and have the Buildpack run successfully.
Please take a look to this https://docs.cloudfoundry.org/devguide/deploy-apps/deploy-app.html#profile
Pre-Runtime Hooks that's you need. I can't give you more details because you haven't provided enough information (project structure, do you have .profile in you root and etc.).
If you need to add shared libs try to use attr_reader :additional_libraries https://github.com/cloudfoundry/java-buildpack/blob/master/docs/extending-droplet.md

CMake: Generate a jar file using multiple jar dependencies (out of source build)

I have spent a few days now to do the following with CMake (3.5.2). Using an out of source build, create a jar file relying on multiple external jar dependencies. In my manifest, I want only to specify Main-Class, nothing else.
This is what I have in my croco/owl2gdb/CMakeLists.txt directory:
cmake_minimum_required (VERSION 3.5)
find_package(Java REQUIRED)
include(UseJava)
project (owl2gdb)
SET(CLASSPATH ${CMAKE_CURRENT_SOURCE_DIR}/lib/commons-lang3-3.3.2.jar:${CMAKE_CURRENT_SOURCE_DIR}/lib/concurrentlinkedhashmap-lru-1.4.2.jar:${CMAKE_CURRENT_SOURCE_DIR}/lib/HermiT.jar)
list(APPEND CMAKE_JAVA_INCLUDE_PATH ${CLASSPATH})
SET(SOURCE
${CMAKE_CURRENT_SOURCE_DIR}/src/owl2gdb/OWL2GDB.java
${CMAKE_CURRENT_SOURCE_DIR}/src/graphdatabase/GraphDatabase.java
${CMAKE_CURRENT_SOURCE_DIR}/src/cpp/CPPFileWriter.java
${CMAKE_CURRENT_SOURCE_DIR}/src/cpp/CppHeaderFileWriter.java
${CMAKE_CURRENT_SOURCE_DIR}/src/cpp/CppSourceFileWriter.java
${CMAKE_CURRENT_SOURCE_DIR}/src/cpp/CppTextTemplate.java
${CMAKE_CURRENT_SOURCE_DIR}/src/cpp/GraphDatabaseHeaderWriter.java
${CMAKE_CURRENT_SOURCE_DIR}/src/cpp/GraphDatabaseSourceWriter.java
${CMAKE_CURRENT_SOURCE_DIR}/src/knowledge/MyOntology.java
${CMAKE_CURRENT_SOURCE_DIR}/src/knowledge/MyOWLClass.java
${CMAKE_CURRENT_SOURCE_DIR}/src/knowledge/MyOWLDataProperty.java
${CMAKE_CURRENT_SOURCE_DIR}/src/knowledge/MyOWLObjectProperty.java
${CMAKE_CURRENT_SOURCE_DIR}/src/knowledge/MyOWLOntology.java
${CMAKE_CURRENT_SOURCE_DIR}/src/tool/Util.java)
set(JAVA_JAR_ENTRY_POINT owl2gdb/OWL2GDB)
add_jar(owl2gdb ${SOURCE})
add_custom_command(TARGET owl2gdb
POST_BUILD
COMMAND jar u0fm owl2gdb.jar ${CMAKE_CURRENT_SOURCE_DIR}/META-INF/MANIFEST.MF ${SOURCE}
)
In my root directory, I do:
mkdir _build
cd _build
cmake ..
make
owl2gdb.jar is indeed generated in _build/owl2gdb/owl2gdb.jar. Then
cd owl2gdb
java -jar owl2gdb.jar
gives me
Exception in thread "main" java.lang.NoClassDefFoundError: com/martiansoftware/jsap/StringParser
Basically, the external jars are not included in my owl2gdb.jar. I don't really know where to go from here. Is there a problem with cmake not properly using CMAKE_JAVA_INCLUDE_PATH? What is missing in my CMakeLists.txt?
a jar tf owl2gdb.jar returns:
META-INF/
META-INF/MANIFEST.MF
tool/Util.class
graphdatabase/GraphDatabase$1.class
graphdatabase/GraphDatabase.class
cpp/CPPFileWriter.class
cpp/GraphDatabaseHeaderWriter.class
cpp/CppHeaderFileWriter.class
cpp/CppTextTemplate.class
cpp/CppSourceFileWriter.class
cpp/GraphDatabaseSourceWriter.class
owl2gdb/OWL2GDB.class
knowledge/MyOWLClass.class
knowledge/MyOWLDataProperty.class
knowledge/MyOntology.class
knowledge/MyOWLObjectProperty.class
knowledge/MyOWLOntology.class
usr/local/me/github/croco/owl2gdb/src/owl2gdb/OWL2GDB.java
usr/local/me/github/croco/owl2gdb/src/owl2gdb/src/graphdatabase/GraphDatabase.java
usr/local/me/github/croco/owl2gdb/src/owl2gdb/src/cpp/CPPFileWriter.java
usr/local/me/github/croco/owl2gdb/src/owl2gdb/src/cpp/CppHeaderFileWriter.java
usr/local/me/github/croco/owl2gdb/src/owl2gdb/src/cpp/CppSourceFileWriter.java
usr/local/me/github/croco/owl2gdb/src/owl2gdb/src/cpp/CppTextTemplate.java
usr/local/me/github/croco/owl2gdb/src/owl2gdb/src/cpp/GraphDatabaseHeaderWriter.java
usr/local/me/github/croco/owl2gdb/src/owl2gdb/src/cpp/GraphDatabaseSourceWriter.java
usr/local/me/github/croco/owl2gdb/src/owl2gdb/src/knowledge/MyOntology.java
usr/local/me/github/croco/owl2gdb/src/owl2gdb/src/knowledge/MyOWLClass.java
usr/local/me/github/croco/owl2gdb/src/owl2gdb/src/knowledge/MyOWLDataProperty.java
usr/local/me/github/croco/owl2gdb/src/owl2gdb/src/knowledge/MyOWLObjectProperty.java
usr/local/me/github/croco/owl2gdb/src/owl2gdb/src/knowledge/MyOWLOntology.java
usr/local/me/github/croco/owl2gdb/src/owl2gdb/src/tool/Util.java
This is the only thing I have in my manifest:
Main-Class: owl2gdb.OWL2GDB

Categories