We Keep Coding

Dynamic gradle command for Exec task fails

I am trying to publish Lambda zip file to s3- s3://my-aws-lambda/<projectName>/[release|SNAPSHOT]/. The task defined below, publishToS3 fails with the message
Caused by: java.io.FileNotFoundException: /Users/me/my-lambda/build/distributions
when I run
./gradlew clean build -x test -x release
Appreciate any help. Thanks.
task buildZip(type: Zip) {
from compileJava
from processResources
into('lib') {
from configurations.runtimeClasspath
}
}
task publishToS3(type: Exec, dependsOn: buildZip) {
onlyIf { file("${project.buildDir}/distributions").exist() }
def artifacts = new FileNameByRegexFinder().getFileNames("${project.buildDir}/distributions", /.*\.zip/)
assert artifacts.size() == 1
def isSnapShot = artifacts[0].endsWith('-SNAPSHOT.zip')
def releaseCmd = ("aws s3 cp " +
"${artifacts[0]} " +
"s3://my-aws-lambdas/${project.name}/${isSnapShot ? 'SNAPSHOT' : 'release'}/ ").trim().tokenize(' ') as List
workingDir "${project.buildDir}/distributions"
commandLine releaseCmd
}
build.dependsOn buildZip

If you are on UNIX environment, you can use the find command to search the files and use it's output.
task scanFiles() {
def a= "find ${project.buildDir} -name *.zip".execute().text
String[] files=a.split('\n')
if(files.length == 1){
println("We'v got a file :"+a)
}
else if(files.length ==0){
println("We've not no files ")
}
else{
println("We've got multiple files\n"+a)
}
}

Importing a groovy file from one package to another groovy file in another package

I have a child java project which has groovy files added in classpath using eclipse. Parent java project triggers some functionality in child which uses Groovy library to run the scripts. So import works fine in eclipse environment with opened child project but if I run it from command line or if I close child project then I get groovy compilation error at import statement. How can I resolve this ? I want to avoid using evaluate() method.
Following is my master groovy:
package strides_business_script
abstract class Business_Script extends Script {
//some stuff
}
Following is the another groovy:
import static strides_business_script.StridesBusiness_Script.*;
import org.json.simple.JSONArray;
import org.json.simple.JSONObject;
String Key = Part_Product_PartDetails
boolean containsData = checkIncomingMessage(Key)
if(containsData) {
def edgeKeyList = [PPR]
JSONArray partDetails = appendEdgeValueToMsg(edgeKeyList,Key,vertex,messageIterator);
//deleteMessages(Key);
JSONObject jsonObject = constructInfoWithPropertyJSON("NAME,PRODUCTTYPE,FGTYPE,UOM,ITEMCLASSIFICATIONBYMARKET");
jsonObject.put("PARTS",partDetails);
send(Product_AggPO_ProductDetails,convertJSONToString(jsonObject));
}
Edit:
My master script Business_Script.groovy resides in scripts/strides_business_script/ folder. All other scripts are in scripts/StridesComputationScripts/ folder and they import the Business_Script.groovy.
I run the application with remote debugging enabled like this:
java -cp "./lib/*:./scripts/strides_business_script/Business_Script.groovy" -Xdebug -Xrunjdwp:transport=dt_socket,address=6969,server=y -Dhibernate.cfg.xml.path=./conf/hibernate.cfg.xml -Dlog4j.configuration=file:./conf/log4j.properties com.biglabs.dataExtractor.dataDump.DataDumpDriver 7
and here I am trying to parse all computation scripts.
for (String scriptName : files) {
Script script = groovyShell.parse(new File(
SCRIPT_PLACED_AT + Constants.SLASH
+ SCRIPT_FILE_FOLDER + Constants.SLASH
+ scriptName));
scriptMping.put(scriptName, script);
}
It throws following exception while parsing using groovy shell:
org.codehaus.groovy.control.MultipleCompilationErrorsException: startup failed:
/home/manoj/strides/release/strides/scripts/StridesComputationScripts/PRODUCT-script.groovy: 2: unable to resolve class strides_business_script.StridesBusiness_Script
# line 2, column 1.
import static strides_business_script.Business_Script.*;
^
/home/manoj/strides/release/strides/scripts/StridesComputationScripts/PRODUCT-script.groovy: 2: unable to resolve class strides_business_script.StridesBusiness_Script
# line 2, column 1.
import static strides_business_script.Business_Script.*;
^
2 errors

Fixed it by adding script path in comiler configuration:
CompilerConfiguration compilerConfiguration = new CompilerConfiguration();
String path = SCRIPT_PLACED_AT;
if(!SCRIPT_PLACED_AT.endsWith("/")){
path = path+ "/";
}
compilerConfiguration.setClasspath(path);
GroovyShell groovyShell = new GroovyShell(
compilerConfiguration);
for (String scriptName : files) {
Script script = groovyShell.parse(new File(
SCRIPT_PLACED_AT + Constants.SLASH
+ SCRIPT_FILE_FOLDER + Constants.SLASH
+ scriptName));
scriptMping.put(scriptName, script);
}

How to read a properties files and use the values in project Gradle script?

I am working on a Gradle script where I need to read the local.properties file and use the values in the properties file in build.gradle. I am doing it in the below manner. I ran the below script and it is now throwing an error, but it is also not doing anything like creating, deleting, and copying the file. I tried to print the value of the variable and it is showing the correct value.
Can someone let me know if this is the correct way to do this? I think the other way is to define everything in the gradle.properties and use it in the build.gradle. Can someone let me know how could I access the properties in build.gradle from build.properties?
build.gradle file:
apply plugin: 'java'
// Set the group for publishing
group = 'com.true.test'
/**
* Initializing GAVC settings
*/
def buildProperties = new Properties()
file("version.properties").withInputStream {
stream -> buildProperties.load(stream)
}
// If jenkins build, add the jenkins build version to the version. Else add snapshot version to the version.
def env = System.getenv()
if (env["BUILD_NUMBER"]) buildProperties.test+= ".${env["BUILD_NUMBER"]}"
version = buildProperties.test
println "${version}"
// Name is set in the settings.gradle file
group = "com.true.test"
version = buildProperties.test
println "Building ${project.group}:${project.name}:${project.version}"
Properties properties = new Properties()
properties.load(project.file('build.properties').newDataInputStream())
def folderDir = properties.getProperty('build.dir')
def configDir = properties.getProperty('config.dir')
def baseDir = properties.getProperty('base.dir')
def logDir = properties.getProperty('log.dir')
def deployDir = properties.getProperty('deploy.dir')
def testsDir = properties.getProperty('tests.dir')
def packageDir = properties.getProperty('package.dir')
def wrapperDir = properties.getProperty('wrapper.dir')
sourceCompatibility = 1.7
compileJava.options.encoding = 'UTF-8'
repositories {
maven { url "http://arti.oven.c:9000/release" }
}
task swipe(type: Delete) {
println "Delete $projectDir/${folderDir}"
delete "$projectDir/$folderDir"
delete "$projectDir/$logDir"
delete "$projectDir/$deployDir"
delete "$projectDir/$packageDir"
delete "$projectDir/$testsDir"
mkdir("$projectDir/${folderDir}")
mkdir("projectDir/${logDir}")
mkdir("projectDir/${deployDir}")
mkdir("projectDir/${packageDir}")
mkdir("projectDir/${testsDir}")
}
task prepConfigs(type: Copy, overwrite:true, dependsOn: swipe) {
println "The name of ${projectDir}/${folderDir} and ${projectDir}/${configDir}"
from('${projectDir}/${folderDir}')
into('${projectDir}/$configDir}')
include('*.xml')
}
build.properties file:
# -----------------------------------------------------------------
# General Settings
# -----------------------------------------------------------------
application.name = Admin
project.name = Hello Cool
# -----------------------------------------------------------------
# ant build directories
# -----------------------------------------------------------------
sandbox.dir = ${projectDir}/../..
reno.root.dir=${sandbox.dir}/Reno
ant.dir = ${projectDir}/ant
build.dir = ${ant.dir}/build
log.dir = ${ant.dir}/logs
config.dir = ${ant.dir}/configs
deploy.dir = ${ant.dir}/deploy
static.dir = ${ant.dir}/static
package.dir = ${ant.dir}/package
tests.dir = ${ant.dir}/tests
tests.logs.dir = ${tests.dir}/logs
external.dir = ${sandbox.dir}/FlexCommon/External
external.lib.dir = ${external.dir}/libs

If using the default gradle.properties file, you can access the properties directly from within your build.gradle file:
gradle.properties:
applicationName=Admin
projectName=Hello Cool
build.gradle:
task printProps {
doFirst {
println applicationName
println projectName
}
}
If you need to access a custom file, or access properties which include . in them (as it appears you need to do), you can do the following in your build.gradle file:
def props = new Properties()
file("build.properties").withInputStream { props.load(it) }
task printProps {
doFirst {
println props.getProperty("application.name")
println props.getProperty("project.name")
}
}
Take a look at this section of the Gradle documentation for more information.
Edit
If you'd like to dynamically set up some of these properties (as mentioned in a comment below), you can create a properties.gradle file (the name isn't important) and require it in your build.gradle script.
properties.gradle:
ext {
subPath = "some/sub/directory"
fullPath = "$projectDir/$subPath"
}
build.gradle
apply from: 'properties.gradle'
// prints the full expanded path
println fullPath

We can use a separate file (config.groovy in my case) to abstract out all the configuration.
In this example, we're using three environments viz.,
dev
test
prod
which has properties serverName, serverPort and resources. Here we're expecting that the third property resources may be same in multiple environments and so we've abstracted out that logic and overridden in the specific environment wherever necessary:
config.groovy
resources {
serverName = 'localhost'
serverPort = '8090'
}
environments {
dev {
serverName = 'http://localhost'
serverPort = '8080'
}
test {
serverName = 'http://www.testserver.com'
serverPort = '5211'
resources {
serverName = 'resources.testserver.com'
}
}
prod {
serverName = 'http://www.productionserver.com'
serverPort = '80'
resources {
serverName = 'resources.productionserver.com'
serverPort = '80'
}
}
}
Once the properties file is ready, we can use the following in build.gradle to load these settings:
build.gradle
loadProperties()
def loadProperties() {
def environment = hasProperty('env') ? env : 'dev'
println "Current Environment: " + environment
def configFile = file('config.groovy')
def config = new ConfigSlurper(environment).parse(configFile.toURL())
project.ext.config = config
}
task printProperties {
println "serverName: $config.serverName"
println "serverPort: $config.serverPort"
println "resources.serverName: $config.resources.serverName"
println "resources.serverPort: $config.resources.serverPort"
}
Let's run these with different set of inputs:
gradle -q printProperties
Current Environment: dev
serverName: http://localhost
serverPort: 8080
resources.serverName: localhost
resources.serverPort: 8090
gradle -q -Penv=dev printProperties
Current Environment: dev
serverName: http://localhost
serverPort: 8080
resources.serverName: localhost
resources.serverPort: 8090
gradle -q -Penv=test printProperties
Current Environment: test
serverName: http://www.testserver.com
serverPort: 5211
resources.serverName: resources.testserver.com
resources.serverPort: 8090
gradle -q -Penv=prod printProperties
Current Environment: prod
serverName: http://www.productionserver.com
serverPort: 80
resources.serverName: resources.productionserver.com
resources.serverPort: 80

Another way... in build.gradle:
Add :
classpath 'org.flywaydb:flyway-gradle-plugin:3.1'
And this :
def props = new Properties()
file("src/main/resources/application.properties").withInputStream { props.load(it) }
apply plugin: 'flyway'
flyway {
url = props.getProperty("spring.datasource.url")
user = props.getProperty("spring.datasource.username")
password = props.getProperty("spring.datasource.password")
schemas = ['db_example']
}

This is for Kotlin DSL (build.gradle.kts):
import java.util.*
// ...
val properties = Properties().apply {
load(rootProject.file("my-local.properties").reader())
}
val prop = properties["myPropName"]
In Android projects (when applying the android plugin) you can also do this:
import com.android.build.gradle.internal.cxx.configure.gradleLocalProperties
// ...
val properties = gradleLocalProperties(rootDir)
val prop = properties["propName"]

Just had this issue come up today. We found the following worked both locally and in our pipeline:
In build.gradle:
try {
apply from: 'path/name_of_external_props_file.properties'
} catch (Exception e) {}
This way when an external props file which shouldn't get committed to Git or whatever (as in our case) you are using is not found in the pipeline, this 'apply from:' won't throw an error in it. In our use case we have a file with a userid and password that should not get committed to Git. Aside from the problem of file-reading: we found that the variables we had declared in the external file, maven_user and maven_pass, had in fact to be declared in gradle.properties. That is they simply needed to be mentioned as in:
projectName=Some_project_name
version=1.x.y
maven_user=
maven_pass=
We also found that in the external file we had to put single-quotes around these values too or Gradle got confused. So the external file looked like this:
maven_user='abc123'
maven_pass='fghifh7435bvibry9y99ghhrhg9539y5398'
instead of this:
maven_user=abc123
maven_pass=fghifh7435bvibry9y99ghhrhg9539y5398
That's all we had to do and we were fine. I hope this may help others.

scala matchers works from jar not from sbt

I have a code blurb thats doing some reflection on scala class files and looking for annotations,something like
// Create a new class loader with the directory
val cl = new URLClassLoader(allpaths.toArray)
getTypeNamesFromPath(sourceDir).map(s => {
val cls = cl.loadClass(s)
cls.getAnnotations.map(an =>
an match {
case q: javax.ws.rs.Path => println("found annotation")
case _ =>
})
})
This code prints "found annotation" when assembling this in a jar and running using java -jar, but doesn't print anything when running from sbt run.
sbt version is 13.8,
scala version 2.11.7
for completeness
private def getTypeNamesFromPath(file: File, currentPath: mutable.Stack[String] = new mutable.Stack[String]()): List[String] = {
if (file.isDirectory) {
var list = List[String]()
for (f <- file.listFiles()) {
currentPath.push(f.getName)
list = list ++ getTypeNamesFromPath(f, currentPath)
}
return list
}
if (currentPath.isEmpty)
throw new IllegalArgumentException(file.getAbsolutePath)
currentPath.pop()
if (file.getName.endsWith(".class")) {
return List(currentPath.foldRight("")((s, b) => if (b.isEmpty) s else b + "." + s) + "." + file.getName.stripSuffix(".class"))
}
return List()
}

so it turns out i need two things...
use URLClassLoader from scala.tools.nsc.util.ScalaClassLoader.URLClassLoader, this will solve loading the right classes.
set the fork := true in build.sbt, because once you do step #1, you will run into issues with class name clashes because sbt runs with own class paths and own version of class loader, more on it here http://www.scala-sbt.org/0.13.0/docs/Detailed-Topics/Forking.html

Gradle creating duplicate start scripts into bin directory

I am trying to create multiple start script files through gradle. But somehow one particular start script file is getting duplicated.
startScripts.enabled = false
run.enabled = false
def createScript(project, mainClass, name) {
project.tasks.create(name: name, type: CreateStartScripts) {
outputDir = new File(project.buildDir, 'scripts')
mainClassName = mainClass
applicationName = name
classpath = jar.outputs.files + project.configurations.runtime
doLast {
def windowsScriptFile = file getWindowsScript()
def unixScriptFile = file getUnixScript()
windowsScriptFile.text = windowsScriptFile.text.replace('%APP_HOME%\\lib\\conf', '%APP_HOME%\\conf')
unixScriptFile.text = unixScriptFile.text.replace('$APP_HOME/lib/conf', '$APP_HOME/conf')
}
}
project.tasks[name].dependsOn(project.jar)
project.applicationDistribution.with {
into("bin") {
from(project.tasks[name])
fileMode = 0755
}
}
}
// Call this for each Main class you want to expose with an app script
createScript(project, 'com.main.A', 'A')
createScript(project, 'com.main.B', 'B')
in bin directory I can see,
A.sh
A.sh
A.bat
A.bat
B.sh
B.bat
What am I missing here? How to fix this?
Thank you for help.

I solved this problem. Actually it was a mistake from my side and thanks to #Opal. I somehow forgot to delete 'mainClassName="com.main.A"' line from the header.
Also I have to add
distZip {
duplicatesStrategy = 'exclude'
}