We Keep Coding

Review of gitlab CI using yml

Hi I am using the following gitlab yml file for setting up my pipeline. The project is a maven Java project. But I am not able to run all the steps successfully. Here is the gitlab yml:
image: maven:3.5-jdk-8
variables:
MAVEN_CLI_OPTS: "-s .m2/settings.xml --batch-mode"
MAVEN_OPTS: "-Dmaven.repo.local=.m2/repository"
include:
- template: Security/SAST.gitlab-ci.yml
cache:
paths:
- .m2/settings.xml
# Define stages
# Stages group various steps into one block,
# if any step fails, the entire stage fails
stages:
- validate
- compile
- SonarQube
- test
validate:
stage: validate
script:
- mvn validate
compile:
stage: compile
script:
- mvn $MAVEN_CLI_OPTS compile
sonarqube-check:
image: maven:3.6.3-jdk-11
stage: SonarQube
variables:
SONAR_USER_HOME: "${CI_PROJECT_DIR}/.sonar"
script:
- mvn sonar:sonar -Dsonar.projectKey=key -Dsonar.host.url=url -Dsonar.login=id
allow_failure: true
spotbugs-sast:
variables:
COMPILE: "false"
SECURE_LOG_LEVEL: "debug"
artifacts:
reports:
sast: gl-sast-report.json
#spotbugs-sast:
# variables:
# SECURE_LOG_LEVEL: "debug"
#FAIL_NEVER: 1
test:
image: maven:3.5-jdk-8
stage: test
script:
- mkdir -p /opt/path/conf/project/
- echo ${CI_PROJECT_DIR}
- cp "${CI_PROJECT_DIR}/project.properties" "/opt/path/conf/project/"
- mvn $MAVEN_CLI_OPTS test -B
But I am getting errors in stages: sonarqube, spotbug-sast and test.
In sonarqube, it isshowing error as: Failed to resolve the project dependency with a list of jar files as:
The following artifacts could not be resolved: webpay:webpay-client:jar:4.0.4, mpienhanced:mpienhanced:jar:1.0.0, webpay:webpay-mpi:jar:4.3.9, webpay:matrix-mpi:jar:1.27.4, webpay:vbv-matrix:jar:1.12.1, webpay:xercesImpl:jar:2.12.0, webpay:xss4j:jar:0.0.1, webpay:xmlParserAPIs:jar:2.11.0, webpay:webpay-mpi-util:jar:4.2.2
In spotbugs-sast I am getting the error as:
[INFO] [Find Security Bugs] [2022-01-13T10:41:39Z] ▶ Found 1 analyzable projects.
[FATA] [Find Security Bugs] [2022-01-13T10:41:39Z] ▶ lstat /root/.m2/repository: no such file or directory
In test stage it is not able to get the properties file from the path that is mentioned in the config file. I have tried to place the properties file at all the places and specify the path but to no luck.
Can someone please help resolve my issues. Thanks in advance.
Let me know if any additional info is required.

You could try going back to a documentation example and tweak it to incrementally make it like the one in your question.
But be warned: spotbugs-sast won't analyze Java much longer.
See GitLab 14.10 (April 2022)
Faster, easier Java scanning in SAST
GitLab Static Application Security Testing (SAST) now uses Semgrep to scan Java code, building on previous support for Go (introduced in GitLab 14.4) and for JavaScript, TypeScript, and Python (introduced in GitLab 13.12).
The Semgrep-based analyzer runs significantly faster—up to 7 times faster in our testing than the existing analyzer that’s based on SpotBugs.
It also doesn’t need to compile your code before scanning, so it’s much simpler to use than SpotBugs.
The Static Analysis and Vulnerability Research teams worked together to translate rules to the Semgrep format, preserving most existing rules.
We also updated, refined, and tested the rules as we converted them.
If you use the GitLab-managed SAST template (SAST.gitlab-ci.yml), both Semgrep and SpotBugs now run whenever Java code is found.
In GitLab Ultimate, the Security Dashboard combines findings from the two analyzers, so you won’t see duplicate vulnerability reports.
In GitLab 15.0, as we announced, we’ll change the GitLab-managed SAST template (SAST.gitlab-ci.yml) to only run the Semgrep-based analyzer for Java code.
The SpotBugs-based analyzer will still scan other JVM languages like Groovy, Kotlin, and Scala.
If you have any questions, feedback, or issues with the new Semgrep-based Java scanning, please file an issue, we’ll be glad to help.
See Documentation and Issue.
And GitLab 15.1 (June 2022) adds improvements which could help:
Static Analysis analyzer updates
GitLab Static Analysis includes many security analyzers that the GitLab Static Analysis team actively manages, maintains, and updates. The following analyzer updates were published during the 15.1 release milestone. These updates bring additional coverage, bug fixes, and improvements.
Secret Detection analyzer updated for better offline support and easier debugging. See CHANGELOG for details.
Improve logging
Use checked-out copy of the repository if git fetch fails
Fall back to scanning the latest commit if automatic diff detection fails
SpotBugs analyzer updated to SpotBugs version 4.7.0 and find-sec-bugs version 1.12.0. See CHANGELOG for details.
Update gradle and grails to support Java 17
Set Java 17 as the system-wide default version
Use ‘assemble’ task for Gradle projects, instead of ‘build’, to support custom GRADLE_CLI_OPTS (see issue #299872)
Add additional detection rules
If you include the GitLab-managed SAST template (SAST.gitlab-ci.yml), you don’t need to do anything to receive these updates. However, if you override or customize your own CI/CD template, you need to update your CI/CD configurations.
To remain on a specific version of any analyzer, you can pin to a minor version of an analyzer. Pinning to a previous version prevents you from receiving automatic analyzer updates and requires you to manually bump your analyzer version in your CI/CD template.
For previous changes, see last month’s updates.
See Documentation and Issue.

Turn off parts of code to speed up build times (Gradle)

I have an Android project that has grown with time, and with the size have grown the gradle build times.
It was bearable while it was under the 65k limit - around 14s.
Now with multidex it takes 36s.
So my question is - are there any ways to "turn off" parts of the code that are not being used so it's back under the 65k limit?
For e.g. turn off the amazon s3 sdk which is brought in via gradle and has n thousands of methods.
I know you can strip code with proguard, but that just bumps up the build time even higher.
I'm happy with it crashing at runtime when I open the parts that use it, just want to make testing quicker.
At the moment when I remove amazon from gradle imports, I obviously get this:
Error:(24, 26) error: package com.amazonaws.auth does not exist
Is there a way to somehow ignore the error? I know that in Picasso, it has a runtime check to see if you have OkHttp, and if you haven't - use standard networking.
static Downloader createDefaultDownloader(Context context) {
if (SDK_INT >= GINGERBREAD) {
try {
Class.forName("com.squareup.okhttp.OkHttpClient");
return OkHttpLoaderCreator.create(context);
} catch (ClassNotFoundException ignored) {}
}
return new UrlConnectionDownloader(context);
}
Is there something like this I could do? Or any other way?

The only realistic way of doing this (that I'm aware of) is to refactor your project so that your packages are split into separate modules. You would therefore have separate gradle build files for each module, but would only have to recompile each module whenever they were touched. You could, for instance, have a data access package and a UI package. That seems like a pretty natural split.
I realize that this is a disappointing answer but the issue you're complaining about is that your build dependencies require all those extra unnecessary libraries and method calls: not that your code uses them.
The only other tip I can give you is that the Google Play API kit has tens of thousands of method calls. If you can use only the pieces that you're using you stand a much better chance of being beneath the 65k limit.

It is possible to specify compile-time dependencies for each build type independently. I use this method to include "production-only" dependencies in only the release builds, reducing the method count for debug builds.
For example, I only include Crashlytics in release builds. So in build.gradle I include the dependency for only my release build (and beta and alpha):
releaseCompile('com.crashlytics.sdk.android:crashlytics:2.5.5#aar') {
transitive = true;
}
Then I abstract the functionality of Crashlytics into a class called CrashReportingService. In my debug source code, this class does nothing:
/app/src/debug/java/com/example/services/CrashReportingService.java:
public class CrashReportingService {
public static void initialise(Context context) {
}
public static void logException(Throwable throwable) {
}
}
And I flesh out the implementation in my release source code:
/app/src/release/java/com/example/services/CrashReportingService.java
public class CrashReportingService {
public static void initialise(Context context) {
Fabric.with(context, new Crashlytics());
}
public static void logException(Throwable throwable) {
Crashlytics.getInstance().core.logException(throwable);
}
}
Crashlytics is now only included in release builds and there is no reference to Crashlytics in my debug builds. Back under 65k methods, hooray!

I have got another option. That also helps to speed up but not as your
demand. That is using demon.
If you use the new Gradle build system with Android (or Android Studio) you might have realized, that even the simplest Gradle call (e.g. gradle project or grade tasks) is pretty slow. On my computer it took around eight seconds for that kind of Gradle calls. You can decrease this startup time of Gradle (on my computer down to two seconds), if you tell Gradle to use a daemon to build. Just create a file named gradle.properties in the following directory:
/home/<username>/.gradle/ (Linux)
/Users/<username>/.gradle/ (Mac)
C:\Users\<username>\.gradle (Windows)
Add this line to the file:
org.gradle.daemon=true
From now on Gradle will use a daemon to build, whether you are using Gradle from command line or building in Android Studio. You could also place the gradle.properties file to the root directory of your project and commit it to your SCM system. But you would have to do this, for every project (if you want to use the daemon in every project).
Note: If you don’t build anything with Gradle for some time (currently
3 hours), it will stop the daemon, so that you will experience a long
start-up time at the next build.
How does the Gradle Daemon make builds faster?
The Gradle Daemon is a long lived build process. In between builds it waits idly for the next build. This has the obvious benefit of only requiring Gradle to be loaded into memory once for multiple builds, as opposed to once for each build. This in itself is a significant performance optimization, but that's not where it stops.
A significant part of the story for modern JVM performance is runtime code optimization. For example, HotSpot (the JVM implementation provided by Oracle and used as the basis of OpenJDK) applies optimization to code while it is running. The optimization is progressive and not instantaneous. That is, the code is progressively optimized during execution which means that subsequent builds can be faster purely due to this optimization process.
Experiments with HotSpot have shown that it takes somewhere between 5
and 10 builds for optimization to stabilize. The difference in
perceived build time between the first build and the 10th for a Daemon
can be quite dramatic.
The Daemon also allows more effective in memory caching across builds. For example, the classes needed by the build (e.g. plugins, build scripts) can be held in memory between builds. Similarly, Gradle can maintain in-memory caches of build data such as the hashes of task inputs and outputs, used for incremental building.
Some other ways to speed up the process
How to Speed Up Your Gradle Build From 90 to 8 Minutes?
How to optimize gradle build performance regarding build duration and RAM usage?

determine size of object: best way to use instrumentation in scala/sbt

According to this question, the standard way to determine the memory size of an object in Java is by using java.lang.instrumentation. After to some research, it looks like there is no Scala specific way to achieve this, so the Java approach should also applies here.
Unfortunately, for a Scala programmer without Java background it is not fully straightforward to adapt this technique in Scala. My questions are:
Question 1
What exactly is happening here? I guess the reason why we have to put a class like ObjectSizeFetcher in a separate JAR is to ensure that it is somehow loaded before the actual program where we want to use it. I assume it is not possible to use instrumentation without the Premain-Class entry and the parameter -javaagent:TheJarContainingObjectFetcher.jar?
Question 2
Is there a simple way to implement the complete work flow in SBT? Currently I only see a somewhat cumbersome solution: I first have to set up a secondary SBT project where I define ObjectSizeFetcher and package it into a JAR. So far I did not figure out how to automatically add the Premain-Class entry to the JAR during packaging, so I would have to solve that manually. Than I can add the resulting JAR to the local libraries of the project where I want to use getObjectSize. For this project I now have to enable fork in run and use javaOptions in run += "-javaagent:TheJarContainingObjectFetcher.jar". Is there a more simple (and less intrusive) work flow to quickly use instrumentation within an existing SBT project? Maybe I can tell SBT directly about a Premain-Class to make this secondary JAR unnecessary?
Question 3
Would you recommend a completely different way to evaluate the memory usage of an object in Scala?

Answer 1: Yes, if you want iinstrumentation, you need to get an instance. You probably can't get it without Premain-Class and -javaagent.
Answer 2: You can (and may need to) use classloaders and create a very simple bootstrap project (in Java or in Scala with Proguard). There are two reasons:
The first reason: conveniency. You may use java.net.URLClassLoader to include standard Scala library and the classes directory of your project. You will not need to repackage it in JAR any more when testing.
The second reason: preventing JAR hell. You probably know that Scala is not binary compatible. You should also know that the Java agent is loaded with the application in the same classloader. If the classloader includes Scala library, application can't simply use another Scala version.
However, if the Java agent doesn't use directly Scala library (e.g. it is a bootstrap application and loads the real agent and its libraries in another classloader), the instrumented application is free to use any Scala library.
Answer 3: I would probably use instrumentation, too.

Answer 3: you can have a look to ktoso/sbt-jol which displays the JOL (Java Object Layout), ie the analysis of object layout schemes in JVMs
// project/plugins.sbt
addSbtPlugin("pl.project13.sbt" % "sbt-jol" % pluginVersionHere)
It does include the size:
> jol:internals example.Entry
...
[info] example.Entry object internals:
[info] OFFSET SIZE TYPE DESCRIPTION VALUE
[info] 0 12 (object header) N/A
[info] 12 4 int Entry.value N/A
[info] 16 4 String Entry.key N/A
[info] 20 4 (loss due to the next object alignment)
[info] Instance size: 24 bytes
[info] Space losses: 0 bytes internal + 4 bytes external = 4 bytes total

Play Framework 2.0 Unit Testing

Having a very basic issue with running tests in a Java Play 2.0 app. It won't pick up my tests.
I've tried to put a test folder in the root of the project and use Junit as the docs seem to suggest. This results in:
$ play test
[info] Loading project definition from /Users/.../play20TestTest/project
[info] Set current project to play20TestTest (in build file:/Users/.../testapps/play20TestTest/)
[info] No tests to run for test:test
[success] Total time: 1 s, completed May 22, 2012 11:16:52 AM
I've also tried putting test under app. This at least picks up my code, but the test packages aren't available on my classpath so I assume it's incorrect.
I made a simple example and copied SimpleTest from the docs exactly: https://github.com/jsimone/Play2UnitTest

You placed your test in test/test/SimpleTest.java, move it to test/SimpleTest.java and it will work with play test command.

Preon on Android

I tried to run a sample Preon application on Android 2.1 without luck. I wonder if running a Preon application on Android is even possible. How hard would it be to make the Preon framework Dalvik friendly?
Preon is a Java library for building codecs for bitstream-compressed data in a declarative way. Think JAXB or Hibernate, but then for binary encoded data written by Wilfred Springer.
Below are my finding when trying to run a simple application that uses Preon in Android:
Preon has a dependency on Pecia. Pecia indirectly depends on stax-api which is not supported out of the box in Android. Is the stax-api used in the Preon core processing? Can I exclude the stax-api from the Preon dependencies?
After excluding pecia from the dependencies (without knowing the consequences), I found out that preon brings multiple copies of the log4j.properties file. I suggest moving log4j.properties files to the /src/test/resources directory on the preon and pecia projects to avoid bringing them with the classes.
Because duplicated log4j.properties files, the android-maven-plugin fails at the package goal with the following message:
[INFO] java.util.zip.ZipException: duplicate entry: log4j.properties
[INFO] ------------------------------------------------------------------------
[INFO] BUILD FAILURE
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 19.717s
[INFO] Finished at: Wed Mar 23 14:30:55 PST 2011
[INFO] Final Memory: 7M/62M

Well, I will answer my own question. It is POSSIBLE to use the Preon framework in Android. However, Preon does not work out of the box. I managed to run a sample application after doing the following changes:
1. I moved all log4.properties in the
preon projects to their
corresponding /src/test/resources
directory.
2. Remove dependency on pecia.
3. Embedded the following interfaces from pecia in preon-binding:
DocumentElement.java
Documenter.java
Para.java
ParaContents.java
4. org.codehaus.preon.code.Codecs:
I commented out the following imports and all its related code (the ones that won't compile after this change):
import javax.xml.stream.XMLStreamException;
import nl.flotsam.pecia.builder.ArticleDocument;
import nl.flotsam.pecia.builder.base.DefaultArticleDocument;
import nl.flotsam.pecia.builder.base.DefaultDocumentBuilder;
import nl.flotsam.pecia.builder.html.HtmlDocumentBuilder;
import nl.flotsam.pecia.builder.xml.StreamingXmlWriter;
import nl.flotsam.pecia.builder.xml.XmlWriter;
5. org.codehaus.preon.codec.ObjectCodeFactory
commented out:
/* target.document(codec.getCodecDescriptor().reference(CodecDescriptor.Adjective.THE, false));
*/
Suggestions:
I suggest to refactor preon code to have documentation code separated from runtime dependencies.
Wilfred, If you want, I could contribute to your project.
Oscar.

you also could take a look at Java Binary Block Parser, the library is compatible with Android 2.1+