We Keep Coding

Understanding Maven dependencies and assembly

I am not very much experienced with Maven and it's compilation and packaging logic gets me confused.
I have some dependencies declares as :
<dependency>
<groupId>com.dependency_group</groupId>
<artifactId>dependency_1</artifactId>
<version>1.0.0</version>
</dependency>
<dependency>
<groupId>com.dependency_group</groupId>
<artifactId>dependency_2</artifactId>
<version>1.0.0</version>
<scope>provided</scope>
</dependency>
So as far as I understand, dependency_1 will be added to the classpath of my program as something that comes along with my jar, and dependency_2 on the other hand, will be added to the classpath as something that the system runtime will provide upon deployment.
Then I run the package goal of Maven and none of my dependencies are packed with my code (I am using the shade plugin, but even without it nothing changes).
I expected that when some dependency is set as compile scope, it will be exported with my compiled code, since AFAICS, there's no point in setting the classpath saying a dependency will come along with my code, and Maven just don't package that dependency with it. It looks to me as if Maven is not obeying it's contract.
So:
1 - What's the logic behind this?
2 - Do I have to always use the Assembly plugin?
3 - Are there cases where people will define a dependency as compile and will not want it packaged within a jar?

Let me shed some light on the main point here. There are fundamentally two kinds of java artifacts:
Applications, i.e. ears, wars, executable jars
Libraries, i.e. jars that are meant to be used as dependencies for other artifacts.
For Applications your reasoning makes perfectly sense. Wars and Ears automatically package all their compile dependencies and you need no assembly plugin for that. For libraries, you do not pack the dependencies into the library. Maven handles transitive dependency resolution and would be confused if you put a fat jar on the classpath.
The thing is that packaging jar can be both a libary or an application. If you want a standalone application, you need to tell Maven to package everything, e.g. by using the assembly plugin or shade plugin.

You use compile scope when you want some dependencies to come along with your code. For example you want Jackson to be a part of your application if you are using it for json serialization.
You use provided scope, if you want dependency to be on the classpath during the compilation but wont be included within your application. It must be provided by running environment. For example you want Lombok as it is compile only library, or you want to have Servlet Api dependency as provided when you are writing servlet application because such app will be ran on servlet container thus there is no need to pack it within your application (it will be available in container runtime)
Do I have to always use the Assembly plugin
Nobody forces you to do so.

clarification of the term "is on the classpath"

When the phrase "is on the classpath" is used in boot documentation, can I get a precise meaning for this for a linux person building a spring boot maven project?
it is not clear about the context in time: is it talking about compile or run time or repackage time or what?
does it mean that a jar/manifest is in a subdirectory of the current project directory such as the target/...classes
does it mean that the jar is in the pom
for a newbie, I read the wikipedia and oracle explanations but this seems to mean something different or it means that i should be born with the knowledge of some sort of modifications to the classpath made by maven during the build process.
or what about during run time for a repackaged jar?
Is there anybody who would have mercy on a newbie and help me on the term?

I try to answer your questions:
You can have a compile classpath as well a runtime classpath (Maven also has a test classpath). They can be different, so it depends on the context, which one is meant.
Technically it doesn't matter, what the directory structure is, since you can add every directory on your disk to your classpath.
If you use Maven, then only JARs, that are defined in the POM, are automatically added to the compile classpath (which is also used for packaging).
Maven uses different classpaths during the build. E.g. for compiling it uses all dependencies in provided or compile scope and adds them to the classpath. For testing, additional dependencies in the test scope are added to the classpath to compile and execute tests.

Compile, Provided, APK - Android dependency scope

While adding new dependencies to android project especially in Android Studio in Dependencies there are three scope options Compile/Provided/APK.
What are the effects of choosing each one, when should we use them ? Besides what the name says.
EDIT:
"Properly handle 'provided' and 'package' scopes to do what they should be doing.
'provided' and 'package' cannot be used with Android Libraries, and will generate an error" .. this is from http://tools.android.com/tech-docs/new-build-system

provided - compile-time only dependency
package - package-time only dependency
compile - compile-time and package-time dependency
provided is commonly used for annotation processing based libraries. Usually these libraries are separated in two artifacts - "annotation" and "compiler". "compiler" is provided dependency because you do not need to use it in application, only for compilation; and "annotation" is compile dependency - it is used in application code and therefore compiles. Or generated code may require additional dependencies while your application may not. E.g. dagger dependencies configuration:
compile 'com.squareup.dagger:dagger:1.2.2'
provided 'com.squareup.dagger:dagger-compiler:1.2.2'

These properties come from maven scopes.
They simply indicate how to treat particular dependencies during each step of build process.
compile - a default approach, it simply means that all dependencies should be available at compile-time. Compile dependencies are available in all classpaths of a project. Furthermore, those dependencies are propagated to dependent projects. A compile-time dependency is generally required at runtime.
package - declares additional configuration for building an application. You can list plugins that add additional functionality to the build process.
provided - it means that runtime environment has these dependencies included. For example, when you look into android.jar library internals you will see java.lang.RuntimeException: Stub! in every method body.
That has some consequences:
You can develop Android applications locally, without having complete Android environment.
Your APK you must run it on an android device or an emulator because they provide implementation of these methods.
Your apps that reference the SDK classes will build properly, because the jar provides the class metadata.
Unless you use some library that provides artifacts (e.g. Robolectric), you have to run tests on your emulator/device.
provided and package cannot be used with Android Libraries, and will generate an error.
Here is how sourceSet looks like:
More info about build system: https://www.youtube.com/watch?v=LCJAgPkpmR0
An awesome article about Gradle: http://www.sinking.in/blog/provided-scope-in-gradle/

Xavier talks here about the APK scope.
in the Android plugin, The equivalent (sort of) of runtime is called apk. You can do
dependencies {
apk files('libs/foo.jar')
}
and it'll only get packaged but won't be on the compile classpath.

With gradle 6.5.1 provided gives the below error
Could not find method provided() for arguments [directory '....'] on object of type org.gradle.api.internal.artifacts.dsl.dependencies.DefaultDependencyHandler.
I used compileOnly to use the dependencies at only compile time and not to inlcude in the final build artifact

Gradle - what is "runtime" dependency configuration used for?

Can you please help me understand what are the typical use-cases in which one would use the runtime dependency configuration (provided by the Java plugin)?
In the Gradle user-guide, Table 23.5. Java plugin - dependency configurations, I can see that the runtime configuration is not used by any tasks - as opposed to e.g. the compile configuration which is used by the compileJava task.
What then is the runtime dependency useful for?

The runtime configuration is for libraries that are needed at runtime but NOT at compile time (For example JDBC drivers and SLF4J api implementations).
You could just add them to your compile configuration, but then they would be on the compile classpath and you would run the risk of accidentally introducing a compile dependency on something from the implementation rather than the api.
It is not for libraries that are 'provided' by a container - it is actually how you provide libraries to your app while making sure you haven't introduced a compile depencency on them.

What is the best way to declare transient dependencies between compile and runtime with Eclipse?

We are looking to revamp our Java build process on Eclipse. Currently we use Gradle! As a part of that effort we are looking at whether we are using Gradle in the best possible way. We use the Eclipse plugin for Gradle and declare our dependencies with compile. Unfortunately this adds a ton of transient dependencies with our generated Eclipse projects which is not desirable. These extra dependencies are only valid at runtime.
So, is there a way to declare a dependency ONCE in Gradle and have its compile dependency set to the first level of dependency and its runtime dependency set to the first level plus transient dependencies?
Currently we use the #jar syntax with compile which gives us the first level dependency for compile, but we still have to declare that dependency again for runtime. Not ideal because we have two places to go in order to update a dependency.
Is there a better way?

I assume you mean transitive dependencies.
If you only want direct dependencies to appear on Gradle's compile class path, your current solution seems reasonable. (In the future we want to have a more accurate compile class path out-of-the-box.) Potential improvements are to make the compile configuration non-transitive (rather than using #jar) or to write a plugin that provides a custom dependency syntax, thereby making the duplication between compile and runtime dependencies go away.
However, this won't help you with Eclipse, as Eclipse has no notion of separate compile and runtime class paths. The only way out is to make run configurations responsible for providing the runtime class path, but this might be difficult to set up (e.g. run configurations are often created on-the-fly), and Gradle doesn't have any out-of-the-box support for this. You could use the generic XML hooks of Gradle's Eclipse plugin to generate run configurations, but I'm not sure if the Eclipse Gradle integration would pick them up.
Due to these difficulties, most Eclipse developers put all runtime dependencies on the Eclipse project class path (regardless of whether they are using Gradle or not), despite the obvious disadvantages.
Our vision is that Gradle can one day act as the build engine for Eclipse. Once this happens, discrepancies between IDE and build tool (class paths, code generation, integration testing, etc.) will be a thing of the past. Gradle can already act as the build engine for NetBeans today, and soon also for IntelliJ IDEA (driven by the requirements for Android Studio).

While I certainly agree with #Peter that an overhaul of the Eclipse build process is a long-term goal, in the short term you may be able to accomplish what you want using Maven and m2eclipse. Whether the tradeoffs of moving to Maven are worth that extra control are entirely up to you.