In my experience, recompiling the source code for the same Android application won't produce the same binary every time. A reproducible build is very useful for developers, but I think the most important benefit from a reproducible build process is security. In open source Android apps, how we can verify produced binary (.apk) is really compiled from reviewed source code? Is there any way to generate reproducible builds from Android SDK or Java?
"Reproducible"/"deterministic" build refers to build processes where the compiler outputs a binary that is always identical, given identical input files/build system/chroot. (The article you linked to is actually talking about being able to build/run the same version of an app that a bug is filed against, which is something different. See this description of how Tor makes deterministic builds.)
Java packages seem to be very difficult to work with, since timestamps are present in many, many places. One thing you could try, in lieu of exact binary compatibility, is to decompile the two .apk's, and see if that decompiled output is identical.
The F-Droid project has been working on reproducible builds for Android for a couple years now. The Android-specific issues are tracked at https://f-droid.org/docs/Reproducible_Builds It is still a tricky process, especially if the app includes NDK code. If the app is only Java, then there are a few relatively easy steps that will get you there, like pre-crunching PNGs and committing them to git.
Google doesn't make it easy to reproduce APKs in the longer term, since they do a lot of random little changes, like:
adding a non-reproducible version to AndroidManifest.xml
different binary releases with the same version number
Definitely check out diffoscope for viewing the differences between two builds. You can see lots of example output at https://verification.f-droid.org
I just found that Telegram claims experimental reproducible builds for their Android app: https://core.telegram.org/reproducible-builds so it might be worth to study how they are achieving it.
The procedure uses Docker to build, and at the end compares the APKs with:
python apkdiff.py telegram_store.apk telegram_built.apk
which suggests they are not byte-by-byte the same.
apkdiff.py is present in-tree at: https://github.com/DrKLO/Telegram/blob/e1c101c334c80387cf10ca9857052e70e19c60af/apkdiff.py
Related
I came upon this simple Library that someone wrote in java GetImageText.java for OCR in images so i tried compiling it on my Ubuntu via terminal but i get several error as shown below in this paste :
Compilation Errors
Can Someone help me with it , it is absolute necessity that i test this code , its explanation can be found here
I think the problem is that i do not have com.sun.image.code.jpeg in my system, although java is definitely installed. Although I am not sure how to import this package without using an IDE.
The problem is that the library you are trying to recompile depends on INTERNAL classes1. Portable libraries are not supposed to do that!
What has happened is that the class has been removed or replaced. This happens from time to time, and that is the reason that people are not supposed to write code that depends on INTERNAL classes.
Solutions:
Bug the authors of the library to fix the problem.
Figure out which version(s) of Java that the library supports, and stick with those.
Find an alternative library that supports the version(s) of Java that you need.
Non-solution: Compiling the library on an older version of Java and running on a newer one is likely to fail. The class needs to be present at runtime, as well as at compile time.
1 - Anything in the "com.sun" tree counts as INTERNAL. Sometimes people have no choice but to have such a dependency. However, they still needs to deal with the potential consequences for portability.
For past 4 years, I have been programming with Eclipse (for Java), and Visual Studio Express (for C#). The IDEs mentioned always seemed to provide every facility a programmer might ask for (related to programming, of course).
Lately I have been hearing about something called "build tools". I heard they're used almost in all kind of real world development. What are they exactly? What problems are they designed to solve? How come I never needed them in past four years? Are they kind of command-line stripped down IDEs?
What are build tools?
Build tools are programs that automate the creation of executable
applications from source code (e.g., .apk for an Android app). Building
incorporates compiling,linking and packaging the code into a usable or
executable form.
Basically build automation is the act of scripting or automating a
wide variety of tasks that software developers do in their day-to-day
activities like:
Downloading dependencies.
Compiling source code into binary code.
Packaging that binary code.
Running tests.
Deployment to production systems.
Why do we use build tools or build automation?
In small projects, developers will often manually invoke the build
process. This is not practical for larger projects, where it is very
hard to keep track of what needs to be built, in what sequence and
what dependencies there are in the building process. Using an
automation tool allows the build process to be more consistent.
Various build tools available(Naming only few):
For java - Ant,Maven,Gradle.
For .NET framework - NAnt
c# - MsBuild.
For further reading you can refer following links:
1.Build automation
2.List of build automation software
Thanks.
Build tools are tools to manage and organize your builds, and are very important in environments where there are many projects, especially if they are inter-connected. They serve to make sure that where various people are working on various projects, they don't break anything. And to make sure that when you make your changes, they don't break anything either.
The reason you have not heard of them before is that you have not been working in a commercial environment before. There is a whole lot of stuff that you have probably not encountered that you will within a commercial environments, especially if you work in software houses.
As others have said, you have been using them, however, you have not had to consider them, because you have probably been working in a different way to the usual commercial way of working.
Build tools are usually run on the command line, either inside an IDE or completely separate from it.
The idea is to separate the work of compiling and packaging your code from creation, debugging, etc.
A build tool can be run on the command or inside an IDE, both triggered by you. They can also be used by continuous integration tools after checking your code out of a repository and onto a clean build machine.
make was an early command tool used in *nix environments for building C/C++.
As a Java developer, the most popular build tools are Ant and Maven. Both can be run in IDEs like IntelliJ or Eclipse or NetBeans. They can also be used by continuous integration tools like Cruise Control or Hudson.
Build tools are generally to transform source code into binaries - it organize source code, set compile flags, manage dependencies... some of them also integrate with running unit test, doing static analysis, a generating documentation.
Eclipse or Visual Studio are also build systems (but more of an IDE), and for visual studio it is the underlying msbuild to parse visual studio project files under the hood.
The origin of all build systems seems like the famous 'make'.
There are build systems for different languages:
C++: make, cmake, premake
Java: ant+ivy, maven, gradle
C#: msbuild
Usually, build systems either using a propriety domain specific language (make, cmake), or xml (ant, maven, msbuild) to specify a build. The current trend is using a real scripting language to write build script, like lua for premake, and groovy for gradle, the advantage of using a scripting is it is much more flexible, and also allows you the to come up with a set of standard APIs(as build DSL).
These are different types of processes by which you can get your builds done.
1. Continuous Integration build: In this mainly developers check-in their code and right after their check-in a build initiates for building of the recent changes so we should know whether the changes done by the developer has worked or not right after the check-in is done. This is preferred for smaller projects or components of the projects. In case where multiple teams are associated with the project or there are a large no. of developers working on the same project this scenario becomes difficult to handle as if there are 'n' no. of check-in’s and the build fails at certain points it becomes highly difficult to trace whether all the breakage has occurred because of one issue or with multiple issues so if the older issues are not addressed properly than it becomes very difficult to trace down the later defects that occurred after that change. The main benefit of these builds is that we get to know whether a particular check-in is successful or not.
2. Gated check-in builds: In this type of check in a build is initiated right after the check in is done keeping the changes in a shelve sets. In this case if the build succeeds than the shelve-set check-in gets committed otherwise it will not be committed to the Team Foundation Server. This gives a slightly better picture from the continuous integration build as only the successful check-in's are allowed to get committed.
3. Nightly builds: This is also referred as Scheduled builds. In this case we schedule the builds to run for a specific time in order to build the changes. All the previous uncommitted changes from the last build are built during this build process. This is practiced when we want to check in multiple times but do not want a build every time we check in our code so we can have a fixed time or period in which we can initiate the build for building of the checked-in code.
The more details about these builds can be found at the below location.
Gated-check in Builds
Continuous Integration Builds
Nightly Builds
Build Process is a Process of compiling your source code for any errors using some build tools and creating builds(which are executable versions of the project). We(mainly developers) do some modifications in the source code and check-in that code for the build process to happen. After the build process it gives two results :
1. Either build PASSES and you get an executable version of your project(Build is ready).
2. It fails and you get certain errors and build is not created.
There are different types of build process like :
1. Nightly Build
2. gated Build
3. Continuous integration build etc.
Build tools help and automates the process of creating builds.
*So in Short Build is a Version of Software in pre-release format used by the Developer or Development team to gain confidence for the final result of their Product by continuously monitoring their Product and solving any issues early during the development process.*
You have been using them - IDE is a build tool. For the command line you can use things like make.
People use command line tools for things like a nightly build - so in the morning with a hangover the programmer has realised that the code that he has been fiddling with with the latest builds of the libraries does not work!
"...it is very hard to keep track of what needs to be built" - Build tools does not help with that all. You need to know what you want to build. (Quoted from Ritesh Gun's answer)
"I heard they're used almost in all kind of real-world development" - For some reason, software developers like to work in large companies. They seem to have more unclear work directives for every individual working there.
"How come I never needed them in past four years". Probably because you are a skilled programmer.
Pseudo, meta. I think build tools do not provide any really real benefit at all. It is just there to add a sense of security arising from bad company practices, lack of direction - bad software architectural leadership leading to bad actual knowledge of the project. You should never have to use build tools(for testing) in your project. To do random testing with a lack of knowledge of the software project does not give any sort of help at all.
You should never ever add something to a project without knowing it's purpose, and how it will work with the other components. Components can be functional separate, but not work together. (This is the responsibility of the software architect I assume).
What if 4-5 components are added into the project. You add a 6th component. Together with the first added component, it might screw up everything. No automatic would help to detect that.
There is no shortcut other than to think think think.
Then there is the auto download from repositories. Why would you ever want to do that? You need to know what you download, what you add to the project. How do you detect changes in versions of the repositories? You need to know. You can't "auto" anything.
What if we were to test bicycles and baby transports blindfolded with a stick and just randomly hit around with it. That seems to be the idea of build tool testing.
I'm sorry there are no shortcut
https://en.wikipedia.org/wiki/Scientific_method
and
https://en.wikipedia.org/wiki/Analysis
i am using Codename.one for mobile apps development, once the build is submitted, they provide the "apk" file. Since there are some limitation in codename one, so i was wondering that is there a way to get the native code for android build. ??
You can get the sources using the include source feature (available for the basic 9 USD subscription) but you would be mistaken to use it for the case described.
You can extend Codename One in any way you want including with native code, there are quite a few cn1lib's most of which are open source which demonstrate rather easily how you can write native code in Codename One.
This is a superior approach since it keeps the portability of Codename One and easy development cycle while allowing you to do almost anything you need.
Notice you can look at the full source code of Codename One itself in its open source project page.
Greets!
I am developing (trying to develop) a VoIP SIP application for Android, and after two weeks of bickering with mjsip, pjsip and the sdk's libraries, I have settled on JAIN-SIP. The libraries look great, in theory. No need to learn any android native code, lots of documentation (yet not enough, since I'm here), etc.
My first attempt was using the Android SDK's SIP libs (yes, I know they're based on JSIP), and it failed after the SipManager.open() refused to open the profile for communication with the server, with no exceptions or any sort of output to help me diagnose the issue. (If you have a potential solution please look here) After dicking around with mjsip and pjsip I decided that, being on a schedule, it would take too much time to figure out the native code, so JAIN seemed like the perfect solution... or is it?
Problem is- the JAIN SIP libraries use the same name, in the packaging, as java's core libraries, therefore causing a dex issue with dalvik. But ok- theres an "ant make all-android" solution to build the sip stack source into a single .jar prepared for android. WRONG. Only the "gov.nist" packages get refactored, leaving the "javax" packages as they were before, causing the same dalvik dex problems. But wait!- theres JarJar, a tool that repackages your jar and refactors the packages, resolving the dex conflict. WRONG. Repackaging using JarJar also renames the references to the actual javax.net packages that the stack requires for encryption and whatnot, causing exceptions and crashes and all sorts of issues. So I spent the majority of yesterday refactoring the javax libraries by hand, which led to
...Caused by: java.lang.Error: Unresolved compilation problem:
So here are my questions...
Has anyone come across any android-ready jain-sip libraries that really work out of the box, without the need to rename,repackage, and refactor?
Maybe my JarJar rules are wrong? Can someone provide me with rules to rename all the packages and references to "javax.sip" and "javax.sdp", but NOT "javax.net"?
Are there any other SIP libs for android (that dont require any knowledge of the NDK or native code)?
Why is this still such a pain in the back in 2014?
P.S. Before pointing me to a google page or other stackoverflow questions, consider that I've been stuck on this for two weeks now, doing nothing else but trying to solve the issue- I can assure you, the internet has been thoroughly searched and the solutions either don't apply or don't work.
Edit May 2016 - The accepted answer is now out of date, refer to thejoelpatrol's response below for the newer solution. HERE
You can find Android specific JAIN SIP packages renamed to com.telestax here https://mobicents.ci.cloudbees.com/job/jain-sip/ (android-jain-sip-ri jar) that we have built at TeleStax to overcome the native JAIN SIP Android issues.
We are working on creating a Mobile SDKs that integrate WebRTC media as well so it's easier to implement messaging and video calls in native Android Apps
We will create a blog post in the coming days on how to set things up. I'll update the thread here once this is done
The accepted answer is out of date. The official builds now include a version suitable for Android, linked from the homepage: https://jsip.java.net/
I'm using them now and can confirm that they work fine.
There are a few extra gotchas but it's not hard to make your own build. It's really just 3-4 Eclipse commands away. Since there is so much demand I will make a proper script to do it, but I will aim for end of the month unless it's super urgent.
Closed. This question is off-topic. It is not currently accepting answers.
Want to improve this question? Update the question so it's on-topic for Stack Overflow.
Closed 10 years ago.
Improve this question
I'm doing a study on large Java projects and would like to view the source code for Eclipse. I have gone to this url (http://wiki.eclipse.org/index.php/CVS_Howto) and figured that the most useful cvs repository for me to look at would be this one:
:pserver:anonymous#dev.eclipse.org:/cvsroot/eclipse (The Eclipse platform project)
However, when looking at this repository, it has so many modules! Which modules should I be trying to check out? I don't necessarily want to build the IDE from source, however, I just want to get the core Eclipse code base to perform some analysis. Would I just check out any modules starting with "org.eclipse..."? Should I be checking out any of the others?
Or is there an easier way to get the source? I read somewhere that you can get the source from the binary version of Eclipse but I am unsure where to find the source.
Just download the source tarball eclipse-cvs.tgz from here
EDIT: This also includes version history, so it may be larger than you need. For just a current version download Platform-SDK.3.5.2 ~ 100MB
It includes the sources in jar files e.g.
org.eclipse.ui.workbench_3.5.2.M20100113-0800.jar
org.eclipse.ui.workbench.source_3.5.2.M20100113-0800.jar
For empirical studies, a good resource to check out is the Qualitas Corpus. It's a collection of open-source Java programs for use in empirical studies. In addition to having source code for a larger number of major projects (including Eclipse) they have multiple versions of each program, so you can track how code changes over time, if that's important. It's a respected corpus in the Software Engineering/Programming Languages research community, and so it may make your work more suitable for comparison in the future.
I'm not a part of their project, but I have used their corpus in a recent study. If you shoot them an email and tell them what your project is about, they'll give you http download access.
I can only answer this bit:
Or is there an easier way to get the
source? I read somewhere that you can
get the source from the binary version
of Eclipse but I am unsure where to
find the source.
In eclipse/plugins/, the jars named *.source_*.jar contain source. For example, in my install:
org.eclipse.osgi_3.4.0.v20080605-1900.jar
org.eclipse.osgi.source_3.4.0.v20080605-1900.jar
The first jar contains the OSGi runtime, and the latter contains the corresponding source.