We Keep Coding

In Java, how do I efficiently trim 0's from the start and the end of a byte array

For reasons outside my control, I need to parse a huge file that has an universe of empty bytes at the beginning and the end of the file, and a very small portion that is actually valid (5 KBs at most). This is the code I came up with:
#NonNull
public static byte[] readFileToByteArray(#NonNull File file, boolean bTrimNulls) throws IOException {
byte[] buffer = new byte[(int) file.length()];
FileInputStream fis = null;
try {
fis = new FileInputStream(file);
if (fis.read(buffer) == -1) {
throw new IOException("EOF reached while trying to read the whole file");
}
} finally {
closeSafely(fis);
}
if (!bTrimNulls) {
return buffer;
}
int nFirstValidByteIndex = 0;
for (int i = 0; i < buffer.length; i++) {
if (buffer[i] != 0) {
nFirstValidByteIndex = i;
break;
}
}
int nLastValidByteIndex = 0;
for (int i = buffer.length - 1; i > 0; i--) {
if (buffer[i] != 0) {
nLastValidByteIndex = i;
break;
}
}
return copyBufferRange(buffer, nFirstValidByteIndex, nLastValidByteIndex + 1);
}
Is there any better alternative to this?
EDIT: The valid bytes in the buffer correspond to an XML file.

I think your solution is rather efficient. In fact, you are looking from both ends of the array the indexes of the first 1's and then create a subarray of data.
Why do you feel you need to improve your algorithm?
Careful: premature optimization is the root of all evil (or at least most of it) in programming, quote by Donald Knuth

Your code has a time complexity of n, which can be too much for large files as you said. Fortunately we know that the non-zero part has a maximum size m, so we can search the file in steps of m. If we miss (hit a zero in the middle of the payload), we need to repeat it until we found it. So the complexity goes to around n/m if the probability of a zero in the payload is sufficiently low.
import java.util.Arrays;
import java.util.Random;
class Test
{
public static int findNonZero(byte[] sparse, int max)
{
// looks quadratic but isn't in practice if the probability of zero in the payload is low, i.e. 1/256 for random values
for(int offset=0;offset<max;offset++)
{
for(int i=0;(i+offset)<sparse.length; i+=max)
{
if(sparse[i+offset]!=0)
{
return i+offset;
}
}
}
// in production code you could handle this differently but this is just an example
throw new RuntimeException("Nonzero value not found");
}
public static byte[] trim(byte[] sparse, int max)
{
int index = findNonZero(sparse, max);
// go to the left and go to the right until you find (max) zeroes
int from = ...
int to = ...
return Arrays.copyOfRange(sparse, from, to);
}
public static void main(String[] args)
{
// create test data
int size = 5000;
byte[] test = new byte[1_000_000_000];
byte[] payload = new byte[size];
Random r = new Random();
r.nextBytes(payload);
payload[0]=(byte)(r.nextInt(Byte.MAX_VALUE-1)+1); // ensure start isnt zero
payload[payload.length-1]=(byte)(r.nextInt(Byte.MAX_VALUE-1)+1); // ensure end isnt zero
System.arraycopy(payload, 0, test, r.nextInt(test.length-size), size);
System.out.println(Arrays.equals(payload,trim(test,size)));
}
}
I left the last part for you, where you need to go to the left and go to the right until you find (max) zeroes and determine from and to indices.
You could further improve the real-world performance by setting subsequent offsets further apart, for example offset_1 = 0, offset_2 = max/2, offset_3 = 1/4 max, offset_4 = 3/4 max and so on.

The code is fine. For really large files one would use a limited buffer, a FileChannel,
a SeekableByteChannel with a ByteBuffer.
Just the code could be a bit nicer. A parameter Path instead of File would be more general and more modern.
public static byte[] readFileToByteArray(#NonNull File file, boolean trimNulls)
throws IOException {
Path path = file.toPath();
byte[] content = Files.readAllBytes(path);
if (trimNulls) {
int start = 0;
while (start < content.length && content[start] == 0) {
++start;
}
int end = content.length;
while (end > start && content[end - 1] == 0) {
--end;
}
content = Arrays.copyOfRange(content, start, end);
}
return content;
}

Received Byte array not always correct

I have an Arduino program that sends via Bluetooth a byte array to a Java program.
The Java program receives the data on JSSC. The array is 72 bytes long (18 floats a´ 4byte).
How JSSC recognizes the end of transmission, or know that a new byte array becomes available?
In most cases, the transfer is correct. But from time to time waste is received.
I've found this example here.
http://www.javaprogrammingforums.com/java-se-api-tutorials/5603-jssc-library-easy-work-serial-ports.html
This
if(event.getEventValue() == 10)
means an linefeed, but bytearray have no linefeeds.
Arduino:
Serial.write(bytearray, size of bytearray);
Java:
class SerialPortReader implements SerialPortEventListener {
public void serialEvent(SerialPortEvent event) {
int dataNumber = 0;
try {
if(event.isRXCHAR()){
if(event.getEventValue() >= BYTE_NUMBER){
receivedByte = usbPort.readBytes(BYTE_NUMBER);
if(receivedByte != null) {
isRequestOK = true;
byte[] myArray = new byte[FLOATSIZE];
for(int i = 0, y = 0; i < receivedByte.length; i++, y++) { //i < receivedByte.length-1
if(myArray == null) {
y = 0;
myArray = new byte[FLOATSIZE];
}
if((i + 1) % FLOATSIZE != 0) {
myArray[y] = receivedByte[i];
}
else {
myArray[y] = receivedByte[i];
receivedValue[dataNumber] = ByteBuffer.wrap(myArray).order(ByteOrder.LITTLE_ENDIAN).getFloat();
myArray = null;
dataNumber++;
}
}
}
}
}
} catch (SerialPortException e) { } ///InvocationTargetException oder NullPointerException
}//--------------------------- End of serialEvent -------------------------------------------------------------
}//--------------------------- End of SerialPortReader ------------------------------------------------------------
Can anybody help me?
Best regards Willi
P.S. This is my first question in this forum. Hopefully I have followed the rules.

Fast Fourier Transform in processing of an input signal from Arduino

I have been trying to get audio signal from an arduino microphone. First I setup my arduino in Free Running Mode, and then I wait for a message from Processing that is a capital A. When Processing sends the serial signal, my arduino should send back 16 audio amplitude samples. Then, processing should take those 16 audio samples and pass them by the Fast Fourier Transform in orden to get a real time frequency spectrum. However, it seems that my Arduino never detects the capital A, and also, if I try to send the samples from my Arduino without waiting for the capital A, when Processing gets the samples it seems something went wrong with the serial communication. I tried with many arduinos and different serial cables to be sure it was no the problem. Here you are the arduino code:
// Arrays to save our results in
int values[16];
// Define various ADC prescaler
const unsigned char PS_16 = (1 << ADPS2);
const unsigned char PS_32 = (1 << ADPS2) | (1 << ADPS0);
const unsigned char PS_64 = (1 << ADPS2) | (1 << ADPS1);
const unsigned char PS_128 = (1 << ADPS2) | (1 << ADPS1) | (1 << ADPS0);
// Setup the serial port and pin 2
void setup() {
Serial.begin(115200);
pinMode(2, INPUT);
// set up the ADC
ADCSRA &= ~PS_128; // remove bits set by Arduino library
// you can choose a prescaler from above.
// PS_16, PS_32, PS_64 or PS_128
ADCSRA |= PS_64; // set our own prescaler to 64
}
void loop() {
unsigned int i;
for(i=0;i<16;i++) {
values[i] = analogRead(2);
}
if(Serial.available() > 0) {
char c = Serial.read();
if(c == 'A') {
for(i=0;i<16;i++) {
Serial.println(values[i]);
}
}
}
delay(10);
}
Here you are the processing code:
import processing.serial.*;
import ddf.minim.analysis.*;
FFT fft;
Serial myPort; // Create object from Serial class
void setup()
{
String portName = Serial.list()[0];
myPort = new Serial(this, portName, 115200);
size(800, 600);
background(255);
fft = new FFT(16, 16000);
}
String mensaje;
float[] magnitudes = new float[16];
int contador = 0;
byte request = true;
void draw()
{
if(request) {
myPort.write('A');//Solicita nuevos datos.
request = false;
}
while(!request) {
if(myPort.available()>0) {
mensaje = myPort.readStringUntil(13);
print(" mensaje: "+mensaje);
if(mensaje != null){
magnitudes[contador++] = int(mensaje);
}
}
if(contador >= 15) {
contador=0;
fft.forward(magnitudes);
request = true;
fill(100);
rect(0,0,800,800);
for (int i = 0; i < 15; i++) {
fill(200);
rect(i*30+10, 0, 10, fft.getBand(i) * 0.1);
}
}
}
}
Thank you a lot for your attention!

Java Proxy Discovering Bot

I have written a class, ProxyFinder which connects to random ips and first pings them, and if they respond, attempts to create a http proxy connection through common proxy ports.
Currently, it is set up just connecting to random ips. This is relatively fast, discovering a few proxys an hour. However, I would like to somehow check if I have already previously connected to an ip. First I tried keeping them in a list, but that was using over 10GB of ram.. I included a method that I tried in the code below which writes the data to a cache using a RandomAccessFile, but this is incredibly slow to search through the entire file for each connection as it gets larger.
I am storing the data in as small of format as possible, simply four bytes for each ip. Even though, this is 4 * 256 * 256 *256 * 256 bytes.. = 16gb of raw ram.. or a 16gb file to search each time you want to test another ip.
I also tried creating a separate thread to generate ips, check them against the file, and then add them to a queue that the probe threads could pull from. It could not keep up with the probe threads either.
How can I quickly check if I have already connected to an IP or not, without being incredibly slow or using ridiculous amounts of memory?
package net;
import java.io.File;
import java.io.RandomAccessFile;
import java.net.HttpURLConnection;
import java.net.InetAddress;
import java.net.InetSocketAddress;
import java.net.Proxy;
import java.net.URL;
import java.util.Arrays;
import java.util.concurrent.atomic.AtomicInteger;
/**
*
* #author Colby
*/
public class ProxyFinder {
/**
* #param args the command line arguments
*/
public static void main(String[] args) throws Exception {
int[] ports = {
1080, 3128, 3128, 8080
};
System.out.println("Starting network probe");
AtomicInteger counter = new AtomicInteger();
for (int i = 0; i < 500; i++) {
new Thread(() -> {
do {
try {
byte[] addrBytes = randomAddress();//could be getNextAddress also
if (addrBytes == null) {
break;
}
InetAddress addr = InetAddress.getByAddress(addrBytes);
if (ping(addr)) {
float percent = (float) ((counter.get() / (256f * 256f * 256f * 256f)) * 100F);
if (counter.incrementAndGet() % 10000 == 0) {
System.out.println("Searching " + percent + "% network search");
}
for (int port : ports) {
try {
Proxy proxy = new Proxy(Proxy.Type.HTTP, new InetSocketAddress(addr, port));
HttpURLConnection con = (HttpURLConnection) new URL("http://google.com").openConnection(proxy);
con.setConnectTimeout(1000);
con.setReadTimeout(1000);
con.setRequestMethod("GET");
con.setRequestProperty("User-Agent", "Mozilla/5.0");
con.getContent();
con.disconnect();
System.out.println("Proxy found!" + addr.getHostAddress() + ":" + port + " Found at " + percent + "% network search");
} catch (Exception e) {
}
}
//
//System.out.println("Ping response: --" + addr.getHostAddress() + "-- Attempt: " + counter.get() + " Percent: " + percent + "%");
} else {
//System.out.println("Ping response failed: " + addr.getHostAddress() + " attempt " + counter.incrementAndGet());
}
} catch (Exception e) {
//e.printStackTrace();
}
} while (true);
}).start();
}
}
private static RandomAccessFile cache;
private static byte[] getNextAddress() throws Exception {
if (cache == null) {
cache = new RandomAccessFile(File.createTempFile("abc", ".tmp"), "rw");
}
byte[] check;
checkFile:
{
byte[] addr = new byte[4];
do {
check = randomAddress();
inner:
{
cache.seek(0);
while (cache.length() - cache.getFilePointer() > 0) {
cache.readFully(addr);
if (Arrays.equals(check, addr)) {
break inner;
}
}
cache.write(check);
break checkFile;
}
} while (true);
}
return check;
}
private static byte[] randomAddress() {
return new byte[]{(byte) (Math.random() * 256), (byte) (Math.random() * 256), (byte) (Math.random() * 256), (byte) (Math.random() * 256)};
}
private static boolean ping(InetAddress addr) throws Exception {
return addr.isReachable(500);
}
}
Also in case anyone is wondering, I've had this running for 12 hours now and it's discovered about 50 proxys, and pinged about 2.09664E-4% of the ip range which is about 1.2 million ips. not bad for the bandwidth allocated (0.5Mbps)
EDIT: I am starting to think that maybe the overhead of storing and checking all of these IPs would be even greater than simply connecting to many duplicates near the end of searching the ip range..

I would not store the whole IP address because of the amount of data. To store them in a array of BitSet would consume less memory.
edit previous code version removed, it was not correct
The version below generates random addresses and persist them in a file. If the persistence file of a previous run is found, the information of the seen addresses is restored from that file.
Following case was not handled correctly in the initial version:
assuming that no address was already seen
1.0.0.1 - seen false
2.0.0.2 - seen false
2.0.0.1 - seen true, which was wrong and is correctly handled by code below
See the comments in the code for further information.
public class KeepSeenAddresses {
static final int FILE_BUFFER_SIZE = 81_920;
static final int RANGES_SIZE = 256;
// to store 256 ranges of 255*255*255+1 addresses
static BitSet[] ranges;
// Random(1) is taken only for demonstration purpose, so the second
// application run will find the same seen addresses from previous run
static Random random = new Random(1);
// for normal use it's better to have better randomness
//static Random random = new Random(System.currentTimeMillis());
public static void main(String[] args)
throws IOException, ClassNotFoundException {
if (!readRanges()) {
initRanges();
}
// this case was failing in the initial solution
// uncomment this block to see how all edge cases
// which where mentioned in other comments are handled
/*
byte[][] addresses = {
{1, 0, 0, 1},
{2, 0, 0, 2},
{2, 0, 0, 1},
{1, 2, 3, 4},
{4, 3, 2, 1},
{(byte)128, 0, 0, 0},
{(byte)255, (byte)255, (byte)255, (byte)255}
};
seenAddress(addresses[0]);
seenAddress(addresses[1]);
seenAddress(addresses[3]);
seenAddress(addresses[5]);
seenAddress(addresses[6]);
for (byte[] addressBytes : addresses) {
System.out.printf("seen %s before: %s%n",
prettyAddress(addressBytes),
seenBefore(addressBytes)
);
}
*/
processAddresses();
persistRanges();
}
/**
* Read the seen addresses from a file.
*
* #return <code>true</code> if the file was found and has the expected
* number of ranges, otherwise <code>false</code>
* #throws IOException
* #throws ClassNotFoundException
*/
private static boolean readRanges() throws IOException, ClassNotFoundException {
File rangesStore = new File("addresses.bin");
if (!rangesStore.exists()) {
return false;
}
System.out.print("found previous rangesStore... ");
try (ObjectInputStream ois = new ObjectInputStream(
new BufferedInputStream(
new FileInputStream(rangesStore), FILE_BUFFER_SIZE
)
)) {
ranges = (BitSet[]) ois.readObject();
}
if (ranges.length != RANGES_SIZE) {
System.out.printf("wrong size of rangesStore: expected %d"
+ " found: %d%n", RANGES_SIZE, ranges.length);
return false;
} else {
System.out.printf("restored ranges: %d%n", ranges.length);
return true;
}
}
/**
* Initialize the address ranges array. All address flags will be set to
* <code>false</code>.
*/
private static void initRanges() {
System.out.print("initialize new rangesStore... ");
ranges = new BitSet[RANGES_SIZE];
for (int i = 0; i < RANGES_SIZE; i++) {
BitSet bitSet = new BitSet(255 * 255 * 255 + 1);
for (int j = 0; j < 255 * 255 * 255 + 1; j++) {
bitSet.clear(j);
}
ranges[i] = bitSet;
}
System.out.printf("initialized ranges: %d%n", RANGES_SIZE);
}
/**
* For demonstration purpose.<br>
* Generates some random IPv4 addresses. If the address was not seen before
* the flag for this address will be set to <code>true</code>.
*/
private static void processAddresses() {
for (int i = 0; i < 10; i++) {
byte[] addrBytes = randomAddress();
boolean seenBefore = seenBefore(addrBytes);
if (!seenBefore) {
seenAddress(addrBytes);
seenBefore = false;
}
System.out.printf("seen %s before: %s%n",
prettyAddress(addrBytes),
seenBefore
);
}
}
/**
* Persist the address ranges array. The file size is around 500MB.
*
* #throws IOException
*/
private static void persistRanges() throws IOException {
System.out.print("persist rangesStore... ");
try (ObjectOutputStream oos = new ObjectOutputStream(
new BufferedOutputStream(
new FileOutputStream("addresses.bin"), FILE_BUFFER_SIZE)
)) {
oos.writeObject(ranges);
}
System.out.printf("written ranges: %d%n", ranges.length);
}
/**
* Keep a flag which address has been seen already.
*
* #param addrBytes IPv4 address in four bytes
*/
static void seenAddress(byte[] addrBytes) {
int rangeIndex = (int) addrBytes[0] & 0xff;
int rangeOffset = ((int) addrBytes[1] & 0xff * 0xffff)
+ ((int) addrBytes[2] & 0xff * 0xff)
+ ((int) addrBytes[3] & 0xff);
ranges[rangeIndex].set(rangeOffset);
}
/**
* Check if the passed address was seen before.
*
* #param addrBytes IPv4 address in four bytes
* #return <code>true</code> if the address was seen before, otherwise
* <code>false</code>
*/
static boolean seenBefore(byte[] addrBytes) {
int rangeIndex = (int) addrBytes[0] & 0xff;
int rangeOffset = ((int) addrBytes[1] & 0xff * 0xffff) + ((int) addrBytes[2] & 0xff * 0xff) + ((int) addrBytes[3] & 0xff);
return ranges[rangeIndex].get(rangeOffset);
}
/**
* Convert the IPv4 address into pretty string.
*
* #param addrBytes IPv4 address in four bytes
* #return pretty String of the IPv4 address
*/
static String prettyAddress(byte[] addrBytes) {
return String.format("%03d.%03d.%03d.%03d",
(int) addrBytes[0] & 0xff,
(int) addrBytes[1] & 0xff,
(int) addrBytes[2] & 0xff,
(int) addrBytes[3] & 0xff);
}
/**
* Generate a random IPv4 address.
*
* #return four bytes of a random generated IPv4 address
*/
private static byte[] randomAddress() {
byte[] bytes = new byte[4];
for (int i = 0; i < bytes.length; i++) {
bytes[i] = (byte) random.nextInt(256);
}
return bytes;
}
}

I have ported code from another solution here to fit this problem:
Java- Mapping multi-dimensional arrays to single
The answer to the above question gives an in depth explanation of how the following code works. If anyone else would like to post a more in depth answer on this thread I will award it the answer.
static BitSet set;
static int pos(int i, int j, int k, int m) {
return ((256*256*256) * i) + ((256*256) * j) + (256 * k) + m;
}
static boolean get(byte[] addr) {
return set.get(pos(addr[0], addr[1], addr[2], addr[3]));
}
static void set(byte[] addr, boolean flag) {
set.set(pos(addr[0], addr[1], addr[2], addr[3]), flag);
}

Use a data base like MySql and hibernarte with level 1 & 2 cache.
It will be faster than RAM if u configure a cache with hibernate and tune ur db to use few gb of cache too. i think they all do. can configure an external cache like ehcahe when an be configured to live on another process + file with limits on size and time. Db knows how to index and seek things faster than even pure RAM - at such sizes as your IP
Plus you can improve by partitioning table data & index by first char, 2nd char etc

Reading specific bytes from RandomAccessFile and testing to see if they equal 0

first time poster here. Thank you in advance for viewing my question. I'm having a ton of trouble with a homework problem in which I have to read a specific range of bytes from a RandomAccessFile, then check that range of bytes to see if they all equal 0. I've looked all over for something that pertains to this, but nothing I've found quite hits the spot. Any help provided will be appreciated.
The problem tells us that there is a certain file that contains data for hypothetical students in a school. Each of these students is represented in 40 bytes of code, but the first four bytes of our file must be an integer with the total number of students in the school(let's say there are 75). Bytes 4 through 43 represent the first student (#0), 44 through 83 represent the second (#1) and so on. When a student transfers to another school, their 40 bytes is overwritten with all 0's (characters).
I've written a method called "transferStudent" that takes a String which represents the file name and the integer which represents the number of students. If there are any exceptions or if the file doesn't overwrite the student's data for some reason, I return false;
Here is my work thus far:
public static Boolean transferStudent(String fileName, int studentNum) {
RandomAccessFile file = new RandomAccessFile(fileName, "rw");
file.writeInt(75);
try {
if (studentNum == 0) {
file.seek(4);
file.writeBytes("0000000000000000000000000000000000000000"); // 40 zero characters
file.seek(4);
for (int i = 0; i < 40; i++) {
if (file.read() == 0) {
return true;
}
}
return false;
}
else if (studentNum > 0) {
file.seek(4 + (studentNum * 40));
file.writeBytes("0000000000000000000000000000000000000000"); // 40 more zeroes
file.seek(4);
for (int i = (4 + (studentNum * 40)); i < (44 + (studentNum * 40)); i++) {
if (file.read() == 0) {
return true;
}
}
return false;
}
else {
return false;
}
}
catch (Exception e) {
return false;
}
}
Whenever I view the binary file that's been created, there are indeed 0's in the range that corresponds with the studentNum. However, the console always prints false - the check isn't working for some reason. I'm on the verge of tearing my hair out over this. Please help!

You're confusing ASCII zeroes "0" with binary zeros. You're writing the former and testing for the latter. An ASCII "0" takes two bytes. Note that 'character' and 'byte' aren't the same in Java.

So I think I've finally figured out the issue: Like EJP stated, I was confusing the ASCII zeroes "0" with binary zeros. As stated, ASCII zeroes take up two bytes of information- This was, and really still is, confusing to me: I view the file that is written, but it appears that only one byte of information is used to write each "0". I'll have to do more research on this topic. Aside from that though, there was another issue with my code- every time I ran the program, the file would receive the zero-characters written to it. There was no issue with that, but there was a second issue with the check - I wasn't doing anything to further advance the file pointer when using the loop for a check.
So there were two things that needed to be done to fix my code:
Firstly, I had to find a way to advance the file pointer so that each spot in my RandomAccessFile was being read correctly.
Secondly, I had to check for the appropriate value when initiating my check: This value should have been "48", which is the ASCII value for the character "0".
Here is my new code:
public static boolean transferStudent(String fileName, int studentNum) throws IOException {
RandomAccessFile file = new RandomAccessFile(fileName, "rw");
boolean trueOrFalse = false;
file.writeInt(75);
try {
if (studentNum == 0) {
file.seek(4);
file.writeBytes("0000000000000000000000000000000000000000"); // 40 zero characters
file.seek(4);
for (int i = 0; i < 40; i++) {
file.seek(4 + i); // Here is where the file pointer is advanced in the for-loop - very crucial
if (file.read() == 48) { // Here is where the file is checked for the appropriate value - the ASCII value for "0"
trueOrFalse = true;
}
}
return trueOrFalse;
}
else if (studentNum > 0) {
file.seek(4 + (studentNum * 40));
file.writeBytes("0000000000000000000000000000000000000000"); // 40 more zeroes
file.seek(4 + (studentNum * 40));
for (int i = 0; i < 40; i++) { // The same happens here as above
file.seek((4 + (studentNum * 40)) + i); // ... and here also
if (file.read() == 48) {
trueOrFalse = true;
}
}
return trueOrFalse;
}
else {
return trueOrFalse;
}
}
catch (Exception e) {
return false;
}
}