I have written a class, ProxyFinder which connects to random ips and first pings them, and if they respond, attempts to create a http proxy connection through common proxy ports.
Currently, it is set up just connecting to random ips. This is relatively fast, discovering a few proxys an hour. However, I would like to somehow check if I have already previously connected to an ip. First I tried keeping them in a list, but that was using over 10GB of ram.. I included a method that I tried in the code below which writes the data to a cache using a RandomAccessFile, but this is incredibly slow to search through the entire file for each connection as it gets larger.
I am storing the data in as small of format as possible, simply four bytes for each ip. Even though, this is 4 * 256 * 256 *256 * 256 bytes.. = 16gb of raw ram.. or a 16gb file to search each time you want to test another ip.
I also tried creating a separate thread to generate ips, check them against the file, and then add them to a queue that the probe threads could pull from. It could not keep up with the probe threads either.
How can I quickly check if I have already connected to an IP or not, without being incredibly slow or using ridiculous amounts of memory?
package net;
import java.io.File;
import java.io.RandomAccessFile;
import java.net.HttpURLConnection;
import java.net.InetAddress;
import java.net.InetSocketAddress;
import java.net.Proxy;
import java.net.URL;
import java.util.Arrays;
import java.util.concurrent.atomic.AtomicInteger;
/**
*
* #author Colby
*/
public class ProxyFinder {
/**
* #param args the command line arguments
*/
public static void main(String[] args) throws Exception {
int[] ports = {
1080, 3128, 3128, 8080
};
System.out.println("Starting network probe");
AtomicInteger counter = new AtomicInteger();
for (int i = 0; i < 500; i++) {
new Thread(() -> {
do {
try {
byte[] addrBytes = randomAddress();//could be getNextAddress also
if (addrBytes == null) {
break;
}
InetAddress addr = InetAddress.getByAddress(addrBytes);
if (ping(addr)) {
float percent = (float) ((counter.get() / (256f * 256f * 256f * 256f)) * 100F);
if (counter.incrementAndGet() % 10000 == 0) {
System.out.println("Searching " + percent + "% network search");
}
for (int port : ports) {
try {
Proxy proxy = new Proxy(Proxy.Type.HTTP, new InetSocketAddress(addr, port));
HttpURLConnection con = (HttpURLConnection) new URL("http://google.com").openConnection(proxy);
con.setConnectTimeout(1000);
con.setReadTimeout(1000);
con.setRequestMethod("GET");
con.setRequestProperty("User-Agent", "Mozilla/5.0");
con.getContent();
con.disconnect();
System.out.println("Proxy found!" + addr.getHostAddress() + ":" + port + " Found at " + percent + "% network search");
} catch (Exception e) {
}
}
//
//System.out.println("Ping response: --" + addr.getHostAddress() + "-- Attempt: " + counter.get() + " Percent: " + percent + "%");
} else {
//System.out.println("Ping response failed: " + addr.getHostAddress() + " attempt " + counter.incrementAndGet());
}
} catch (Exception e) {
//e.printStackTrace();
}
} while (true);
}).start();
}
}
private static RandomAccessFile cache;
private static byte[] getNextAddress() throws Exception {
if (cache == null) {
cache = new RandomAccessFile(File.createTempFile("abc", ".tmp"), "rw");
}
byte[] check;
checkFile:
{
byte[] addr = new byte[4];
do {
check = randomAddress();
inner:
{
cache.seek(0);
while (cache.length() - cache.getFilePointer() > 0) {
cache.readFully(addr);
if (Arrays.equals(check, addr)) {
break inner;
}
}
cache.write(check);
break checkFile;
}
} while (true);
}
return check;
}
private static byte[] randomAddress() {
return new byte[]{(byte) (Math.random() * 256), (byte) (Math.random() * 256), (byte) (Math.random() * 256), (byte) (Math.random() * 256)};
}
private static boolean ping(InetAddress addr) throws Exception {
return addr.isReachable(500);
}
}
Also in case anyone is wondering, I've had this running for 12 hours now and it's discovered about 50 proxys, and pinged about 2.09664E-4% of the ip range which is about 1.2 million ips. not bad for the bandwidth allocated (0.5Mbps)
EDIT: I am starting to think that maybe the overhead of storing and checking all of these IPs would be even greater than simply connecting to many duplicates near the end of searching the ip range..
I would not store the whole IP address because of the amount of data. To store them in a array of BitSet would consume less memory.
edit previous code version removed, it was not correct
The version below generates random addresses and persist them in a file. If the persistence file of a previous run is found, the information of the seen addresses is restored from that file.
Following case was not handled correctly in the initial version:
assuming that no address was already seen
1.0.0.1 - seen false
2.0.0.2 - seen false
2.0.0.1 - seen true, which was wrong and is correctly handled by code below
See the comments in the code for further information.
public class KeepSeenAddresses {
static final int FILE_BUFFER_SIZE = 81_920;
static final int RANGES_SIZE = 256;
// to store 256 ranges of 255*255*255+1 addresses
static BitSet[] ranges;
// Random(1) is taken only for demonstration purpose, so the second
// application run will find the same seen addresses from previous run
static Random random = new Random(1);
// for normal use it's better to have better randomness
//static Random random = new Random(System.currentTimeMillis());
public static void main(String[] args)
throws IOException, ClassNotFoundException {
if (!readRanges()) {
initRanges();
}
// this case was failing in the initial solution
// uncomment this block to see how all edge cases
// which where mentioned in other comments are handled
/*
byte[][] addresses = {
{1, 0, 0, 1},
{2, 0, 0, 2},
{2, 0, 0, 1},
{1, 2, 3, 4},
{4, 3, 2, 1},
{(byte)128, 0, 0, 0},
{(byte)255, (byte)255, (byte)255, (byte)255}
};
seenAddress(addresses[0]);
seenAddress(addresses[1]);
seenAddress(addresses[3]);
seenAddress(addresses[5]);
seenAddress(addresses[6]);
for (byte[] addressBytes : addresses) {
System.out.printf("seen %s before: %s%n",
prettyAddress(addressBytes),
seenBefore(addressBytes)
);
}
*/
processAddresses();
persistRanges();
}
/**
* Read the seen addresses from a file.
*
* #return <code>true</code> if the file was found and has the expected
* number of ranges, otherwise <code>false</code>
* #throws IOException
* #throws ClassNotFoundException
*/
private static boolean readRanges() throws IOException, ClassNotFoundException {
File rangesStore = new File("addresses.bin");
if (!rangesStore.exists()) {
return false;
}
System.out.print("found previous rangesStore... ");
try (ObjectInputStream ois = new ObjectInputStream(
new BufferedInputStream(
new FileInputStream(rangesStore), FILE_BUFFER_SIZE
)
)) {
ranges = (BitSet[]) ois.readObject();
}
if (ranges.length != RANGES_SIZE) {
System.out.printf("wrong size of rangesStore: expected %d"
+ " found: %d%n", RANGES_SIZE, ranges.length);
return false;
} else {
System.out.printf("restored ranges: %d%n", ranges.length);
return true;
}
}
/**
* Initialize the address ranges array. All address flags will be set to
* <code>false</code>.
*/
private static void initRanges() {
System.out.print("initialize new rangesStore... ");
ranges = new BitSet[RANGES_SIZE];
for (int i = 0; i < RANGES_SIZE; i++) {
BitSet bitSet = new BitSet(255 * 255 * 255 + 1);
for (int j = 0; j < 255 * 255 * 255 + 1; j++) {
bitSet.clear(j);
}
ranges[i] = bitSet;
}
System.out.printf("initialized ranges: %d%n", RANGES_SIZE);
}
/**
* For demonstration purpose.<br>
* Generates some random IPv4 addresses. If the address was not seen before
* the flag for this address will be set to <code>true</code>.
*/
private static void processAddresses() {
for (int i = 0; i < 10; i++) {
byte[] addrBytes = randomAddress();
boolean seenBefore = seenBefore(addrBytes);
if (!seenBefore) {
seenAddress(addrBytes);
seenBefore = false;
}
System.out.printf("seen %s before: %s%n",
prettyAddress(addrBytes),
seenBefore
);
}
}
/**
* Persist the address ranges array. The file size is around 500MB.
*
* #throws IOException
*/
private static void persistRanges() throws IOException {
System.out.print("persist rangesStore... ");
try (ObjectOutputStream oos = new ObjectOutputStream(
new BufferedOutputStream(
new FileOutputStream("addresses.bin"), FILE_BUFFER_SIZE)
)) {
oos.writeObject(ranges);
}
System.out.printf("written ranges: %d%n", ranges.length);
}
/**
* Keep a flag which address has been seen already.
*
* #param addrBytes IPv4 address in four bytes
*/
static void seenAddress(byte[] addrBytes) {
int rangeIndex = (int) addrBytes[0] & 0xff;
int rangeOffset = ((int) addrBytes[1] & 0xff * 0xffff)
+ ((int) addrBytes[2] & 0xff * 0xff)
+ ((int) addrBytes[3] & 0xff);
ranges[rangeIndex].set(rangeOffset);
}
/**
* Check if the passed address was seen before.
*
* #param addrBytes IPv4 address in four bytes
* #return <code>true</code> if the address was seen before, otherwise
* <code>false</code>
*/
static boolean seenBefore(byte[] addrBytes) {
int rangeIndex = (int) addrBytes[0] & 0xff;
int rangeOffset = ((int) addrBytes[1] & 0xff * 0xffff) + ((int) addrBytes[2] & 0xff * 0xff) + ((int) addrBytes[3] & 0xff);
return ranges[rangeIndex].get(rangeOffset);
}
/**
* Convert the IPv4 address into pretty string.
*
* #param addrBytes IPv4 address in four bytes
* #return pretty String of the IPv4 address
*/
static String prettyAddress(byte[] addrBytes) {
return String.format("%03d.%03d.%03d.%03d",
(int) addrBytes[0] & 0xff,
(int) addrBytes[1] & 0xff,
(int) addrBytes[2] & 0xff,
(int) addrBytes[3] & 0xff);
}
/**
* Generate a random IPv4 address.
*
* #return four bytes of a random generated IPv4 address
*/
private static byte[] randomAddress() {
byte[] bytes = new byte[4];
for (int i = 0; i < bytes.length; i++) {
bytes[i] = (byte) random.nextInt(256);
}
return bytes;
}
}
I have ported code from another solution here to fit this problem:
Java- Mapping multi-dimensional arrays to single
The answer to the above question gives an in depth explanation of how the following code works. If anyone else would like to post a more in depth answer on this thread I will award it the answer.
static BitSet set;
static int pos(int i, int j, int k, int m) {
return ((256*256*256) * i) + ((256*256) * j) + (256 * k) + m;
}
static boolean get(byte[] addr) {
return set.get(pos(addr[0], addr[1], addr[2], addr[3]));
}
static void set(byte[] addr, boolean flag) {
set.set(pos(addr[0], addr[1], addr[2], addr[3]), flag);
}
Use a data base like MySql and hibernarte with level 1 & 2 cache.
It will be faster than RAM if u configure a cache with hibernate and tune ur db to use few gb of cache too. i think they all do. can configure an external cache like ehcahe when an be configured to live on another process + file with limits on size and time. Db knows how to index and seek things faster than even pure RAM - at such sizes as your IP
Plus you can improve by partitioning table data & index by first char, 2nd char etc
Related
This question already has an answer here:
How to find start and End IP address for CIDR in snowflake
(1 answer)
Closed 1 year ago.
for example For cidr 2c0f:eb00:400::/40 (jw.org) ip_start should be - 2c0f:eb00:400:: ip_end will be - 2c0f:eb00:4ff:ffff:ffff:ffff:ffff:ffff
I filed an internal bug to fix the PARSE_IP() internal function for this edge case (SNOW-374145).
In the meantime, you can use a Java UDF, like:
select ipv6range('FF02:0:0:0:0:1:FF00::/104');
Returns:
[
"/ff02:0:0:0:0:1:ff00:0",
"/ff02:0:0:0:0:1:ffff:ffff"
]
With code taken from https://github.com/edazdarevic/CIDRUtils:
create or replace function ipv6range(s string)
returns array
language java
handler='MyClass.x'
as $$
/*
* The MIT License
*
* Copyright (c) 2013 Edin Dazdarevic (edin.dazdarevic#gmail.com)
* Permission is hereby granted, free of charge, to any person obtaining a copy
* of this software and associated documentation files (the "Software"), to deal
* in the Software without restriction, including without limitation the rights
* to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
* copies of the Software, and to permit persons to whom the Software is
* furnished to do so, subject to the following conditions:
* The above copyright notice and this permission notice shall be included in
* all copies or substantial portions of the Software.
* THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
* IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
* FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
* AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
* LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
* OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN
* THE SOFTWARE.
*
* */
import java.math.BigInteger;
import java.net.InetAddress;
import java.net.UnknownHostException;
import java.nio.ByteBuffer;
import java.util.ArrayList;
import java.util.List;
public class MyClass{
public static String[] x(String x) throws Exception{
CIDRUtils cidr = new CIDRUtils(x);
return new String[]{cidr.startAddress.toString(), cidr.endAddress.toString()};
}
}
/**
* A class that enables to get an IP range from CIDR specification. It supports
* both IPv4 and IPv6.
*/
public class CIDRUtils {
private final String cidr;
private InetAddress inetAddress;
InetAddress startAddress;
InetAddress endAddress;
private final int prefixLength;
public CIDRUtils(String cidr) throws UnknownHostException {
this.cidr = cidr;
/* split CIDR to address and prefix part */
if (this.cidr.contains("/")) {
int index = this.cidr.indexOf("/");
String addressPart = this.cidr.substring(0, index);
String networkPart = this.cidr.substring(index + 1);
inetAddress = InetAddress.getByName(addressPart);
prefixLength = Integer.parseInt(networkPart);
calculate();
} else {
throw new IllegalArgumentException("not an valid CIDR format!");
}
}
private void calculate() throws UnknownHostException {
ByteBuffer maskBuffer;
int targetSize;
if (inetAddress.getAddress().length == 4) {
maskBuffer =
ByteBuffer
.allocate(4)
.putInt(-1);
targetSize = 4;
} else {
maskBuffer = ByteBuffer.allocate(16)
.putLong(-1L)
.putLong(-1L);
targetSize = 16;
}
BigInteger mask = (new BigInteger(1, maskBuffer.array())).not().shiftRight(prefixLength);
ByteBuffer buffer = ByteBuffer.wrap(inetAddress.getAddress());
BigInteger ipVal = new BigInteger(1, buffer.array());
BigInteger startIp = ipVal.and(mask);
BigInteger endIp = startIp.add(mask.not());
byte[] startIpArr = toBytes(startIp.toByteArray(), targetSize);
byte[] endIpArr = toBytes(endIp.toByteArray(), targetSize);
this.startAddress = InetAddress.getByAddress(startIpArr);
this.endAddress = InetAddress.getByAddress(endIpArr);
}
private byte[] toBytes(byte[] array, int targetSize) {
int counter = 0;
List<Byte> newArr = new ArrayList<Byte>();
while (counter < targetSize && (array.length - 1 - counter >= 0)) {
newArr.add(0, array[array.length - 1 - counter]);
counter++;
}
int size = newArr.size();
for (int i = 0; i < (targetSize - size); i++) {
newArr.add(0, (byte) 0);
}
byte[] ret = new byte[newArr.size()];
for (int i = 0; i < newArr.size(); i++) {
ret[i] = newArr.get(i);
}
return ret;
}
public String getNetworkAddress() {
return this.startAddress.getHostAddress();
}
public String getBroadcastAddress() {
return this.endAddress.getHostAddress();
}
public boolean isInRange(String ipAddress) throws UnknownHostException {
InetAddress address = InetAddress.getByName(ipAddress);
BigInteger start = new BigInteger(1, this.startAddress.getAddress());
BigInteger end = new BigInteger(1, this.endAddress.getAddress());
BigInteger target = new BigInteger(1, address.getAddress());
int st = start.compareTo(target);
int te = target.compareTo(end);
return (st == -1 || st == 0) && (te == -1 || te == 0);
}
}
$$
;
Btw, the error shown by PARSE_IP() (until fixed) is "Error parsing IP: Invalid IPv6 address. IP address includes too many fields."
Hello i'm a beginner on the use of javacard ! I want to write a file (less than 1024kb) in my smart card so what i do is to transform the file to an array of byte (byte[]), for example i have an array with the length of 638 ! My code to write and read the data works fine (no error) but when i read the data from the smart card and compare it to the default array, they are not same (same length but different content). I read a lot of topic here but i was unable to find one which can solve my problem ! Thank you for help
I use JCDK 2.2.2
My applet that read and write data:
.....
public void process(APDU apdu) {
byte[] buffer = apdu.getBuffer();
switch(buffer[ISO7816.OFFSET_INS]) {
case INS_SET_DATA:
// byte bufferDataLength = (byte)(apdu.setIncomingAndReceive());
//We want to start data copy
if (buffer[ISO7816.OFFSET_P1] == 0x00 && buffer[ISO7816.OFFSET_P2] == 0x00) {
data = new byte[(short) 638];
} else {
// copying the apdu data into byte array Data
// array copy: (src, offset, target, offset,copy size)
Util.arrayCopyNonAtomic(buffer, (short) ISO7816.OFFSET_CDATA, data,
(short) ((short) buffer[ISO7816.OFFSET_P1] * 100),
(short) buffer[ISO7816.OFFSET_P2]
);
}
break;
case INS_GET_DATA:
// array copy: (src, offset, target, offset,copy size)
short t = (short)(buffer[ISO7816.OFFSET_P2] & 0xFF);
short o = (short)(buffer[ISO7816.OFFSET_P1] & 0xFF);
Util.arrayCopyNonAtomic(data, o, buffer, ISO7816.OFFSET_CDATA, t);
apdu.setOutgoingAndSend(ISO7816.OFFSET_CDATA, t);
break;
}
}
.....
The code i use to send the data to the smartcard (apduio package of JCDK 2.2.2)
.....
switch(choice) {
case 8:
byte[] configData = fileToByteArray("D:\\jcard\\config.dat");
byte[] chunk = new byte[100];
System.out.println("Config Length: " + configData.length); // Got 638
int configLength = configData.length;
double round = Math.floor((double)configLength / 100);
int lastPart = configLength % 100;
System.out.println("Round : " + round); // Got 6
System.out.println("Last Part : " + lastPart); // Got 38
double partToSend = lastPart == 0 ? round : round + 1;
//Initialize the array of byte in my applet to have the length of data i want to write in
apdu.command[Apdu.INS] = MyAppletClient.INS_SET_DATA;
apdu.command[Apdu.P1] = 0x00;
apdu.command[Apdu.P2] = 0x00;
cad.exchangeApdu(apdu);
handleResponse(apdu);
boolean allPartSend = true;
for (int i = 0; i < round; i++) {
//array copy: (src, offset, target, offset, copy size)
System.arraycopy(configData, (i * 100), chunk, 0, 100);
System.out.println("Data Length: " + chunk.length); // Got 100
apdu.command[Apdu.P1] = (byte) i;
apdu.command[Apdu.P2] = 100;
apdu.setDataIn(Data);
cad.exchangeApdu(apdu);
if (apdu.getStatus() != 0x9000) {
System.out.println("["+i+"] An error occurred with status: " + apdu.getStatus());
allPartSend = false;
break;
}
}
if(allPartSend) {
byte[] finalPart = new byte[lastPart];
System.arraycopy(configData, (int) (round * 100), finalPart, 0, lastPart);
apdu.command[Apdu.P1] = (byte) round;
apdu.command[Apdu.P2] = (byte) lastPart;
apdu.setDataIn(finalPart);
cad.exchangeApdu(apdu);
if (apdu.getStatus() != 0x9000) {
System.out.println("An error occurred with status: " + apdu.getStatus());
break;
} else {
System.out.println("OK");
}
} else {
System.out.println("Fail to send all data");
}
break;
case 9:
int cfgLength = 638; //Because i know the array has that length
byte[] res = new byte[cfgLength];
double rnd = Math.floor((double)cfgLength / 100);
int last = fpLength % 100, readLength = 0;
boolean allSend = true;
for(int i = 0; i < rnd; i++) {
apdu.command[Apdu.INS] = MyAppletClient.INS_GET_DATA;
apdu.command[Apdu.P1] = (byte) (i * 100);
apdu.command[Apdu.P2] = 100;
cad.exchangeApdu(apdu);
if (apdu.getStatus() != 0x9000) {
System.out.println("An error occurred with status: " + apdu.getStatus());
} else {
readLength += apdu.dataOut.length;
// System.out.println("Datalength : " + apdu.dataOut.length); // Got 100
//array copy: (src, offset, target, offset, copy size)
System.arraycopy(apdu.dataOut, 0, res, (i*100), apdu.dataOut.length);
}
}
if(allSend) {
apdu.command[Apdu.INS] = MyAppletClient.INS_GET_DATA;
apdu.command[Apdu.P1] = (byte) ((int)rnd * 100);
apdu.command[Apdu.P2] = (byte) last;
cad.exchangeApdu(apdu);
if (apdu.getStatus() != 0x9000) {
System.out.println("An error occurred with status: " + apdu.getStatus());
break;
} else {
readLength += apdu.dataOut.length;
//array copy: (src, offset, target, offset, copy size)
System.arraycopy(apdu.dataOut, 0, res, (int)(rnd * 100 ), apdu.dataOut.length);
}
} else {
System.out.println("Fail to get all the part");
}
byte[] cfgData = fileToByteArray("D:\\jcard\\config.dat");
System.out.println("ReadLength : " + readLength); // Got 638
System.out.println("Array equals : " + Arrays.equals(cfgData, res)); // Got false but expected true
break;
}
}
.....
It looks like you are mixing ISO case 1 and 3 commands. An ISO case 1 means that no CDATA or RDATA is present, and ISO case 3 means just CDATA, no RDATA. Generally it is best to let a command be only one of the possible 4 cases (case 2 is just RDATA and case 4 both CDATA and RDATA. For instance, you can create a CREATE RECORD command or something similar.
You should call setIncomingAndReceive right before copying the data to the array. If you do not do this then only part of the data may be present in the buffer. Note though that setIncomingAndReceive is a convenience method, and you may be able to find better methods for handling large amounts of data.
Note that ISO 7816-4 already defines a file based card system; copying part of that existing standard may be more useful than creating your own from scratch. Also note that you should prefer APDU#getOffsetCdata rather than the constant. That method is compatible with extended length APDU's, and you may want to upscale to those in the future.
My guess is that the reason why you are receiving different data from the one you wrote to the card is this line of code:
switch(choice) {
...
case 9:
...
for(int i = 0; i < rnd; i++) {
...
apdu.command[Apdu.P1] = (byte) (i * 100);
Unlike in your code for writing data to the card, here you are passing the data offset in P1 parameter. Please remember, that P1 is only one byte and it will overflow for i=3, so it is not possible to encode offsets of 638-bytes long array in P1.
The quickest solution of your problem is to use the same approach as in your writing code, i.e.:
switch(choice) {
...
case 9:
...
for(int i = 0; i < rnd; i++) {
...
apdu.command[Apdu.P1] = (byte) i;
and do the multiplication in the JavaCard code:
short o = (short) ((short) buffer[ISO7816.OFFSET_P1] * 100);
This solution should work, but it is not perfect as your JavaCard code supports arbitrary length of read/write (P2 - copy length), but offsets are fixed to 0, 100, 200 etc. It means that it works only as long as you send P2=100 to the card, which your present code does. But then why sending this parameter in the first place if it is fixed to one value?
I agree with Maarten Bodewes that it would be best to follow the ISO/IEC 7816-4 and use the command syntax specified there. UPDATE BINARY and READ BINARY commands described in the standard should fit your needs.
I have been trying to get audio signal from an arduino microphone. First I setup my arduino in Free Running Mode, and then I wait for a message from Processing that is a capital A. When Processing sends the serial signal, my arduino should send back 16 audio amplitude samples. Then, processing should take those 16 audio samples and pass them by the Fast Fourier Transform in orden to get a real time frequency spectrum. However, it seems that my Arduino never detects the capital A, and also, if I try to send the samples from my Arduino without waiting for the capital A, when Processing gets the samples it seems something went wrong with the serial communication. I tried with many arduinos and different serial cables to be sure it was no the problem. Here you are the arduino code:
// Arrays to save our results in
int values[16];
// Define various ADC prescaler
const unsigned char PS_16 = (1 << ADPS2);
const unsigned char PS_32 = (1 << ADPS2) | (1 << ADPS0);
const unsigned char PS_64 = (1 << ADPS2) | (1 << ADPS1);
const unsigned char PS_128 = (1 << ADPS2) | (1 << ADPS1) | (1 << ADPS0);
// Setup the serial port and pin 2
void setup() {
Serial.begin(115200);
pinMode(2, INPUT);
// set up the ADC
ADCSRA &= ~PS_128; // remove bits set by Arduino library
// you can choose a prescaler from above.
// PS_16, PS_32, PS_64 or PS_128
ADCSRA |= PS_64; // set our own prescaler to 64
}
void loop() {
unsigned int i;
for(i=0;i<16;i++) {
values[i] = analogRead(2);
}
if(Serial.available() > 0) {
char c = Serial.read();
if(c == 'A') {
for(i=0;i<16;i++) {
Serial.println(values[i]);
}
}
}
delay(10);
}
Here you are the processing code:
import processing.serial.*;
import ddf.minim.analysis.*;
FFT fft;
Serial myPort; // Create object from Serial class
void setup()
{
String portName = Serial.list()[0];
myPort = new Serial(this, portName, 115200);
size(800, 600);
background(255);
fft = new FFT(16, 16000);
}
String mensaje;
float[] magnitudes = new float[16];
int contador = 0;
byte request = true;
void draw()
{
if(request) {
myPort.write('A');//Solicita nuevos datos.
request = false;
}
while(!request) {
if(myPort.available()>0) {
mensaje = myPort.readStringUntil(13);
print(" mensaje: "+mensaje);
if(mensaje != null){
magnitudes[contador++] = int(mensaje);
}
}
if(contador >= 15) {
contador=0;
fft.forward(magnitudes);
request = true;
fill(100);
rect(0,0,800,800);
for (int i = 0; i < 15; i++) {
fill(200);
rect(i*30+10, 0, 10, fft.getBand(i) * 0.1);
}
}
}
}
Thank you a lot for your attention!
Hi I have a requirement where I need to generate SessionID like thing but that too concurrently. I am thinking to take the code from tomcat 7 source "SessionIdGenerator". But I am not sure whether will it genetate uniqueID concurrently without any modification.
class itself is here:
public class SessionIdGenerator {
private Logger logger = LoggerFactory.getLogger();
/**
* Queue of random number generator objects to be used when creating session
* identifiers. If the queue is empty when a random number generator is
* required, a new random number generator object is created. This is
* designed this way since random number generators use a sync to make them
* thread-safe and the sync makes using a a single object slow(er).
*/
private Queue<SecureRandom> randoms = new ConcurrentLinkedQueue<SecureRandom>();
/**
* The Java class name of the secure random number generator class to be
* used when generating session identifiers. The random number generator
* class must be self-seeding and have a zero-argument constructor. If not
* specified, an instance of {#link SecureRandom} will be generated.
*/
private String secureRandomClass = null;
/**
* The name of the algorithm to use to create instances of
* {#link SecureRandom} which are used to generate session IDs. If no
* algorithm is specified, SHA1PRNG is used. To use the platform default
* (which may be SHA1PRNG), specify the empty string. If an invalid
* algorithm and/or provider is specified the {#link SecureRandom} instances
* will be created using the defaults. If that fails, the {#link
* SecureRandom} instances will be created using platform defaults.
*/
private String secureRandomAlgorithm = "SHA1PRNG";
/**
* The name of the provider to use to create instances of
* {#link SecureRandom} which are used to generate session IDs. If
* no algorithm is specified the of SHA1PRNG default is used. If an invalid
* algorithm and/or provider is specified the {#link SecureRandom} instances
* will be created using the defaults. If that fails, the {#link
* SecureRandom} instances will be created using platform defaults.
*/
private String secureRandomProvider = null;
/** Node identifier when in a cluster. Defaults to the empty string. */
private String jvmRoute = "";
/** Number of bytes in a session ID. Defaults to 16. */
private int sessionIdLength = 16;
/**
* Specify a non-default #{link {#link SecureRandom} implementation to use.
*
* #param secureRandomClass The fully-qualified class name
*/
public void setSecureRandomClass(String secureRandomClass) {
this.secureRandomClass = secureRandomClass;
}
/**
* Specify a non-default algorithm to use to generate random numbers.
*
* #param secureRandomAlgorithm The name of the algorithm
*/
public void setSecureRandomAlgorithm(String secureRandomAlgorithm) {
this.secureRandomAlgorithm = secureRandomAlgorithm;
}
/**
* Specify a non-default provider to use to generate random numbers.
*
* #param secureRandomProvider The name of the provider
*/
public void setSecureRandomProvider(String secureRandomProvider) {
this.secureRandomProvider = secureRandomProvider;
}
/**
* Specify the node identifier associated with this node which will be
* included in the generated session ID.
*
* #param jvmRoute The node identifier
*/
public void setJvmRoute(String jvmRoute) {
this.jvmRoute = jvmRoute;
}
/**
* Specify the number of bytes for a session ID
*
* #param sessionIdLength Number of bytes
*/
public void setSessionIdLength(int sessionIdLength) {
this.sessionIdLength = sessionIdLength;
}
/**
* Generate and return a new session identifier.
*/
public String generateSessionId() {
byte random[] = new byte[16];
// Render the result as a String of hexadecimal digits
StringBuilder buffer = new StringBuilder();
int resultLenBytes = 0;
while (resultLenBytes < sessionIdLength) {
getRandomBytes(random);
for (int j = 0;
j < random.length && resultLenBytes < sessionIdLength;
j++) {
byte b1 = (byte) ((random[j] & 0xf0) >> 4);
byte b2 = (byte) (random[j] & 0x0f);
if (b1 < 10)
buffer.append((char) ('0' + b1));
else
buffer.append((char) ('A' + (b1 - 10)));
if (b2 < 10)
buffer.append((char) ('0' + b2));
else
buffer.append((char) ('A' + (b2 - 10)));
resultLenBytes++;
}
}
if (jvmRoute != null && jvmRoute.length() > 0) {
buffer.append('.').append(jvmRoute);
}
return buffer.toString();
}
private void getRandomBytes(byte bytes[]) {
SecureRandom random = randoms.poll();
if (random == null) {
random = createSecureRandom();
}
random.nextBytes(bytes);
randoms.add(random);
}
/**
* Create a new random number generator instance we should use for
* generating session identifiers.
*/
private SecureRandom createSecureRandom() {
SecureRandom result = null;
long t1 = System.currentTimeMillis();
if (secureRandomClass != null) {
try {
// Construct and seed a new random number generator
Class<?> clazz = Class.forName(secureRandomClass);
result = (SecureRandom) clazz.newInstance();
} catch (Exception e) {
logger.debug("exception:"+e);
}
}
if (result == null) {
// No secureRandomClass or creation failed. Use SecureRandom.
try {
if (secureRandomProvider != null &&
secureRandomProvider.length() > 0) {
result = SecureRandom.getInstance(secureRandomAlgorithm,
secureRandomProvider);
} else if (secureRandomAlgorithm != null &&
secureRandomAlgorithm.length() > 0) {
result = SecureRandom.getInstance(secureRandomAlgorithm);
}
} catch (NoSuchAlgorithmException e) {
logger.debug("exception:"+e);
} catch (NoSuchProviderException e) {
logger.debug("exception:"+e);
}
}
if (result == null) {
// Invalid provider / algorithm
try {
result = SecureRandom.getInstance("SHA1PRNG");
} catch (NoSuchAlgorithmException e) {
logger.debug("exception:"+e);
}
}
if (result == null) {
// Nothing works - use platform default
result = new SecureRandom();
}
// Force seeding to take place
result.nextInt();
long t2=System.currentTimeMillis();
if( (t2-t1) > 100 )
logger.debug("sessionIdGenerator algorithm:"+result.getAlgorithm()+"time taken:" +Long.valueOf(t2-t1));
return result;
}
}
I'd use java.util.UUID.randomUUID() instead. Very simple and guarantees uniqueness.
I have a sequence of bytes that I have to search for in a set of Binary files using Java.
Example: I'm searching for the byte sequence DEADBEEF (in hex) in a Binary file.
How would I go about doing this in Java? Is there a built-in method, like String.contains() for Binary files?
No, there is no built-in method to do that. But, directly copied from HERE (with two fixes applied to the original code):
/**
* Knuth-Morris-Pratt Algorithm for Pattern Matching
*/
class KMPMatch {
/**
* Finds the first occurrence of the pattern in the text.
*/
public static int indexOf(byte[] data, byte[] pattern) {
if (data.length == 0) return -1;
int[] failure = computeFailure(pattern);
int j = 0;
for (int i = 0; i < data.length; i++) {
while (j > 0 && pattern[j] != data[i]) {
j = failure[j - 1];
}
if (pattern[j] == data[i]) { j++; }
if (j == pattern.length) {
return i - pattern.length + 1;
}
}
return -1;
}
/**
* Computes the failure function using a boot-strapping process,
* where the pattern is matched against itself.
*/
private static int[] computeFailure(byte[] pattern) {
int[] failure = new int[pattern.length];
int j = 0;
for (int i = 1; i < pattern.length; i++) {
while (j > 0 && pattern[j] != pattern[i]) {
j = failure[j - 1];
}
if (pattern[j] == pattern[i]) {
j++;
}
failure[i] = j;
}
return failure;
}
}
private int bytesIndexOf(byte[] source, byte[] search, int fromIndex) {
boolean find = false;
int i;
for (i = fromIndex; i < (source.length - search.length); i++) {
if (source[i] == search[0]) {
find = true;
for (int j = 0; j < search.length; j++) {
if (source[i + j] != search[j]) {
find = false;
}
}
}
if (find) {
break;
}
}
if (!find) {
return -1;
}
return i;
}
You can find sequence of bytes from giga-bytes order file using bigdoc.
Lib and Example here on Github at: https://github.com/riversun/bigdoc
package org.example;
import java.io.File;
import java.util.List;
import org.riversun.bigdoc.bin.BigFileSearcher;
public class Example {
public static void main(String[] args) throws Exception {
byte[] searchBytes = "hello world.".getBytes("UTF-8");
File file = new File("/var/tmp/yourBigfile.bin");
BigFileSearcher searcher = new BigFileSearcher();
List<Long> findList = searcher.searchBigFile(file, searchBytes);
System.out.println("positions = " + findList);
}
}
If you want to search it on memory,check this.
Examples here on Github at: https://github.com/riversun/finbin
import java.util.List;
import org.riversun.finbin.BigBinarySearcher;
public class Example {
public static void main(String[] args) throws Exception {
BigBinarySearcher bbs = new BigBinarySearcher();
byte[] iamBigSrcBytes = "Hello world.It's a small world.".getBytes("utf-8");
byte[] searchBytes = "world".getBytes("utf-8");
List<Integer> indexList = bbs.searchBytes(iamBigSrcBytes, searchBytes);
System.out.println("indexList=" + indexList);
}
}
Returns all the matched positions in the array of bytes
It also can withstand a large array of bytes:)
For those who prefer libraries, there is an implementation (source below) of the Knuth-Morris-Pratt algorithm in Twitter's Elephant-Bird open source library (Apache license).
You can find the library on Github at: https://github.com/twitter/elephant-bird
package com.twitter.elephantbird.util;
import java.io.IOException;
import java.io.InputStream;
import java.util.Arrays;
/**
* An efficient stream searching class based on the Knuth-Morris-Pratt algorithm.
* For more on the algorithm works see: http://www.inf.fh-flensburg.de/lang/algorithmen/pattern/kmpen.htm.
*/
public class StreamSearcher {
protected byte[] pattern_;
protected int[] borders_;
// An upper bound on pattern length for searching. Throws exception on longer patterns
public static final int MAX_PATTERN_LENGTH = 1024;
public StreamSearcher(byte[] pattern) {
setPattern(pattern);
}
/**
* Sets a new pattern for this StreamSearcher to use.
* #param pattern
* the pattern the StreamSearcher will look for in future calls to search(...)
*/
public void setPattern(byte[] pattern) {
if (pattern.length > MAX_PATTERN_LENGTH) {
throw new IllegalArgumentException("The maximum pattern length is " + MAX_PATTERN_LENGTH);
}
pattern_ = Arrays.copyOf(pattern, pattern.length);
borders_ = new int[pattern_.length + 1];
preProcess();
}
/**
* Searches for the next occurrence of the pattern in the stream, starting from the current stream position. Note
* that the position of the stream is changed. If a match is found, the stream points to the end of the match -- i.e. the
* byte AFTER the pattern. Else, the stream is entirely consumed. The latter is because InputStream semantics make it difficult to have
* another reasonable default, i.e. leave the stream unchanged.
*
* #return bytes consumed if found, -1 otherwise.
* #throws IOException
*/
public long search(InputStream stream) throws IOException {
long bytesRead = 0;
int b;
int j = 0;
while ((b = stream.read()) != -1) {
bytesRead++;
while (j >= 0 && (byte)b != pattern_[j]) {
j = borders_[j];
}
// Move to the next character in the pattern.
++j;
// If we've matched up to the full pattern length, we found it. Return,
// which will automatically save our position in the InputStream at the point immediately
// following the pattern match.
if (j == pattern_.length) {
return bytesRead;
}
}
// No dice, Note that the stream is now completely consumed.
return -1;
}
/**
* Builds up a table of longest "borders" for each prefix of the pattern to find. This table is stored internally
* and aids in implementation of the Knuth-Moore-Pratt string search.
* <p>
* For more information, see: http://www.inf.fh-flensburg.de/lang/algorithmen/pattern/kmpen.htm.
*/
protected void preProcess() {
int i = 0;
int j = -1;
borders_[i] = j;
while (i < pattern_.length) {
while (j >= 0 && pattern_[i] != pattern_[j]) {
j = borders_[j];
}
borders_[++i] = ++j;
}
}
}
If the approved answer is too complex for your task then try simple code below. It uses native Java libs only
/**
* Search byte array for the byte pattern
*
* #param array to look into
* #param pattern to search
* #return index of first occurrence of the pattern, -1 otherwise
*/
public static int searchBytePattern(byte[] array, byte[] pattern) {
if (pattern.length > array.length) {
return -1;
}
for (int i = 0; i <= array.length - pattern.length; i++) {
if (Arrays.compare(array, i, i + pattern.length, pattern, 0, pattern.length) == 0) {
return i;
}
}
return -1;
}