I'm trying to read serial data from my Arduino using Java. I have followed the tutorial Arduino and Java.
So far I have it working except, I'm not reading the whole of the serial data at once. For example, the Arduino should be sending 0.71, and I read in a 0 then a .71 or some other combination. Sometimes I read it in fine, but more often than not, the data is broken up.
I have a hunch that if I changed the data type of what I am sending to a byte, my program would be okay, however I need float precision in the data I am transferring. How do I fix this problem?
Serial protocols such as the one used for serial over USB are byte oriented, not packet oriented. As such, there's no guarantee that a read will return the entire 'message' sent by the other end, or just part of it, as you're observing, because there's no concept of message or packet.
Instead, you need to delimit your messages in some way - such as by appending a newline - or preprend messages with a length field, so you know how many bytes to read.
I do this by making my Arduino send 'frames', separated by a 'gap'. It is easy to configure a timeout (at least in Perl it is) for reading data from the serial port. So what I do is:
Allow data being read during the duration of the data frame plus an extra few milliseconds:
[ (number of bytes) × 10 bits × 1000 ms / (baud rate) ] + 100 milliseconds
Then the gap between two values or frames being sent, should be longer than this value.
The program easily synchronizes to the data stream, because of the strategic timout.
I also add a simple preamble in my data to check data integrity.
Related
I am trying to send data from an android phone in Host card emulation mode to a reader application. I do understand the maximum size of an apdu should be about 260bytes. However I need to send well beyond that (a few thousand bytes). I know I can divide the data and send it in "chunks", but I am really worried about cost of that on the general performance.Is there anyway I can send a bigger apdu than 260 bytes. I don't mind a little hack too if I have to. Cheers
To answer my own question. There are 2 types of APDUs according to the maximum size of data they accommodate. Normal sized APDUs (256 bytes) and extended APDUs with payload of upto 65536 bytes. However not all smartcards and readers support the extended APDU length.
Now on the android side of things, the extended length APDUs is not supported by the Android OS. Even though most of the NFC controllers support it. Therefore this is a software limitation and not a hardware one. See the
getMaxTransceiveLength method in
https://android.googlesource.com/platform/packages/apps/Nfc/+/master/nci/src/com/android/nfc/dhimpl/NativeNfcManager
I'm collecting a large number of UDP packets (time dependant) coming from a service on the same network. These packets are being deserialised into structures that contain numbers (float and int) in memory and processed. We could say we are collecting time series data. However, it's not the kind of time series data that you get from supervising a service (mostly the same value for a period of time). These values constantly vary, true, not by very much. But they vary, nevertheless.
Besides this, I would like to send that data to a server in the cloud and on that server store the time series data.
My question is: what possibilities are there to compress the data in order to send smaller packets over the wire to the server (we could send the incoming UDP packets in batches over TCP) and store them? I'm particularly interested in not using the whole storage I have attached to the server. The data for one session is close to 32MB and i would have multiple sessions at the same time. One session's data is not related to another session. They are totally independent.
You can compress time series data using this library: https://github.com/dgryski/go-tsz
It's based on this paper from Facebook:
http://www.vldb.org/pvldb/vol8/p1816-teller.pdf
We have found that about 96% of all time stamps
can be compressed to a single bit.
[...]
Roughly 51% of all values are compressed to a single bit since
the current and previous values are identical. About 30% of
the values are compressed with the control bits ‘10’ (case b),
with an average compressed size of 26.6 bits. The remaining
19% are compressed with control bits ‘11’, with an average
size of 36.9 bits, due to the extra 13 bits of overhead required
to encode the length of leading zero bits and meaningful bits.
You can use a key-value store like bolt (or probably better: rocksdb which supports compression) and store multiple points for each key. For example you could store one key-value pair every 10 minutes, where the value would be all of the points that occurred during that 10 minute window.
This should give you both good performance and high compression.
I recently bought an Arduino with an LCD screen. I want to push information from my computer to the Arduino. I came across a great article, How to make a physical Gmail notifier. From what I understand, I have to send the information using Serial and read it in the C/C++ code on the Arduino. That is fine, but I want to send different information to the device.
Say I want to have one part of the LCD-screen showing the temperature outside and another part of the screen display when the next bus is coming. Is there any way to "mark" the information I send with Serial, or does everything end up in the same "channel"?
If that is the case, is there a logical, simple way to separate this information so it does not mistake bus-information for temperature and vice versa?
You need a protocol for sending information across the serial line, so that the data can be collected the other end in a way that makes sense. A simple protocol may be:
T:16.0 09.34 // Temperature, 16.0°C measured at 09.34
B:11b 11.46 // Bus, route 11b, arrives at 11.46 at your bus-stop.
M:mats#example.com 11kb 10.23 // Mail from mats#example.com, it's 11KB and arrived at 10.23
Each line contains one type of information.
Assuming the line of communication is reliable (and as long as your wire isn't several dozen feet, it should be), you don't need more than that. If the communicatio is unreliable, you need some sort of "start" and "end" markers (or a start and a length), a checksum and some way of dealing with "it went wrong". You will also need to read with a timeout, so that when you don't get enough data, the system starts over again with the next bit of information.
Is there any way to "mark" the information I send with Serial
Definitely. YOU decide how the information is sent if you have control over the information passing over the serial port on your computer.
or does everything end up in the same "channel"?
Well, the serial port is a kind of a channel I guess, since all information you wish to send to the Arduino goes over the port.
is there a logical, simple way to separate this information so it does not mistake bus-information for temperature and vice versa.
Yes. Say you want to send temperature data. Create a byte array for example in this manner: {T23.4} = Temperature data
The bracket '{' signals to the receiving code in the arduino that information is coming down the line with some data. The letter T indicates temperature. Everything after the letter 'T' up to the '}' is data. (23.4)
Bus information could be {Bxxx} where xxx is the data.
I am using simple UDP connection.
I would like to know if by default the connection has "Carriage return" enabled or disabled, and how could I set that property?
thanks,
ray.
Eh, that's not entirely accurate. UDP isn't differentiated by virtue of sending text vs. binary. All network protocols ultimately send data as bit streams (binary). What typically differentiates it is that unlike TCP, there is no back and forth to establish sequence numbers for tracking packets, and no ACK flag to signal that a packet was received. UDP will send packets with no regard to whether or not they get to the destination.
Edit: Ray maybe you should provide a little more detail about what you're trying to do. Carriage Return is an ascii character just like any other. It has a numerical representation and occupies a byte of space just like the other ascii characters. So asking if it's "enabled" for UDP transmission isn't really a valid question. Any series of bits can be sent via UDP, or TCP, or any other protocol - which means UDP doesn't even understand what ASCII is, or the letter "b", or a carriage return. It's all just a bunch of 1's and 0's, and UDP is aware of IP addresses and Port numbers - just enough to send your bits of data somewhere. What your application does with those bits is the question.
UDP traffic is session/connection less. So you can't have a "connection" on UDP.
UDP is used to pass binary data rather than text and there is no way to disable carriage return or any other character.
UDP broadcasts binary data - if you encode \r and/or \n to bytes and add it to the message, it will be sent. No filtering, no conversion on this protocol layer.
I am trying to understand the CANopen protocol.
For now, I do not have any CAN hardware nor the CANopen stack to experiment with.
I would like to know how to write a Java program to simply interpret CANopen messages that are received at the RS-232 port.
Are there CAN interfaces that are installed as a serial port?
Will I be able to write a program to process CANopen messages? I want only to be able to receive and interpret messages. Is it as simple as creating a buffer for the input stream and then break up the transmission into separate messages according to the SOF and EOF? How do I know what is the SOF/EOF since it is only 1-bit long?
Why is there a limit on the number of PDOs of a CAN node?
How do I process the PDO to identify the node from which it is sent and the data type and value? Is the PDO a standard CAN frame?
I don't know of any CAN interface that connects to the serial port (it wouldn't be too hard to create one based on a microcontroller with CAN and serial ports). However, standard serial ports would be too slow to support the higher speeds available in CAN.
Generally, when using the API for a CAN interface, you will be able to read messages consisting of ID, Length and up to eight bytes of data. You don't need to care about SOF/EOF. Even if interfacing directly on the low level with a CAN controller (that is, if you have a CAN interface for which you need to write the driver/API yourself), you still don't need to care about those details. And you don't want to try to access the CAN bus without using a CAN controller at all...
If you want to pretend that you have a CAN interface, you may create a stub function which returns those three items: an ID, a data length and a 64-bit data buffer. This is basically what all CAN interface APIs will give you. And when transmitting CAN messages, you will use the same parameters (ID, length data).
PDOs are defined by their use of the CAN ID field. In theory the number of PDOs for a device is not really that limited, but the predefined connection set have only allocated a small number (four) of PDOs for each node.
The PDOs are standard CAN frames. As mentioned, the CAN ID identifies the PDO. In the predefined connection set (which most devices follow), the CAN ID of all messages consists of a functional part and a module-ID part (the module ID may be hard coded for the device, or configurable by dip switches for example). Bits 10-7 of the CAN ID is the function code and bit 6-0 is the module number. For example TxPDO1 from a device with module ID 0x10 would have CAN ID 0x190. The upper four bits of the 11-bit CAN ID, ((CAN_ID & 0x780) >> 7), gives you the function code (TxPDO1 = 3) and the rest of the bits,(CAN_ID & 0x7f), gives the module id (which in this example was 0x10). So if you read a message on the CAN bus with CAN ID 0x190, you would know that this was a PDO from the device with module ID 0x10.
(A simpler way to express this might be to say that TxPDO1 has CAN ID set to 0x180+<module ID>, TxPDO2 has CAN ID set to 0x280+<module ID>, etc.)
How you should interpret the data in the PDO depends on your device.
I suggest that you find a good CANopen tutorial. Unfortunately most of them make everything sound much more complicated than it really is. So look around until you find one that appears understandable.
There are many CAN interfaces that can run off a serial port - VSCOM, Vector, and many others. There are also free programs that allow you to send and receive raw CAN frames - CANhacker, etc. Google for a few of them.
What I haven't found is a free program that can do interpret CANopen - most are pay programs. The exception is Wireshark for Linux - it uses SocketCAN to pull in packets and can parse all the CANopen frames.
I run my CAN bus a 1 Mbit/s and use a VSCOM interface to monitor it on the serial port.
CANFestival is a good open source stack that ports easily to Linux as well as bare machines.