I am using Android-Universal-Image-Loader to load images from remote server over HTTPS on my Android application. To have access to images the client should provide a valid token and sometimes server can return "expired crsf token" error. In order to handle this behavior a custom ImageDownloader should be defined. Below is the base implementation of method that should be overrrided in my implementation.
protected InputStream getStreamFromNetwork(String imageUri, Object extra) throws IOException {
HttpURLConnection conn = createConnection(imageUri, extra);
int redirectCount = 0;
while (conn.getResponseCode() / 100 == 3 && redirectCount < MAX_REDIRECT_COUNT) {
conn = createConnection(conn.getHeaderField("Location"), extra);
redirectCount++;
}
InputStream imageStream;
try {
imageStream = conn.getInputStream();
} catch (IOException e) {
// Read all data to allow reuse connection (http://bit.ly/1ad35PY)
IoUtils.readAndCloseStream(conn.getErrorStream());
throw e;
}
if (!shouldBeProcessed(conn)) {
IoUtils.closeSilently(imageStream);
throw new IOException("Image request failed with response code " + conn.getResponseCode());
}
return new ContentLengthInputStream(new BufferedInputStream(imageStream, BUFFER_SIZE), conn.getContentLength());
}
I want to rewrite it to handle invalid token errors. For example, if the server returns such error it should be recognized, token should be regenerated and request repeated.
The only solution I come up with is like this (shortened code):
imageStream = conn.getInputStream();
byte[] body = org.apache.commons.io.IOUtils.toByteArray(imageStream);
if (body.length < 300 // high probability to contain err message
&& isInvalidToken(body)) {
// handle error
}
return new ByteArrayInputStream(body);
Is is safe to use such kind of solution, considering I use it only for thumbnails of max 80kb size? Are there any other solutions?
Your solution is safe, although it's nicer if you create your ImageDownloaderInputStream class that implements InputStream and wraps the original InputStream. You can pre-load (buffer) some chunk from the underlying input stream to detect if the content is valid or not.
The only method you should override is read().
If the content is valid, you can serve the buffer content to the caller, when the buffer is empty, directly stream from the underlying InputStream.
If the content is invalid, just simply read another stream, or return a zero-length stream.
public class ImageDownloaderInputStream extends InputStream {
private byte[] buffer = null;
private int bufLen = 0;
private int bufIndex = 0;
private boolean isContentValid;
private InputStream wrapped;
public ImageDownloaderInputStream (InputStream wrapped) {
this.wrapped = wrapped;
}
#Override
public ind read() {
if(buffer == null) {
// check content and fill buffer
this.isContentValid = checkContent();
}
if (this.isContentValid) {
if(bufIndex < bufLen) {
return buffer[bufIndex++] & 0xFF;
} else {
return wrapped.read();
}
} else {
// error handling: zero-length stream
return -1;
}
}
private boolean checkContent() {
// fill the buffer
this.buffer = new byte[1024];
this.bufLen = wrapped.read(this.buffer);
// read more if not enough
// check the content
return true;
// return false;
}
}
You can check for a valid token after you checked that the response was 200 OK like so:
conn.getResponseCode() == HttpStatus.RESPONSE_OK && isValidToken(body)
If these conditions are not met then you handle it accordingly i.e repeat the request x times.
I would consider having a isValidToken(...) method instead of your isInvalidToken(...) so that you don't have to negate the response of the method.
Have you considered something like this?
if(conn.getResponseCode()==HttpStatus.RESPONSE_OK) else{ //repeat request...}
Related
I'm trying to get how many bytes are available for reading on a AOA connection but when I call the available () method of FileInputStream I get an IOException with the message: 'ioctl failed: EINVAL (Invalid argument)'
Note 1: I'm calling Java methods from C++ Using JNI
Note 2: If, instead of calling available() method I try to read, it work's perfectly
What am I doing wrong?
Is there another way to ask for available bytes before reading from the socket on an AOA connection?
This is the Java Code:
public int testConnection () throws Exception {
UsbManager usbManager = (UsbManager) this.context.getSystemService(Context.USB_SERVICE);
final UsbAccessory[] accessoryList = usbManager.getAccessoryList();
if (accessoryList == null || accessoryList.length == 0) {
return -1;
} else {
UsbAccessory usb = accessoryList[0];
try {
ParcelFileDescriptor fileDescriptor = usbManager.openAccessory(usb);
if (fileDescriptor != null) {
FileDescriptor fd = fileDescriptor.getFileDescriptor();
FileInputStream inStream = new FileInputStream(fd);
inStream.available ();
} else {
return -2;
}
} catch (IOException e) {
return -3;
} catch (Exception e) {
return -4;
}
}
return 0;
}
If you do a 'Ctrl' + 'Clic' on the available method, you will see that the method isn't available for Android targets. The implement is changed from bare native and always returns an Exception.
I am trying to mimic a TCP server for tests with Vertx based on existing infrastructure that I have to work with.
The server I am mimicking works completely async and knows the length of the incoming buffer based on a pre-header in the buffer that indicates the length of the request.
I need to read the first 6 characters of the incoming request on each client socket that connect to my mock TCP server. from this pre-header I read the actual length of the request (e.g. for xx3018, i know the full length of the request is 3018).
Then I need to read the rest of the buffer according to the length, match it to a map of responses and return the right response for the request.
Example for a working mock server with plain java (fast implementation so other development won't be blocked :) )
public void run(String... args) throws Exception {
log.info("Starting TCP Server");
ServerSocket serverSocket = new ServerSocket(1750);
while (true) {
try {
Socket socket = serverSocket.accept();
CompletableFuture.runAsync(() -> {
Exception e = null;
while (e == null) {
try {
InputStream inputStream = socket.getInputStream();
OutputStream outputStream = socket.getOutputStream();
byte[] preHeader = new byte[6];
inputStream.read(preHeader);
String preHeaderValue = new String(preHeader);
log.info("Pre header: {}", preHeaderValue);
int length = Integer.valueOf(preHeaderValue.substring(2));
log.info("Request full length: {}", length);
byte[] request = new byte[length - 6];
inputStream.read(request);
String requestValue = new String(request);
log.info("Request: {}", requestValue);
String response = this.requestResponseProvider.getResponse(preHeaderValue + requestValue);
log.info("Response: {}", response);
outputStream.write(response.getBytes());
} catch (Exception ex) {
log.error("Encountered a problem: {}", e.getMessage());
e = ex;
}
}
});
} catch (Exception e) {
log.error("Encountered a problem: {}", e.getMessage());
}
}
}
I can't seem to find a way to control the input stream the same way I control it with plain java.
After a very long time of leaving this issue aside, I decided to play with it a bit.
I remembered using the following module for a different project: https://github.com/vert-x3/vertx-tcp-eventbus-bridge
I also remembered that in the tcp bridge's internal protocol, it appends the length of the payload to the buffer that is being sent via the tcp bridge, I looked into the source code to find out how it handles chunks (aka frames)
I found the following: https://github.com/vert-x3/vertx-tcp-eventbus-bridge/blob/master/src/main/java/io/vertx/ext/eventbus/bridge/tcp/impl/protocol/FrameParser.java which does exactly what I wanted to achieve :)
I modified it a bit, converted to Kotlin, and made it so I can control the header size and the way it extracts the payload length.
The following is a rough quick and dirty example of controlling the read flow with Vert.x NetServer:
suspend fun main() {
val vertx = Vertx.vertx()
initServer(vertx)
initClient(vertx)
}
suspend fun initServer(vertx: Vertx) {
val server = vertx.createNetServer(netServerOptionsOf(port = 8888, host = "localhost"))
server
.connectHandler { socket ->
val parser = FrameParser(
headerSize = 4,
headerHandler = {
it.getInt(0)
},
handler = {
println(it.toString())
println("---")
}
)
socket.handler(parser)
socket.exceptionHandler {
it.printStackTrace()
socket.close()
}
}
.listenAwait()
}
suspend fun initClient(vertx: Vertx) {
val client = vertx.createNetClient()
val socket = client.connectAwait(port = 8888, host = "localhost")
val message = "START|${"foobarfoobar".repeat(100)}|END"
val length = message.length
repeat(5) {
repeat(100) {
vertx.setPeriodic(10) {
socket.write(
Buffer.buffer()
.appendInt(length)
.appendString(message)
)
}
}
delay(1000)
}
}
/**
* Based on: https://github.com/vert-x3/vertx-tcp-eventbus-bridge/blob/master/src/main/java/io/vertx/ext/eventbus/bridge/tcp/impl/protocol/FrameParser.java
*/
class FrameParser(
private val headerSize: Int,
private val headerHandler: (Buffer) -> Int,
private val handler: (Buffer) -> Unit
) : Handler<Buffer?> {
private var _buffer: Buffer? = null
private var _offset = 0
override fun handle(buffer: Buffer?) {
append(buffer)
var offset: Int
while (true) {
// set a rewind point. if a failure occurs,
// wait for the next handle()/append() and try again
offset = _offset
// how many bytes are in the buffer
val remainingBytes = bytesRemaining()
// at least expected header size
if (remainingBytes < headerSize) {
break
}
// what is the length of the message
val length: Int = headerHandler(_buffer!!.getBuffer(_offset, _offset + headerSize))
_offset += headerSize
if (remainingBytes - headerSize >= length) {
// we have a complete message
handler(_buffer!!.getBuffer(_offset, _offset + length))
_offset += length
} else {
// not enough data: rewind, and wait
// for the next packet to appear
_offset = offset
break
}
}
}
private fun append(newBuffer: Buffer?) {
if (newBuffer == null) {
return
}
// first run
if (_buffer == null) {
_buffer = newBuffer
return
}
// out of data
if (_offset >= _buffer!!.length()) {
_buffer = newBuffer
_offset = 0
return
}
// very large packet
if (_offset > 0) {
_buffer = _buffer!!.getBuffer(_offset, _buffer!!.length())
}
_buffer!!.appendBuffer(newBuffer)
_offset = 0
}
private fun bytesRemaining(): Int {
return if (_buffer!!.length() - _offset < 0) {
0
} else {
_buffer!!.length() - _offset
}
}
}
I hope to find any help on my old annoying problem.
I have a TCP sever program with java and client program with c#
packet protocol between those two is simply consist of 4byte length & body ASCII data.
The Problem is that C# client faces FormatException which is from parsing fail on length byte. If I look into an error from client side, then client is trying to parse somewhere in the body which is not length header.
But apparently, Server does not send broken packet.
meanwhile, at the server, I could find an Broken pipe error whenever this kind of problem happens.
Unfortunately this error does not always happen and was not able to recreate the problem situation. it makes me difficult to find exact cause of this problem
Please see below codes for server side
public class SimplifiedServer {
private Map<InetAddress, DataOutputStream> outMap;
private Map<InetAddress,DataInputStream> inMap;
protected void onAcceptNewClient(Socket client) {
DataOutputStream out = null;
DataInputStream in = null;
try {
out = new DataOutputStream(client.getOutputStream());
in = new DataInputStream(client.getInputStream());
} catch (IOException e) {
e.printStackTrace();
}
outMap.put(client.getInetAddress(), out);
inMap.put(client.getInetAddress(), in);
}
public void writeToAll(String packet) {
outMap.forEach((key, out) -> {
try {
byte[] body = packet.getBytes("UTF-8");
int len = body.length;
if (len > 9999) {
throw new IllegalArgumentException("packet length is longer than 10000, this try will be neglected");
}
String lenStr = String.format("%04d%s", len, packet);
byte[] obuf = lenStr.getBytes();
synchronized (out) {
out.write(obuf);
out.flush();
}
} catch (IOException e) {
e.printStackTrace();
}
});
}
public void listenClient(Socket client) {
try {
DataOutputStream out = outMap.get(client.getInetAddress());
DataInputStream in = inMap.get(client.getInetAddress());
while (true) {
byte[] received = SimplePacketHandler.receiveLpControlerData(in);
byte[] lenBytes = new byte[4];
for( int i = 0 ; i < 4 ; i ++){
lenBytes[i] = in.readByte();
}
String lenString = new String(lenBytes);
int length = Integer.parseInt(lenString);
byte[] data = new byte[length];
for ( int i = 0 ; i < length ; i ++){
data[i] = in.readByte();
}
if ( data == null ){
System.out.println("NetWork error, closing socket :" + client.getInetAddress());
in.close();
out.close();
outMap.remove(client.getInetAddress());
inMap.remove(client.getInetAddress());
return;
}
doSomethingWithData(out, data);
}
} catch (NumberFormatException e) {
e.printStackTrace();
} catch ( Exception e ) {
e.printStackTrace();
} finally {
try {
System.out.println(client.getRemoteSocketAddress().toString() + " closing !!! ");
// remove stream handler from map
outMap.remove(client.getInetAddress());
inMap.remove(client.getInetAddress());
//close socket.
client.close();
} catch (IOException e) {
e.printStackTrace();
}
}
}
}
And here is client side code
public class ClientSide
{
public TcpClient client;
public String ip;
public int port;
public NetworkStream ns;
public BinaryWriter writer;
public BinaryReader reader;
public Boolean isConnected = false;
public System.Timers.Timer t;
public String lastPacketSucceeded = String.Empty;
public ClientSide(String ip, int port)
{
this.ip = ip;
this.port = port;
client = new TcpClient();
}
public bool connect()
{
try
{
client.Connect(ip, port);
}
catch (SocketException e)
{
Console.WriteLine(e.ToString());
return false;
}
Console.WriteLine("Connection Established");
reader = new BinaryReader(client.GetStream());
writer = new BinaryWriter(client.GetStream());
isConnected = true;
return true;
}
public void startListen()
{
Thread t = new Thread(new ThreadStart(listen));
t.Start();
}
public void listen()
{
byte[] buffer = new byte[4];
while (true)
{
try
{
reader.Read(buffer, 0, 4);
String len = Encoding.UTF8.GetString(buffer);
int length = Int32.Parse(len);
byte[] bodyBuf = new byte[length];
reader.Read(bodyBuf, 0, length);
String body = Encoding.UTF8.GetString(bodyBuf);
doSomethingWithBody(body);
}
catch (FormatException e)
{
Console.WriteLine(e.Message);
}
}
}
public void writeToServer(String bodyStr)
{
byte[] body = Encoding.UTF8.GetBytes(bodyStr);
int len = body.Length;
if (len > 10000)
{
Console.WriteLine("Send Abort:" + bodyStr);
}
len = len + 10000;
String lenStr = Convert.ToString(len);
lenStr = lenStr.Substring(1);
byte[] lengthHeader = Encoding.UTF8.GetBytes(lenStr);
String fullPacket = lenStr + bodyStr;
byte[] full = Encoding.UTF8.GetBytes(fullPacket);
try
{
writer.Write(full);
}
catch (Exception)
{
reader.Close();
writer.Close();
client.Close();
reader = null;
writer = null;
client = null;
Console.WriteLine("Send Fail" + fullPacket);
}
Console.WriteLine("Send complete " + fullPacket);
}
}
Considering it is impossible to recreate problem, I would guess this problem is from multithread issue. but I could not find any further clue to fix this problem.
Please let me know if you guys need any more information to solve this out.
Any help will be great appreciated, thanks in advance.
A broken pipe exception is caused by closing the connection on the other side. Most likely the C# client has a bug, causing the format exception which causes it to close the connection and therefore the broken pipe on the server side. See what is the meaning of Broken pipe Exception?.
Check the return value of this read:
byte[] bodyBuf = new byte[length];
reader.Read(bodyBuf, 0, length);
According to Microsoft documentation for BinaryReader.Read https://msdn.microsoft.com/en-us/library/ms143295%28v=vs.110%29.aspx
[The return value is ] The number of bytes read into buffer. This might be less than the number of bytes requested if that many bytes are not available, or it might be zero if the end of the stream is reached.
If it reads less than the length bytes then next time it will be parsing the length using data somewhere in the middle of the last message.
These broke pipe exceptions happen when the client (browser) has closed the connection, but the server (your tag) continues to try to write to the stream.
This usually happens when someone clicks Back, Stop, etc. in the browser and it disconnects from the server before the request is finished. Sometimes, it can happen because, for example, the Content-Length header is incorrect (and the browser takes its value as true).
Usually, this is a non-event, and nothing to worry about. But if you are seeing them in your dev environment when you know you have not interrupted your browser, you might dig a bit more to find out why.
WLS server will try to filter these exceptions from the web container out of the log, since it is due to client (browser) action and we can't do anything about it. But the server doesn't catch all of them.
refer from :: https://community.oracle.com/thread/806884
I'm trying to read a webpage using following code :
URL url = new URL("somewebsitecomeshere");
URLConnection c = url.openConnection();
if(getHttpResponseCode(c) == 200)
{
if (isContentValid(c))//accept html/xml only!
{
InputStream is = c.getInputStream();
Reader r = new InputStreamReader(is);
System.out.println(r.toString());
//after commenting this everything works great!
setHTMLString(getStringFromReader(r));
System.out.println(getHTMLString());
ParserDelegator parser = new ParserDelegator();
parser.parse(r, new Parser(url), true);
r.close();
is.close();
try {
Thread.sleep(500);
} catch (InterruptedException e) {
// TODO Auto-generated catch block
e.printStackTrace();
}
}
else
log("content is not valid!");
}
else
{
System.out.println("ERROR" + c.getContentType() + c.getURL());
}
//---------------------------------------------------
private String getStringFromReader(Reader reader) throws IOException {
char[] arr = new char[8*1024]; // 8K at a time
StringBuffer buf = new StringBuffer();
int numChars;
while ((numChars = reader.read(arr, 0, arr.length)) > 0) {
buf.append(arr, 0, numChars);
}
//Reset position to 0
reader.reset();
return buf.toString();
}
if try to read string using getStringFromReader() the rest of the code will be ignored due to changing position of Reader to EOF so I tried to reset the position to 0 but I got the following error :
java.io.IOException: reset() not supported
at java.io.Reader.reset(Unknown Source)
at sample.getStringFromReader(Spider.java:248)
at default(sample.java:286)
at default.main(sample.java:130)
How can I reset the Reader position to 0?
Short answer, your stream doesn't support reset or mark methods. Check the result of:
is.markSupported()
Long answer, an InputStream is a flow of bytes. Bytes can come from a file, a network resource, a string, etc. So basically, there are streams that don't support resetting the reader position to the start of the stream, while others do (random access file).
A stream from a web site will normally use underlying network connection to provide the data. It means that it's up to the underlying network protocol (TCP/IP for example) to support or not resetting the stream, and normally they don't.
In order to reset any stream you would have to know the entire flow, from start to end. Network communications send a bunch of packages (which may be in order or not) to transfer data. Packages may get lost or even be duplicated, so normally information is buffered and interpreted as it is received. It would be very expensive to reconstruct all messages at network level. So that is normally up to the receiver, if it wants to do that.
In your case If what you want is print the input stream I would recommend creating a custom InputStream, which receives the original InputStream and whenever it is read it prints the read value and returns it at the same time. For example:
class MyInputStream extends InputStream {
InputStream original = null;
public MyInputStream(InputStream original) {
this.original = original;
}
#Override
public int read() throws IOException {
int c = original.read();
System.out.printf("%c", c);
return c;
}
}
Then wrap your original InputStream with that:
.
.
.
InputStream myIs = new MyInputStream(is);
Reader r = new InputStreamReader(myIs);
.
.
.
Hope it helps.
InputStreamReader does not support reset(). Also, you did not call mark(0) before.
What you could do is wrap your reader in a BufferedReader of a sufficient size so that reset is supported. If you cannot do that, then you should try to open a new connection to your URL.
My application loops through about 200 urls that are all jpg images.
In the simulator it reads ok, then stores the byte array in persistentStore with no problems.
On the device, it gives java.io.IOException: TCP read timed out on basically every image.
Every now and then, one gets through. Not even sure how. The image sizes don't give insight either. Some are 6k, some are 11k. Size doesn't seem to matter for timing out.
I'll try to post what I believe to be the relevant code, but I am not really an expert here, so if I left something out, please say so.
Call http connection through loop and join thread:
for(int i = 0; i < images.size(); i ++)
{
try {
String url = images.elementAt(i).toString();
HttpRequest data3 = new HttpRequest(url, "GET", false);
data3.start();
data3.join();
} catch (IOException e) {
Dialog.inform("wtf " + e);
}
}
Make the actual connection in HttpConnection class with the proper suffix:
try
{
HttpConnection connection = (HttpConnection)Connector.open(url + updateConnectionSuffix());
int responseCode = connection.getResponseCode();
if(responseCode != HttpConnection.HTTP_OK)
{
connection.close();
return;
}
String contentType = connection.getHeaderField("Content-type");
long length = connection.getLength();
InputStream responseData = connection.openInputStream();
connection.close();
outputFinal(responseData, contentType, length);
}
catch(IOException ex)
{
} catch (SAXException ex) {
} catch (ParserConfigurationException ex) {
}
Finally, read the stream and write the bytes to a byte array:
else if(contentType.equals("image/png") || contentType.equals("image/jpeg") || contentType.equals("image/gif"))
{
try
{
if((int) length < 1)
length = 15000;
byte[] responseData = new byte[(int) length];
int offset = 0;
int numRead = 0;
StringBuffer rawResponse = new StringBuffer();
int chunk = responseData.length-offset;
if(chunk < 1)
chunk = 1024;
while (offset < length && (numRead=result.read(responseData, offset, chunk)) >= 0){
rawResponse.append(new String(responseData, offset, numRead));
offset += numRead;
}
String resultString = rawResponse.toString();
byte[] dataArray = resultString.getBytes();
result.close();
database db = new database();
db.storeImage(venue_id, dataArray);
}
catch( Exception e )
{
System.out.println(">>>>>>>----------------> total image fail: " + e);
}
}
Things to consider:
Length is always byte length in simulator. In device it is always -1.
The chunk var is a test to see if I force a 15k byte array, will it try to read as expected since byte[-1] gave an out of bounds exception. The results are the same. Sometimes it writes. Mostly it times out.
Any help is appreciated.
You can adjust the length of TCP timeouts on Blackberry using the parameter 'ConnectionTimeout'.
In your code here:
HttpConnection connection = (HttpConnection)Connector.open(url + updateConnectionSuffix());
You'll want to append ConnectionTimeout. You might write it into updateConnectionSuffix() or just append it.
HttpConnection connection = (HttpConnection)Connector.open(url + updateConnectionSuffix() + ";ConnectionTimeout=54321");
This sets the timeout to 54321 milliseconds.
Timeouts occur when the client is waiting for the server to send an ack and it doesn't get one in a specified amount of time.
edit: also, are you able to use the browser and stuff? You may also want to play with the deviceside parameter.
I think the problem may be that you're closing the connection before reading the bytes from the input stream. Try moving the connection.close() after the bytes have been read in.