We Keep Coding

Where do you close a writer in TwitterStream?

I'm working on compiling a bunch of tweets for an information retrieval class. I'm trying this using both the REST API and the Streaming API through twitter4j. When using the Streaming API, I use the following modifications to this example:
final LimitedFileWriter output = new LimitedFileWriter("Tweets","tweets");
TwitterStream twitterStream = new TwitterStreamFactory(cb.build()).getInstance();
StatusListener listener = new StatusListener() {
#Override
public void onStatus(Status status) {
try{
output.write("#" + status.getUser().getScreenName() + " -- " + status.getText()+"\n");
}
catch(IOException e){
e.printStackTrace();
}
}
}
twitterStream.addListener(listener);
twitterStream.sample("en");
//output.close();
It seems I can't ever close my writer. The writer I am using simply wraps BufferedWriter, while keeping track of file size. If the file exceeds a certain size (128MB), the writer will close the current file and create a new file. Here are the relevant class functions:
public void write(String s) throws IOException
{
if(bytesWritten + s.getBytes(charset).length >= MAXSIZE){
output.close();
bytesWritten = 0;
fileNum++;
String fileName = directory + "/" + baseName+fmt.format(fileNum);
currentFile = new File(fileName);
output = new BufferedWriter
(new OutputStreamWriter(new FileOutputStream(fileName),charset));
}
output.write(s);
bytesWritten += s.getBytes(charset).length;
}
public void close() throws IOException{
output.close();
}
If I try to close the writer after twitterStream.sample() (commented out), the program crashes because I am trying to write to a closed file. If my understanding is correct, this is because the TwitterStream class creates a new thread which runs concurrently with the main thread. Then, the main thread closes the stream and the twitterStream can no longer write to it.
If that's the case, where should I close my writer?

If I have understood your question correctly, you want to be able to turn of the tweets collection at some point, close your open file writers and have a clean exit. To achieve it you can use a synchronized block.
final Object lock = new Object();
final LimitedFileWriter output = new LimitedFileWriter("Tweets","tweets");
TwitterStream twitterStream = new TwitterStreamFactory(cb.build()).getInstance();
StatusListener listener = new StatusListener() {
#Override
public void onStatus(Status status) {
try{
output.write("#" + status.getUser().getScreenName() + " -- " + status.getText()+"\n");
// free the lock
if (some_condition_like_I_have_enough_files) {
synchronized (lock) {
lock.notify();
}
}
catch(IOException e){
e.printStackTrace();
}
}
}
twitterStream.addListener(listener);
twitterStream.sample("en");
try {
synchronized (lock) {
lock.wait();
}
} catch (InterruptedException e) {
e.printStackTrace();
}
// close the twitterstream
// close the writer

read and write files in java using separate threads

I have created two threads and modified the run function so that one thread reads one line and the other writes the same line to the new file. This happens till the whole file is copied. The problem i am getting is that even though i have used variables to control that the threads execute one by one but still the threads are executing unevenly i.e one thread executes multiple times and then the control transfers. Any solutions i have attached the code. I am new to java as it is only for class assignment so the code might not be the most optimized.
public class thread1 extends Thread {
//To create producer and consumer as threads
//Shared variable
public static int x = 0;//checks if all lines are read
public static String line; /holds lines from file
public static int j = 0;//variable to switch between threads based upon its value
public thread1(String threadName) { //Constuctor
super(threadName); //Call to constructor of Thread class
}
public void run() {
while (x != -1)
{
if (Thread.currentThread().getName().contains("Reader")) {
if (x != -1&&j==0)
{
j=1;
String fileName = "d:/salfar.txt";
try {
// FileReader reads text files in the default encoding.
FileReader fileReader =
new FileReader(fileName);
// Always wrap FileReader in BufferedReader.
BufferedReader bufferedReader =
new BufferedReader(fileReader);
for (int check = 0; check <= x; check++) {
line = bufferedReader.readLine();
}
if (line == null) {
x = -1;
} else {
System.out.println(line);
x++;
}
// Always close files.
bufferedReader.close();
} catch (FileNotFoundException ex) {
System.out.println(
"Unable to open file '"
+ fileName + "'");
} catch (IOException ex) {
System.out.println(
"Error reading file '"
+ fileName + "'");
// Or we could just do this:
// ex.printStackTrace();
}
}
yield();
}
else if (Thread.currentThread().getName().contains("writer")) {
if (x != -1 && line != null&&j==1)
{
j=0;
String fileName = "d:/salfar1.txt";
try {
// Assume default encoding.
FileWriter fileWriter =
new FileWriter(fileName, true);
// Always wrap FileWriter in BufferedWriter.
BufferedWriter bufferedWriter =
new BufferedWriter(fileWriter);
// Note that write() does not automatically
// append a newline character.
bufferedWriter.write(line);
bufferedWriter.newLine();
System.out.println("y");
// Always close files.
bufferedWriter.close();
} catch (IOException ex) {
System.out.println(
"Error writing to file '"
+ fileName + "'");
// Or we could just do this:
// ex.printStackTrace();
}
}
Thread.yield();
}
else{}
}
}
public static void main(String[] args) {
thread1 p = new thread1("Reader");
thread1 c = new thread1("writer");
p.start();
c.start();
}
}
Thanks

You cannot control the order of thread execution. However, to perform read and write operation via separate threads, you should use BlockingQueue which has the following properties:
A Queue that additionally supports operations that wait for the queue to become non-empty when retrieving an element, and wait for space to become available in the queue when storing an element.
ReaderThread will read from the input file.
import java.io.BufferedReader;
import java.io.File;
import java.io.FileNotFoundException;
import java.io.FileReader;
import java.io.IOException;
import java.util.concurrent.BlockingQueue;
public class ReaderThread implements Runnable{
protected BlockingQueue<String> blockingQueue = null;
public ReaderThread(BlockingQueue<String> blockingQueue){
this.blockingQueue = blockingQueue;
}
#Override
public void run() {
BufferedReader br = null;
try {
br = new BufferedReader(new FileReader(new File("./inputFile.txt")));
String buffer =null;
while((buffer=br.readLine())!=null){
blockingQueue.put(buffer);
}
blockingQueue.put("EOF"); //When end of file has been reached
} catch (FileNotFoundException e) {
e.printStackTrace();
} catch (IOException e) {
e.printStackTrace();
} catch(InterruptedException e){
}finally{
try {
br.close();
} catch (IOException e) {
e.printStackTrace();
}
}
}
}
WriterThread will write to output file.
import java.io.File;
import java.io.FileNotFoundException;
import java.io.PrintWriter;
import java.util.concurrent.BlockingQueue;
public class WriterThread implements Runnable{
protected BlockingQueue<String> blockingQueue = null;
public WriterThread(BlockingQueue<String> blockingQueue){
this.blockingQueue = blockingQueue;
}
#Override
public void run() {
PrintWriter writer = null;
try {
writer = new PrintWriter(new File("outputFile.txt"));
while(true){
String buffer = blockingQueue.take();
//Check whether end of file has been reached
if(buffer.equals("EOF")){
break;
}
writer.println(buffer);
}
} catch (FileNotFoundException e) {
e.printStackTrace();
} catch(InterruptedException e){
}finally{
writer.close();
}
}
}
From Launcher class start your multithreaded read and write.
import java.util.concurrent.ArrayBlockingQueue;
import java.util.concurrent.BlockingQueue;
public class Launcher {
public static void main(String[] args) {
BlockingQueue<String> queue = new ArrayBlockingQueue<String>(1024);
ReaderThread reader = new ReaderThread(queue);
WriterThread writer = new WriterThread(queue);
new Thread(reader).start();
new Thread(writer).start();
}
}

Here is my solutions. My idea is to use the actually file name that our threads will be reading and writing to. There is only one issue that we need to make sure of, that no two threads are trying to operate on the same file. The solution to this is to simply have a synchronized code block in your run method.
We must recall that Strings are not mutable in Java. Consider the following:
String s1 = "test.txt";
String s2 = "test.txt";
Now, we must ask our selves how does the jvm reuse the immutable "test.txt". In this case both s1 & s2 String objects point to the same "test.txt".
Understanding this concept will also do the trick for us:
public class Client {
public static void main( String args [] ) {
String filename = "test.txt";
String filename2 = "test.txt";
Reader reader = new Reader( filename ) ;
Writer writer = new Writer( filename2 ) ;
while(true) {
reader.run();
writer.run();
}
}
}
public class Writer implements Runnable {
public String filename;
public Writer( String filename ) {
this.filename = filename;
}
#Override
public void run() {
synchronized( this.filename ) {
System.out.println( "writing to a file:" + this.filename );
}
}
}
public class Reader implements Runnable {
public String filename;
public Reader( String filename ) {
this.filename = filename;
}
#Override
public void run() {
synchronized( this.filename ) {
System.out.println( "reading a file:" + this.filename );
}
}
}

Java chatprogram, infinit loop stops

I have to build a chat program.
There is the server class, the client class and two threads to write and receive messages.
the two threads should run in an infinite loop and check all the time if there is an input and print that input afterwards.
But my program works for just one round. So the server and the client can write one single message, afterwards it stops and does not check for another message. Why does the thread not start again from the begin when it's never interrupted? --> see the code beneath
I hope you know what my problem is, it's quite hard for me to describe.
Thread to read a new Message
public class MsgWriter extends Thread {
private Socket s;
public MsgWriter(Socket s){
this.s = s;
}
public void run(){
int i = 0;
OutputStream out = null;
PrintWriter writer;
Scanner input;
while(!interrupted()){
try{
synchronized(s){
input = new Scanner (System.in);
out = s.getOutputStream();
writer = new PrintWriter(out);
String toserver = input.nextLine();
writer.write(toserver);
writer.flush();
System.out.println("me: " + toserver);
}
try {
Thread.sleep((int) (100 * Math.random()));
} catch (InterruptedException e) {
// TODO Auto-generated catch block
e.printStackTrace();
}
}catch(Exception e) {
}
}
}
}
Thread to check if there is a new message and prints it.
public class MsgReader extends Thread {
Socket s;
public MsgReader(Socket s){
this.s = s;
}
public void run() {
int i = 0;
while (!interrupted()) {
try{
synchronized(s){
InputStream in = s.getInputStream();
BufferedReader reader = new BufferedReader(new InputStreamReader(in));
String s = null;
while((s=reader.readLine()) != null){
System.out.println("d");
}
}
try {
Thread.sleep((int) (100 * 1));
} catch (InterruptedException e) {
// TODO Auto-generated catch block
e.printStackTrace();
}
}catch (Exception e){
}
}
}
}
The Server class starts a new server and waits for a client, afterwards it starts the two threads. The same with the client class, it connects to the server and starts the threads.

You're probably thowing an exception somewhere. In your catch blocks, print the error.
try {
} catch (Exception e) {
System.out.println("Error: " + e);
}

synchronizing file output in a static class

I have several threads which need to write to two different text files. So far I have this code:
public class Logger {
public static void printToGameLog(String value){
Writer writer = null;
try {
writer = new BufferedWriter(new OutputStreamWriter(
new FileOutputStream("GameLog.txt", true), "utf-8"));
synchronized(writer){
writer.write(outputString + "\r\n");
}
} catch (IOException ex){
System.out.println("cannot create log file");
}
}
public static void printToServerLog(String value){
Writer writer = null;
try {
writer = new BufferedWriter(new OutputStreamWriter(
new FileOutputStream("serverLog.txt", true), "utf-8"));
synchronized(writer){
writer.write(outputString + "\r\n");
}
} catch (IOException ex){
System.out.println("cannot create log file");
}
}
}
Is this an acceptable way of ensuring no more than one thread is writing to the same file at the same time?
If a thread calls one of these methods and enters the sync block, then what happens if another thread comes along and tries to execute the same method. When it tries to use the local variable writer, will it try and obtain the same object that has been locked by the other thread and therefore block? I would have thought that it would simply create its own separate variable, which would mean I should make writer a static class variable instead?

Since there are separate log files, I don't see why you need to have class-level synchronization. Seems like a needless bottleneck. I'd provide sync for each method separately (since it's fine for them to hit separate files simultaneously):
public class Logger
{
private static final Object GAME_LOG_LOCK = new Object();
private static final Object SERVER_LOG_LOCK = new Object();
public static void printToGameLog(String value){
synchronized (GAME_LOG_LOCK) {
Writer writer = null;
try {
writer = new BufferedWriter(new OutputStreamWriter(
new FileOutputStream("GameLog.txt", true), "utf-8"));
writer.write(outputString + "\r\n");
} catch (IOException ex){
System.out.println("cannot create log file");
}
}
}
public static void printToServerLog(String value){
synchronized (SERVER_LOG_LOCK) {
Writer writer = null;
try {
writer = new BufferedWriter(new OutputStreamWriter(
new FileOutputStream("serverLog.txt", true), "utf-8"));
writer.write(outputString + "\r\n");
} catch (IOException ex){
System.out.println("cannot create log file");
}
}
}
}

That's a null pointer exception in your code, try this way of using the synchronized block on a static method
synchronized(Logger.class){
or another alternative is to set the whole methods synchronized, like this
public static synchronized void printToGameLog(String value){
and
public static synchronized void printToServerLog(String value){
I'm not convinced that you need synchronization in here, you only need synchronization if you have a state that is being read/written from multiple threads.

Here is another take on your problem. It uses a single thread to write the log file and only this thread has access to the files. The threads that have to log something write against a BlockingQueue:
public class ThreadedLog {
//This is some code to test the logger
public static void main(String[] args) throws UnsupportedEncodingException, FileNotFoundException {
ThreadedLog log = new ThreadedLog("/tmp/test.txt");
// Start 100 thread that write against the log
for (int i = 0; i < 100; i++) {
new Thread(new TestLogger(log)).start();
}
}
private static class TestLogger implements Runnable {
private ThreadedLog log;
public TestLogger(ThreadedLog log) {
this.log = log;
}
#Override
public void run() {
for (int i = 0; i < 5000; i++) {
try {
log.log("This is entry " + i + " from thread " + Thread.currentThread().getId());
} catch (InterruptedException ex) {
}
}
System.out.println(Thread.currentThread().getId() + " is done");
}
}
//________________________________________________________________________________________________
/*
* This is the code for the actual logger
*
*/
private final BlockingQueue<String> queue = new ArrayBlockingQueue<>(10000);
private String fileName;
private Thread thread;
private Writer writer;
public ThreadedLog(String fileName) throws UnsupportedEncodingException, FileNotFoundException {
this.fileName = fileName;
thread = new Thread(new LoggingThread());
writer = new BufferedWriter(new OutputStreamWriter(
new FileOutputStream(fileName, true), "utf-8"));
thread.start();
}
private class LoggingThread implements Runnable {
#Override
public void run() {
try {
for (;;) {
ThreadedLog.this.writer.write(queue.take() + "\r\n");
ThreadedLog.this.writer.flush();
}
} catch (InterruptedException | IOException e) {
e.printStackTrace();
try {
ThreadedLog.this.writer.close();
} catch (Exception ex) {
}
}
}
}
public void log(String string) throws InterruptedException {
queue.put(string);
}
}

how to read from standard input non-blocking?

long end=System.currentTimeMillis()+60*10;
InputStreamReader fileInputStream=new InputStreamReader(System.in);
BufferedReader bufferedReader=new BufferedReader(fileInputStream);
try
{
while((System.currentTimeMillis()<end) && (bufferedReader.readLine()!=null))
{
}
bufferedReader.close();
}
catch(java.io.IOException e)
{
e.printStackTrace();
}
I actually tried doing the above for reading in 600 miliseconds time after which it should not allow reading but the readline of the bufferedreader is blocking.Please help

Using BufferedReader.available() as suggested by Sibbo isn't reliable. Documentation of available() states:
Returns an estimate of the number of bytes that can be read... It is never correct to use the return value of this method to allocate a buffer.
In other words, you cannot rely on this value, e.g., it can return 0 even if some characters are actually available.
I did some research and unless you are able to close the process input stream from outside, you need to resort to an asynchronous read from a different thread. You can find an example how to read without blocking line by line here.
Update: Here is a simplified version of the code from the link above:
public class NonblockingBufferedReader {
private final BlockingQueue<String> lines = new LinkedBlockingQueue<String>();
private volatile boolean closed = false;
private Thread backgroundReaderThread = null;
public NonblockingBufferedReader(final BufferedReader bufferedReader) {
backgroundReaderThread = new Thread(new Runnable() {
#Override
public void run() {
try {
while (!Thread.interrupted()) {
String line = bufferedReader.readLine();
if (line == null) {
break;
}
lines.add(line);
}
} catch (IOException e) {
throw new RuntimeException(e);
} finally {
closed = true;
}
}
});
backgroundReaderThread.setDaemon(true);
backgroundReaderThread.start();
}
public String readLine() throws IOException {
try {
return closed && lines.isEmpty() ? null : lines.poll(500L, TimeUnit.MILLISECONDS);
} catch (InterruptedException e) {
throw new IOException("The BackgroundReaderThread was interrupted!", e);
}
}
public void close() {
if (backgroundReaderThread != null) {
backgroundReaderThread.interrupt();
backgroundReaderThread = null;
}
}
}

You could check with BufferedReader.available() > 0 if there are chars to read.
String s;
while((System.currentTimeMillis()<end))
{
if (bufferedReader.available() > 0)
s += bufferedReader.readLine();
}
bufferedReader.close();

long end=System.currentTimeMillis()+60*10;
InputStreamReader fileInputStream = new InputStreamReader(System.in);
BufferedReader bufferedReader = new BufferedReader(fileInputStream);
try {
while ((System.currentTimeMillis() < end)) {
if (bufferedReader.ready()) {
System.out.println(bufferedReader.readLine());
}
}
} catch (IOException e) {
e.printStackTrace();
} finally {
try {
if (bufferedReader != null) {
bufferedReader.close();
}
} catch (IOException e) {
e.printStackTrace();
}
}

The only reliable way would be to start a worker thread and do the actual reading inside it, while the caller thread would monitor the latency.
If the worker thread is waiting longer that allowed, the master thread would terminate it and throw an exception.

BufferReader.readLine() can block for a very long time if a line is extremely long like 1M chars.
Does your file contains such long lines?
If yes, you may have to break up the lines, or use per-char read methods like BufferReader.read().