I want to implement my own logger that write logs in file. It could be run from many threads and an issue is how to synchronized access to log file.
private synchronized static void writeToFile(String tag, String msg,
Throwable tr, Context ctx) {
try {
String s = System.getProperty("line.separator");
File f = Environment.getExternalStorageDirectory();
Log.i(TAG, "Path to app " + f.getAbsolutePath());
File l = new File(f, "log.txt");
if (!l.exists()) {
l.createNewFile();
}
String e = Log.getStackTraceString(tr);
StringBuilder b = new StringBuilder();
b.append(HttpCommand.getDateForUrl(System.currentTimeMillis()));
b.append(tag);
b.append(msg);
b.append(e);
b.append(s);
OutputStream out = new FileOutputStream(l);
out.write(b.toString().getBytes());
out.flush();
out.close();
} catch (IOException e) {
Log.e(TAG, "Failed to create backup");
}
}
Is it enough to sync access to database with sync by class if I pass it in different threads?
synchronized(X.class) {
writeTiFile()
}
Since your writeToFile() function is synchronized, you don't have to wrap any calls to it with the synchronized {} block.
See here for more info:
http://docs.oracle.com/javase/tutorial/essential/concurrency/locksync.html
You can use ConcurrentLinkedQueue, put all messages to Queue and create thread that take from queue and write it to file.
Related
I am reading files from HDFS directory using multi-threading using a Producer-Consumer model, leveraging BlockingQueue.
Here is my code;
producer class:
public void readURLS() {
final int capacity = Integer.MAX_VALUE;
BlockingQueue<String> queue = new LinkedBlockingQueue<>(capacity);
try {
FileSystem hdfs = FileSystem.get(hadoopConf);
FileStatus[] status = hdfs.listStatus(new Path("MYHDFS_PATH"));
int i = 0;
for (FileStatus file : status) {
LOG.info("Thread {} started: ", i++);
LOG.info("Reading file {} ", file.getPath().getName());
new Thread(new FetchData(queue, file.getPath(), hadoopConf)).start();
}
} catch (IOException e) {
LOG.error("IOException occured while listing files from HDFS directory");
}
}
FetchData:
#Override
public void run() {
LOG.info("Inside reader to start reading the files ");
try (BufferedReader bufferedReader =
new BufferedReader(new InputStreamReader
(FileSystem.get(hadoopConf).open(file), StandardCharsets.UTF_8))) {
String line;
while ((line = bufferedReader.readLine()) != null) {
if (Thread.interrupted()) {
throw new InterruptedException();
}
LOG.info("Line is :{}", line);
queue.put(line);
}
} catch (IOException e) {
LOG.error("file : {} ", file.toString());
throw new IOException(e);
} catch (InterruptedException e) {
LOG.error("An error has occurred: ", e);
Thread.currentThread().interrupt();
}
While executing the code it throws me InterruptedIOException:
java.io.IOException: Failed on local exception: java.io.**InterruptedIOException**: Interruped while waiting for IO on channel java.nio.channels.SocketChannel[connected
Any idea why. My idea is to loop over each file and read each file using a separate thread.
I'm also getting same behavior when using HDFS from multiple (many!) threads, and do not know the answer to the question "why?", but keeping the number of threads accessing HDFS concurrently seems to help.
In your case I would recommend to use an ExecutorService with limited number of threads, and fine-tune that number to the limit when you do not get exceptions.
So, create the ExecutorService (with 10 threads as a starting point):
final ExecutorService executorService = Executors.newFixedThreadPool(10);
and instead of your
new Thread(new FetchData(queue, file.getPath(), hadoopConf)).start();
do
executorService.submit(new FetchData(queue, file.getPath(), hadoopConf));
Another improvement is since org.apache.hadoop.fs.FileSystem implements Closeable, you should close it. In your code every thread creates a new instance of FileSystem, but does not close it. So I would extract it into a variable inside your try:
try (FileSystem fileSystem = FileSystem.get(hadoopConf);
BufferedReader bufferedReader =
new BufferedReader(new InputStreamReader
(fileSystem.open(file), StandardCharsets.UTF_8))) {
UPDATE:
Although the above code seems to be the right approach for the Closeable objects, by default FileSystem.get will return cached instances from the
/** FileSystem cache */
static final Cache CACHE = new Cache();
and thus things will break horribly when close() will be called on them.
You could either disable the FileSystem cache by setting fs.hdfs.impl.disable.cache config param to true, or make sure the FileSystem instance(s) only closed when all workers have finished. It also seems that you could just use a single instance of FileSystem for all your workers, although I can't find any confirmation in javadocs that this will work properly without extra synchronisation.
I'm trying to download a pdf file using URLConnection. Here's how I setup the connection object.
URL serverUrl = new URL(url);
urlConnection = (HttpURLConnection) serverUrl.openConnection();
urlConnection.setDoInput(true);
urlConnection.setRequestMethod("GET");
urlConnection.setRequestProperty("Content-Type", "application/pdf");
urlConnection.setRequestProperty("ENCTYPE", "multipart/form-data");
String contentLength = urlConnection.getHeaderField("Content-Length");
I obtained inputstream from the connection object.
bufferedInputStream = new BufferedInputStream(urlConnection.getInputStream());
And the output stream to write the file contents.
File dir = new File(context.getFilesDir(), mFolder);
if(!dir.exists()) dir.mkdir();
final File f = new File(dir, String.valueOf(documentName));
f.createNewFile();
final BufferedOutputStream bufferedOutputStream = new BufferedOutputStream(new FileOutputStream(f, true)); //true for appendMode
BlockingQueue is created so that threads performing read and write operations can access the queue.
final BlockingQueue<ByteArrayWrapper> blockingQueue = new ArrayBlockingQueue<ByteArrayWrapper>(MAX_VALUE,true);
final byte[] dataBuffer = new byte[MAX_VALUE];
Now created thread to read data from InputStream.
Thread readerThread = new Thread(new Runnable() {
#Override
public void run() {
try {
int count = 0;
while((count = bufferedInputStream.read(dataBuffer, 0, dataBuffer.length)) != -1) {
ByteArrayWrapper byteArrayWrapper = new ByteArrayWrapper(dataBuffer);
byteArrayWrapper.setBytesReadCount(count);
blockingQueue.put(byteArrayWrapper);
}
blockingQueue.put(null); //end of file
} catch(Exception e) {
e.printStackTrace();
} finally {
try {
bufferedInputStream.close();
} catch (IOException e) {
e.printStackTrace();
}
}
}
});
Now the writer thread reads those file contents.
Thread writerThread = new Thread(new Runnable() {
#Override
public void run() {
try {
while(true) {
ByteArrayWrapper byteWrapper = blockingQueue.take();
if(null == byteWrapper) break;
bufferedOutputStream.write(byteWrapper.getBytesRead(), 0, byteWrapper.getBytesReadCount());
}
bufferedOutputStream.flush();
} catch(Exception e) {
e.printStackTrace();
} finally {
try {
bufferedOutputStream.close();
} catch (IOException e) {
e.printStackTrace();
}
}
}
});
Finally, threads are started.
readerThread.start();
writerThread.start();
Theoretically it should read the file from InputStream and save it to the target file. However, in reality, it produces blank pdf file. At some other time, it shows invalid pdf format exception. File size matches with content length of the InputStream. Is there anything I'm missing?
I'm not familiar with ByteArrayWrapper. Does it just hold a reference to the array, like this?
public class ByteArrayBuffer {
final private byte[] data;
public ByteArrayBuffer(byte[] data) {
this.data = data;
}
public byte[] getBytesRead() {
return data;
}
/*...etc...*/
}
If so. that would be the problem: all of the ByteArrayWrapper objects are backed by the same array. Which is repeatedly overwritten by the writer. Even though BlockingQueue did the hard work of safely publishing each object from one thread to the other.
The simplest fix might be to make the ByteArrayWrapper effectively immutable i.e. don't change it after publishing it to another thread. Taking a copy of the array on construction would be simplest:
public ByteArrayWrapper(byte[] data) {
this.data = Arrays.copyOf(data, data.length);
}
One other problem is that "BlockingQueue does not accept null elements" (see BlockingQueue docs), and so the "end of input" sentinel value doesn't work. Replacing null with a
private static ByteArrayWrapper END = new ByteArrayWrapper(new byte[]{});
in the appropriate places will fix that.
By making those changes to a copy of the code I was able to retrieve a faithful copy of a PDF file.
Try to use Android DownloadManager (http://developer.android.com/reference/android/app/DownloadManager.html) it is used to handle long-running HTTP requests in the background.
Here you don't need to think about received bytes and the progress is displayed in the notification bar.
There is a good tutorial here: http://blog.vogella.com/2011/06/14/android-downloadmanager-example/
I am using a ExecutorService to have multiple threads writing text into a file, but i cannot manage to synchronize the run() method and instead of having the proper line by line String i ask, i have a mixup of all the characters of the Strings because they write it at the same time.
import java.io.BufferedReader
...
class WriteDns implements Runnable {
File file;
String text;
WriteDns(File file, String text) {
this.file = file;
this.text = text;
}
public void run() {
synchronized (this) {
try (BufferedWriter bw = new BufferedWriter(new OutputStreamWriter(
new FileOutputStream(file)))) {
bw.write(turnDns() + "\n");
} catch (IOException e) {
System.out.println("Error");
}
}
}
public String turnDns() {
int space = text.indexOf(' ');
String ip = text.substring(0, space);
String theRest = text.substring(space);
String temp = ip;
try {
ip = InetAddress.getByName(ip).getHostName();
if (ip == temp)
return "NotFound " + theRest;
return ip + " " + theRest;
} catch (UnknownHostException e) {
System.out.println("Error in change");
return "-changeErr " + theRest;
}
}
}
public class Main00 {
static File oldFile = new File("oldfile.txt");
public static void main(String[] args) {
readLines();
}
public static void readLines() {
try (BufferedReader br = new BufferedReader(new FileReader(oldFile))) {
File f = new File("file.txt");
ExecutorService service = Executors.newFixedThreadPool(10);
for (String t = br.readLine(); t != null; t = br.readLine()) {
service.execute(new WriteDns(f, t));
}
service.shutdown();
} catch (IOException e) {
e.printStackTrace();
}
}
}
You're synchronising on this but you're making a new instance of your thread worker for every thread, so each thread is locking on itself and never waiting for any other threads. You need to lock on an object that is visible to all threads, perhaps a static object or pass in a lock object when you instantiate your WriteDns.
With that said, having multiple threads open on one file is inherently prone to problems like you're experiencing, and you gain nothing from multiple threads writing since your bottleneck is your storage medium and not your processor. You should rather have multiple threads providing information/data to one dedicated writer thread that has exclusive access to the file you want to write to, as #FlorianSchaetz suggested.
I'm working on compiling a bunch of tweets for an information retrieval class. I'm trying this using both the REST API and the Streaming API through twitter4j. When using the Streaming API, I use the following modifications to this example:
final LimitedFileWriter output = new LimitedFileWriter("Tweets","tweets");
TwitterStream twitterStream = new TwitterStreamFactory(cb.build()).getInstance();
StatusListener listener = new StatusListener() {
#Override
public void onStatus(Status status) {
try{
output.write("#" + status.getUser().getScreenName() + " -- " + status.getText()+"\n");
}
catch(IOException e){
e.printStackTrace();
}
}
}
twitterStream.addListener(listener);
twitterStream.sample("en");
//output.close();
It seems I can't ever close my writer. The writer I am using simply wraps BufferedWriter, while keeping track of file size. If the file exceeds a certain size (128MB), the writer will close the current file and create a new file. Here are the relevant class functions:
public void write(String s) throws IOException
{
if(bytesWritten + s.getBytes(charset).length >= MAXSIZE){
output.close();
bytesWritten = 0;
fileNum++;
String fileName = directory + "/" + baseName+fmt.format(fileNum);
currentFile = new File(fileName);
output = new BufferedWriter
(new OutputStreamWriter(new FileOutputStream(fileName),charset));
}
output.write(s);
bytesWritten += s.getBytes(charset).length;
}
public void close() throws IOException{
output.close();
}
If I try to close the writer after twitterStream.sample() (commented out), the program crashes because I am trying to write to a closed file. If my understanding is correct, this is because the TwitterStream class creates a new thread which runs concurrently with the main thread. Then, the main thread closes the stream and the twitterStream can no longer write to it.
If that's the case, where should I close my writer?
If I have understood your question correctly, you want to be able to turn of the tweets collection at some point, close your open file writers and have a clean exit. To achieve it you can use a synchronized block.
final Object lock = new Object();
final LimitedFileWriter output = new LimitedFileWriter("Tweets","tweets");
TwitterStream twitterStream = new TwitterStreamFactory(cb.build()).getInstance();
StatusListener listener = new StatusListener() {
#Override
public void onStatus(Status status) {
try{
output.write("#" + status.getUser().getScreenName() + " -- " + status.getText()+"\n");
// free the lock
if (some_condition_like_I_have_enough_files) {
synchronized (lock) {
lock.notify();
}
}
catch(IOException e){
e.printStackTrace();
}
}
}
twitterStream.addListener(listener);
twitterStream.sample("en");
try {
synchronized (lock) {
lock.wait();
}
} catch (InterruptedException e) {
e.printStackTrace();
}
// close the twitterstream
// close the writer
i've a java servlet that makes some reports. When a user choose a report it makes a query on a db and stream the xls report to the client. All in synchronous way. The problem is that sometimes i've a lot of records to fetch from the db and i would like to give a better user experience, allowing the user to do something else while the report is processing and popping out in some way the link when the process is finished. Is there a java library or some techniques to avoid the long waiting and achieve that goal?
Right now i've prepared a piece of code that in a asynchronous way completes the report and sends an email to the registered client, with the url from wich download the file, but it has to be replaced with something else because i can no longer communicate by email.
Thanks in advance
heres my take on this, i dont know of a single library that will exactly match you needs, youd probably need some custom development here.
I believe you have implemented async service that on completion sends
out an email for notification. Instead of sending out an email, let
that thread update a job table of some sort -- an entry in a db table
or some application/session scoped map.
Have a servlet/restful ws
expose that job table at some url. Poll the url at regular
intervals. Ajax poll is a standard feature in js libraries JQuery,
Prototype.
When you get a response that some report is complete, show
some popup or may be a facebook you-have-notification kind of thing
on the client side.
i have not considered authentication/authorization issues here, you need to take care of that as well.
Hope this helps
A multithreaded client server program to download my image files.
Since there are four files to download the client makes 4 connection attempts. This is not limited to 4 but the files sent by the FileServer will get repeated after the fourth attempt. The save dialog and file saving is done in different threads so as to not hamper the file downloading.
Here is the FileServer...
public class FileServer {
private final ExecutorService exec = Executors.newCachedThreadPool();
final String[] fileNames = {
"C:\\Users\\clobo\\Pictures\\Arpeggios\\Ex 1.jpg",
"C:\\Users\\clobo\\Pictures\\Arpeggios\\Ex 2.jpg",
"C:\\Users\\clobo\\Pictures\\Arpeggios\\Ex 3.jpg",
"C:\\Users\\clobo\\Pictures\\Arpeggios\\Ex 4.jpg"
};
public void start() throws IOException {
ServerSocket socket = new ServerSocket(7777);
System.out.println("Waiting for client message...");
while (!exec.isShutdown()) {
try {
for (final String fileName : fileNames){
final Socket conn = socket.accept();
exec.execute(new Runnable() {
public void run() {
sendFile(conn,fileName);
}
});
}
} catch (RejectedExecutionException e) {
if (!exec.isShutdown())
log("task submission rejected", e);
}
}
}
public void stop() {
System.out.println("Shutting down server...");
exec.shutdown();
}
private void log(String msg, Exception e) {
Logger.getAnonymousLogger().log(Level.WARNING, msg, e);
}
public void sendFile(Socket conn, String fileName) {
File myFile = new File(fileName);
if (!myFile.exists()) {
log("File does not exist!",null);
}
// file does exist
System.out.println(Thread.currentThread().getName());
System.out.println("AbsolutePath:" + myFile.getAbsolutePath());
System.out.println("length: " + myFile.length());
if (myFile.exists()) {
try {
ObjectOutputStream oos = new ObjectOutputStream(
conn.getOutputStream());
oos.writeObject(myFile);
oos.close();
} catch (IOException e) {
log("IOException Error", e);
e.printStackTrace();
}
}
}
public static void main(String[] args) throws IOException {
FileServer fs = new FileServer();
fs.start();
}
}
here is the FileServerClient...
public class FileServerClient {
private final ExecutorService exec = Executors.newCachedThreadPool();
Frame myFrame = new Frame();
List<File> fileList = new ArrayList<File>();
public void receiveFileFromServer() throws Exception{
Socket sock = null;
InputStream socketInputStream = null;
String host = "localhost";
int port = 7777;
for (int i=0;i<4;i++) {
sock = new Socket(host, port);
socketInputStream = sock.getInputStream();
System.out.println("Connection successful...");
// recieve the file
ObjectInputStream ois = new ObjectInputStream(socketInputStream);
// file from server is deserialized
final File myfile = (File) ois.readObject();
fileList.add(myfile);
// deserialized file properties
System.out.println("AbsolutePath: " + myfile.getAbsolutePath());
System.out.println("FileName:" + myfile.getName());
System.out.println("length" + myfile.length());
exec.execute(new Runnable() {
public void run() {
saveFile(myfile);
}
});
}
}
private void saveFile(File myfile) {
FileDialog fileDialog = new FileDialog(myFrame,
"Choose Destination for "+ myfile.getName(), FileDialog.SAVE);
fileDialog.setDirectory(null);
fileDialog.setFile("enter file name here");
fileDialog.setVisible(true);
String targetFileName = fileDialog.getDirectory()
+ fileDialog.getFile() + ".jpg";
System.out.println("File will be saved to: " + targetFileName);
copyBytes(myfile, targetFileName);
}
private void copyBytes(File originalFile, String targetFileName) {
try {
FileInputStream in = new FileInputStream(originalFile);
FileOutputStream out = new FileOutputStream(targetFileName);
int c;
while ((c = in.read()) != -1) {
out.write(c);
}
out.close();
in.close();
} catch (Exception e) {
log("IOException Error", e);
}
}
private void log(String msg, Exception e) {
Logger.getAnonymousLogger().log(Level.WARNING, msg, e);
}
public static void main(String[] args) throws Exception {
FileServerClient client = new FileServerClient();
client.receiveFileFromServer();
}
}
You could make an asynchronous request from the client. Lets assume that you client is an html page. When the user selects a report and clicks on 'submit' you could fire an ajax request with the report parameters (jquery can be useful for this). It would be good to keep a section on the user homepage that says something like 'prepared reports'. The client can then goto the prepared report section to download the report. As specified in the comments above, you may also have to implement a popup that informs the user that the requested report is ready. the popup is shown when the ajax requests returns successfully. However, the client may have logged out by the time the report finishes, so it may be a good idea to make the download link available again in the 'prepared reports' section when the user logs in.