I am making an Email Client(for class) and i want to store the messages i send everyday in a single file.I am going to use serialization for this.I tried appending to a file by setting append to true in FileOutputStream.Below is the code i wrote.But it throws a StreamCorruptedException(stack trace given below).I cannot seem to find the reason why.I would like to know whether there is a better way to store multiple objects to a single file using serialization.Any help is appreciated.Thanks in advance.
//For reading and writing objects to files
interface MyFileHandler<T> {
public void write(T input );
public ArrayList<T> read();
}
class FileHandlerObject implements MyFileHandler<EmailMessage>
{
public void write(EmailMessage input){
try{
FileOutputStream fileOutputStream = new FileOutputStream("emails.ser",true);
ObjectOutputStream objectOutputStream = new ObjectOutputStream(fileOutputStream);
objectOutputStream.writeObject(input);
objectOutputStream.flush();
objectOutputStream.close();
}
catch(IOException e)
{
e.printStackTrace();
}
}
public ArrayList<EmailMessage> read(){
ArrayList<EmailMessage> messages=new ArrayList<EmailMessage>();
try{
FileInputStream fileInputStream=new FileInputStream("emails.ser");
ObjectInputStream objectInputStream=new ObjectInputStream(fileInputStream);
boolean cont = true;
while (cont) {
if(fileInputStream.available()!=0)
{
Object obj=objectInputStream.readObject();
EmailMessage message=(EmailMessage)obj;
messages.add(message);
}
else{
cont=false;
}
}
objectInputStream.close();
return messages;
}
catch(IOException e)
{
e.printStackTrace();
}
catch(ClassNotFoundException c)
{
c.printStackTrace();
}
catch(ClassCastException c)
{
c.printStackTrace();
}
return null;
}
public static void main(String[] args) {
FileHandlerObject f=new FileHandlerObject();
ArrayList<EmailMessage> a=f.read();
a=f.read();
}
}
And the Email Message class that i serialize
public class EmailMessage implements Serializable{//implement serializables
private String recipient;
private String subject;
private String content;
private String date;
public void setRecipient(String recipient)
{
this.recipient=recipient;
}
public String getRecipient()
{
return this.recipient;
}
public void setSubject(String subject){
this.subject=subject;
}
public void setContent(String content){
this.content=content;
}
public String getSubject(){
return this.subject;
}
public String getContent(){
return this.content;
}
public void setDate(String date)
{
this.date=date;
}
public String getDate()
{
return this.date;
}
public String printDetails()
{
String details="Recipient: "+getRecipient()+
"\nSubject: "+getSubject()+
"\nEmail content: "+getContent()+
"\nDate Sent: "+getDate();
return details;
}}
And below is the stack trace
java.io.StreamCorruptedException: invalid type code: AC
at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1723)
at java.io.ObjectInputStream.readObject(ObjectInputStream.java:508)
at java.io.ObjectInputStream.readObject(ObjectInputStream.java:466)
at FileHandlerObject.read(MyFileHandler.java:46)
at FileHandlerObject.main(MyFileHandler.java:89)
I have to write a simple "Word Count" Topology in Java and Storm. In particular, i have an external data source generating CSV (comma separated) string like
Daniel, 0.5654, 144543, user, 899898, Comment,,,
These strings are inserted into a RabbitMQ queue called "input". This datasource works well, and i can see the strings in the queue.
Now, i modified the classic topology adding the RabbitMQSpout. The goal is to do a word count for the first field of every CSV line, and publish results into a new queue called "output". The problem is that i cannot see any tuple inside the new queue, but the topology was submitted and RUNNING.
So, summing up:
external data source puts items into the input queue
RabbitMQSpout takes items from input queue and insert them into topology
classic word-count topology isperformed
last bolt puts results into output queue
Problem:
i can see items inside input queue, but nothing into output, even if i used same method to send item into the queue in the external data source (and it works) and in RabbitMQExporter (does not work...)
Some code below
RabbitMQSpout
public class RabbitMQSpout extends BaseRichSpout {
public static final String DATA = "data";
private SpoutOutputCollector _collector;
private RabbitMQManager rabbitMQManager;
#Override
public void open(Map map, TopologyContext topologyContext, SpoutOutputCollector spoutOutputCollector) {
_collector = _collector;
rabbitMQManager = new RabbitMQManager("localhost", "rabbitmq", "rabbitmq", "test");
}
#Override
public void nextTuple() {
Utils.sleep(1000);
String data = rabbitMQManager.receive("input");
if (data != null) {
_collector.emit(new Values(data));
}
}
#Override
public void declareOutputFields(OutputFieldsDeclarer outputFieldsDeclarer) {
outputFieldsDeclarer.declare(new Fields(DATA));
}
}
SplitBolt
public class SplitBolt extends BaseRichBolt {
private OutputCollector _collector;
public SplitSentenceBolt() { }
#Override
public void prepare(Map stormConf, TopologyContext context, OutputCollector collector) {
this._collector = collector;
this.SPACE = Pattern.compile(",");
}
#Override
public void execute(Tuple input) {
String sentence = input.getStringByField(RabbitMQSpout.DATA);
String[] words = sentence.split(",");
if (words.length > 0)
_collector.emit(new Values(words[0]));
}
#Override
public void declareOutputFields(OutputFieldsDeclarer declarer) {
declarer.declare(new Fields("word"));
}
#Override
public Map<String, Object> getComponentConfiguration() {
return null;
}
}
WordCountBolt
public class WordCountBolt extends BaseBasicBolt {
Map<String, Integer> counts = new HashMap<String, Integer>();
#Override
public void execute(Tuple tuple, BasicOutputCollector collector) {
String word = tuple.getString(0);
Integer count = counts.get(word);
if (count == null)
count = 0;
count++;
counts.put(word, count);
System.out.println(count);
collector.emit(new Values(word, count));
}
#Override
public void declareOutputFields(OutputFieldsDeclarer declarer) {
declarer.declare(new Fields("word", "count"));
}
}
RabbitMQExporterBolt
public RabbitMQExporterBolt(String rabbitMqHost, String rabbitMqUsername, String rabbitMqPassword,
String defaultQueue) {
super();
this.rabbitMqHost = rabbitMqHost;
this.rabbitMqUsername = rabbitMqUsername;
this.rabbitMqPassword = rabbitMqPassword;
this.defaultQueue = defaultQueue;
}
#Override
public void prepare(#SuppressWarnings("rawtypes") Map map, TopologyContext topologyContext, OutputCollector outputCollector) {
this.collector=outputCollector;
this.rabbitmq = new RabbitMQManager(rabbitMqHost, rabbitMqUsername, rabbitMqPassword, defaultQueue);
}
#Override
public void execute(Tuple tuple) {
String word = tuple.getString(0);
Integer count = tuple.getInteger(1);
String output = word + " " + count;
rabbitmq.send(output);
}
#Override
public void declareOutputFields(OutputFieldsDeclarer outputFieldsDeclarer) {
outputFieldsDeclarer.declare(new Fields("word"));
}
}
Topology
public class WordCountTopology {
private static final String RABBITMQ_HOST = "rabbitmq";
private static final String RABBITMQ_USER = "rabbitmq";
private static final String RABBITMQ_PASS = "rabbitmq";
private static final String RABBITMQ_QUEUE = "output";
public static void main(String[] args) throws Exception {
TopologyBuilder builder = new TopologyBuilder();
builder.setSpout("spout", new RabbitMQSpout(), 1);
builder.setBolt("split", new SplitSentenceBolt(), 1)
.shuffleGrouping("spout");
builder.setBolt("count", new WordCountBolt(), 1)
.fieldsGrouping("split", new Fields("word"));
Config conf = new Config();
conf.setDebug(true);
if (args != null && args.length > 0) {
builder.setBolt("exporter",
new RabbitMQExporterBolt(
RABBITMQ_HOST, RABBITMQ_USER,
RABBITMQ_PASS, RABBITMQ_QUEUE ),
1)
.shuffleGrouping("count");
conf.setNumWorkers(3);
StormSubmitter.submitTopologyWithProgressBar(args[0], conf, builder.createTopology());
} else {
conf.setMaxTaskParallelism(3);
LocalCluster cluster = new LocalCluster();
cluster.submitTopology("word-count", conf, builder.createTopology());
Thread.sleep(10000);
cluster.shutdown();
}
}
}
RabbitMQManager
public class RabbitMQManager {
private String host;
private String username;
private String password;
private ConnectionFactory factory;
private Connection connection;
private String defaultQueue;
public RabbitMQManager(String host, String username, String password, String queue) {
super();
this.host = host;
this.username = username;
this.password = password;
this.factory = null;
this.connection = null;
this.defaultQueue = queue;
this.initialize();
this.initializeQueue(defaultQueue);
}
private void initializeQueue(String queue){
ConnectionFactory factory = new ConnectionFactory();
factory.setHost(host);
factory.setUsername(username);
factory.setPassword(password);
Connection connection;
try {
connection = factory.newConnection();
Channel channel = connection.createChannel();
boolean durable = false;
boolean exclusive = false;
boolean autoDelete = false;
channel.queueDeclare(queue, durable, exclusive, autoDelete, null);
channel.close();
connection.close();
} catch (IOException | TimeoutException e) {
e.printStackTrace();
}
}
private void initialize(){
factory = new ConnectionFactory();
factory.setHost(host);
factory.setUsername(username);
factory.setPassword(password);
try {
connection = factory.newConnection();
} catch (IOException | TimeoutException e) {
e.printStackTrace();
}
}
public void terminate(){
if (connection != null && connection.isOpen()){
try {
connection.close();
} catch (IOException e) {
e.printStackTrace();
}
}
}
private boolean reopenConnectionIfNeeded(){
try {
if (connection == null){
connection = factory.newConnection();
return true;
}
if (!connection.isOpen()){
connection = factory.newConnection();
}
} catch (IOException | TimeoutException e) {
e.printStackTrace();
return false;
}
return true;
}
public boolean send(String message){
return this.send(defaultQueue, message);
}
public boolean send(String queue, String message){
try {
reopenConnectionIfNeeded();
Channel channel = connection.createChannel();
channel.basicPublish("", queue, null, message.getBytes());
channel.close();
return true;
} catch (IOException | TimeoutException e) {
e.printStackTrace();
}
return false;
}
public String receive(String queue) {
try {
reopenConnectionIfNeeded();
Channel channel = connection.createChannel();
Consumer consumer = new DefaultConsumer(channel);
return channel.basicConsume(queue, true, consumer);
} catch (Exception e) {
e.printStackTrace();
return null;
}
}
}
-> controller.java
public controller() {
public controller(DataInterpreter interpret,ControllerClientUtility util, InterfaceConnection inter) {
interpreter = interpret;
utility = util;
interfaced = inter;
}
}
...
public void closeOne(String vpnSessionId) throws Exception {
try{
if ( interfaced.connect() && (interfaced.CheckIntegrity(SessionId)) ){
interfaced.kill(vpnSessionId);
}else{
closeAll();
}
}catch(Exception e){
if ( e.getMessage().startsWith("INTERFACE_ERR:") ){
closeAll();
}else{
throw new Exception(e);
}
}
}
-> methods in InterfaceConnection.java
public String getReponseFor(String command) throws Exception{
if (send(command)){
return receive();
}
else{
throw new Exception("INTERFACE_ERR: Could not get Response");
}
}
public List<String> getListOfConnections() throws Exception{
String statusResponse = getReponseFor("something");
..(regex searches and then make a list connectionsConnected)
return connectionsConnected;
}
public boolean CheckIntegrity(String SessionId){
try {
List<String> connections = new ArrayList<String>();
connections = getListOfConnections();
if (connections.contains(SessionId)){
return true;
}
return false;
}catch(Exception e){
return false;
}
}
Is there a way to mock the output of getListOfConnections ? I tried doing something like this but did not work
-> controllerTest.java
#Mock private InterfaceConnection interfaced;
#Before
public void beforeTests() throws Exception {
MockitoAnnotations.initMocks(this);
impl = new Controller(interpreter,utility,interfaced);
...
#Test
public void testDisconnectOneSessionWithBadSessionId_sendCommand() throws Exception{
String badSessionId = "123:123";
List<String> mockConnections = new ArrayList<String>();
mockConnections.add("asdasds");
when(interfaced.getListOfConnections()).thenReturn(mockConnections);
impl.closeOne(badSessionId);
Mockito.verify(utility)....
}
I hope I'm clear, thanks in advance.
I am using a global variables "GlobalVariables" in a separated class and I am try to use it in the following code but it is always gives me the error :
The method getApplication() is undefined for the type UploadPicture
I tried the following but still have error:
((GlobalVariables) this.getApplication()).set_FileUploading(false);
The qustion was already asked here but unfortunatlly all the answors didn't work with me and gave me same error! any suggestion please?
public class UploadPicture extends AsyncTask<Void, Long, Boolean> {
private DropboxAPI<?> mApi;
private String mPath;
private File mFile;
private long mFileLen;
private UploadRequest mRequest;
private Context mContext;
private String mErrorMsg;
private File outFiles;
public UploadPicture(Context context, DropboxAPI<?> api, String dropboxPath, File file) {
mContext = context.getApplicationContext();
mFileLen = file.length();
mApi = api;
mPath = dropboxPath;
mFile = file;
}
#Override
protected Boolean doInBackground(Void... params) {
try {
FileInputStream fis = new FileInputStream(mFile);
String path = mPath + outFiles.getName();
mRequest = mApi.putFileOverwriteRequest(path, fis, mFile.length(),
new ProgressListener() {
#Override
public long progressInterval() {
return 500;
}
#Override
public void onProgress(long bytes, long total) {
//publishProgress(bytes);
}
}
);
if (mRequest != null) {
mRequest.upload();
((GlobalVariables) UploadPicture.this.getApplication()).set_FileUploading(false);
return true;
}
} catch (DropboxUnlinkedException e) {
// This session wasn't authenticated properly or user unlinked
mErrorMsg = "This app wasn't authenticated properly.";
} catch (DropboxFileSizeException e) {
// File size too big to upload via the API
mErrorMsg = "This file is too big to upload";
} catch (DropboxPartialFileException e) {
// We canceled the operation
mErrorMsg = "Upload canceled";
} catch (DropboxServerException e) {
// Server-side exception. These are examples of what could happen,
// but we don't do anything special with them here.
if (e.error == DropboxServerException._401_UNAUTHORIZED) {
// Unauthorized, so we should unlink them. You may want to
// automatically log the user out in this case.
} else if (e.error == DropboxServerException._403_FORBIDDEN) {
// Not allowed to access this
} else if (e.error == DropboxServerException._404_NOT_FOUND) {
// path not found (or if it was the thumbnail, can't be
// thumbnailed)
} else if (e.error == DropboxServerException._507_INSUFFICIENT_STORAGE) {
// user is over quota
} else {
// Something else
}
// This gets the Dropbox error, translated into the user's language
mErrorMsg = e.body.userError;
if (mErrorMsg == null) {
mErrorMsg = e.body.error;
}
} catch (DropboxIOException e) {
// Happens all the time, probably want to retry automatically.
mErrorMsg = "Network error. Try again.";
} catch (DropboxParseException e) {
// Probably due to Dropbox server restarting, should retry
mErrorMsg = "Dropbox error. Try again.";
} catch (DropboxException e) {
// Unknown error
mErrorMsg = "Unknown error. Try again.";
} catch (FileNotFoundException e) {
e.printStackTrace();
}
return false;
}
}
Edit: I am adding now my "VariableGlobales" calss:
public class GlobalVariables extends Application {
private Boolean _IsIOIORunning=false;
private Boolean _FileUploading=false;
public Boolean get_IsIOIORunning()
{
return _IsIOIORunning;
}
public void set_IsIOIORunning(Boolean _IsIOIORunning)
{
this._IsIOIORunning = _IsIOIORunning;
}
public Boolean get_FileUploading()
{
return _FileUploading;
}
public void set_FileUploading(Boolean _FileUploading)
{
this._FileUploading = _FileUploading;
}
It's normal UploadPicture doesn't extend GlobalVariables but it extend AsyncTask.
That it's my "GlobalVariables "
public class AppInfo extends Application {
private static Context context;
private static String user;
public void onCreate(){
super.onCreate();
AppInfo.context = getApplicationContext();
user = null;
}
public static Context getAppContext() {return AppInfo.context;}
public static String getUser() {return user;}
public static void setUser(String user) {AppInfo.user = user;}
}
And I call it everywhere like that:
AppInfo.getUser();
Edit:
GlobalVariables should use static method and variables:
public class GlobalVariables extends Application {
private static Boolean _IsIOIORunning=false;
private static Boolean _FileUploading=false;
public static Boolean get_IsIOIORunning() {
return _IsIOIORunning;
}
public static void set_IsIOIORunning(Boolean _IsIOIORunning) {
GlobalVariables._IsIOIORunning = _IsIOIORunning;
}
public static Boolean get_FileUploading(){
return _FileUploading;
}
public static void set_FileUploading(Boolean _FileUploading){
GlobalVariables._FileUploading = _FileUploading;
}
}
Here is my demo:
PoolableObjectFactoryImpl.java
public class PoolableObjectFactoryImpl implements PoolableObjectFactory<Result> {
private static Logger logger = Logger.getLogger("BackgroundLog");
#Override
public void activateObject(Result obj) throws Exception {
logger.info("==activate result.==");
obj.setResult(-999);
}
#Override
public void destroyObject(Result obj) throws Exception {
logger.info("==destroy result.==");
obj = null;
}
#Override
public Result makeObject() throws Exception {
logger.info("==make result.==");
Result result = new Result();
return result;
}
#Override
public void passivateObject(Result obj) throws Exception {
logger.info("==passivate result.==");
obj.setResult(-999);
}
#Override
public boolean validateObject(Result obj) {
/*if(obj.getResult() == -999){
logger.info("==validate result true.==");
return true;
}else{
logger.info("==validate result false.==");
return false;
}*/
logger.info("==validate result true.==");
return true;
}
}
ThreadPool.java
public class ThreadPool extends GenericObjectPool {
private static Logger logger = Logger.getLogger("BackgroundLog");
private static ThreadPool pool = null;
private Map<String, String> map = getConfig();
private ThreadPool() {
this.setFactory(new PoolableObjectFactoryImpl());
this.setMaxActive(Integer.parseInt(map.get("maxActive")));
this.setWhenExhaustedAction(Byte.valueOf(map.get("whenExhaustedAction")));
this.setMaxWait(Long.parseLong(map.get("maxWait")));
this.setMaxIdle(Integer.parseInt(map.get("maxIdle")));
this.setTestOnBorrow(Boolean.valueOf(map.get("testOnBorrow")));
this.setTestOnReturn(Boolean.valueOf(map.get("testOnReturn")));
this.setTimeBetweenEvictionRunsMillis(Long.parseLong(map.get("timeBetweenEvictionRunsMillis")));
this.setNumTestsPerEvictionRun(Integer.parseInt(map.get("numTestsPerEvictionRun")));
this.setMinEvictableIdleTimeMillis(Long.parseLong(map.get("minEvictableIdleTimeMillis")));
this.setTestWhileIdle(Boolean.valueOf(map.get("testWhileIdle")));
}
public static ThreadPool getInstance() {
if (pool == null) {
synchronized (ThreadPool.class) {
if (pool == null) {
logger.info("thread pool is initialized.");
pool = new ThreadPool();
}
}
}
return pool;
}
/**
*
* <p>Title: getConfig</p>
* <p>Description: get pool configuration</p>
* #return
*/
public Map<String, String> getConfig() {
Map<String, String> map = new HashMap<String, String>();
Properties props = new Properties();
try {
InputStream in = Thread.currentThread().getContextClassLoader().getResourceAsStream("pool.properties");
props.load(in);
Enumeration en = props.propertyNames();
while (en.hasMoreElements()) {
String key = (String) en.nextElement();
map.put(key, props.getProperty(key));
}
in.close();
} catch (Throwable t) {
logger.error(t.getMessage(), t);
}
return map;
}
}
Result.java
public class Result {
private int result;
public Result(){
}
public int getResult(){
return this.result;
}
public void setResult(int result){
this.result = result;
}
}
Test.java
public class Test implements Runnable {
private static Logger logger = Logger.getLogger("BackgroundLog");
private String name = null;
public Test(String name){
this.name = name;
}
public String getName(){
return this.name;
}
public void setName(String name){
this.name = name;
}
#Override
public void run() {
ThreadPool pool = ThreadPool.getInstance();
for(int i=0;i<1000;i++){
try {
Result result = (Result)pool.borrowObject();
logger.info("numActive: "+ pool.getNumActive()+"\t"+"numIdle: "+pool.getNumIdle());
logger.info("thread "+getName()+" "+i+" borrow object from pool "+result.getResult()+".");
result.setResult(0);
pool.returnObject(result);
logger.info("return object to pool.");
Thread.sleep(100);
} catch (Exception e) {
logger.info("thread "+getName()+" "+i);
e.printStackTrace();
}
}
}
public static void main(String[] args) {
for(int i=0;i<50;i++){
Thread t = new Thread(new Test("t"+i));
t.start();
}
}
}
Next is the configuration properties:
Next is the threads view from Jprofiler when it has 4 threads:
After Test.java is running a few minutes,some threads keep beling blocked,only one is still running but does not print any log.I don't really understand thread thing.
can anyone explain why? how to avoid threads being blocked?
Consider posting logs of an execution cycle.
Did you try commenting Thread.sleep line, because sleep will hold onto the lock it has acquired till the thread is in sleep mode.
Try replacing "Thread.sleep(100);" with:
try {
synchronized (this) {
this.wait(200);
}
} catch (InterruptedException e) {
}