How is a JSONObject from Twitter4J converted to an IOFile for use with the JsonParser from BaseX?
So, looking to use Twitter4J to get either a file or stream.
The JsonParser looks to work with a File fine:
JsonParser
public JsonParser(IO source,
MainOptions opts,
JsonParserOptions jopts)
throws java.io.IOException
Constructor.
Parameters:
source - document source
opts - database options
jopts - parser options
Throws:
java.io.IOException - I/O exception
although other IO works:
org.basex.io
Class IO
java.lang.Object
org.basex.io.IO
Direct Known Subclasses:
IOContent, IOFile, IOStream, IOUrl
How is a File acquired from a JSONObject here?
Snippet using Twitter4J:
private JSONObject jsonOps(Status status) throws JSONException, BaseXException {
String string = TwitterObjectFactory.getRawJSON(status);
JSONObject json = new JSONObject(string);
String language = json.getString("lang");
log.fine(language);
return json;
}
Writing a JSONObject to file file from Twitter4J seems to work, at least the file looks correct when opened manually:
public void writeJsonToFile(String fileName) throws FileNotFoundException, UnsupportedEncodingException, IOException {
FileOutputStream fileOutputStream = null;
OutputStreamWriter outputStreamWriter = null;
BufferedWriter bufferedWriter = null;
fileOutputStream = new FileOutputStream(fileName);
outputStreamWriter = new OutputStreamWriter(fileOutputStream, "UTF-8");
bufferedWriter = new BufferedWriter(outputStreamWriter);
bufferedWriter.write(tweets.toString());
bufferedWriter.close();
outputStreamWriter.close();
fileOutputStream.close();
}
And once the file is written, it seems quite easy to "parse" the JSON, or at least instantiate a parser, as:
private void parseJsonFile(String fileName) throws IOException {
JsonParser jsonParser = new JsonParser(new IOFile(fileName), new MainOptions());
}
full code on github.
In no way is this a complete solution. In fact, it seems a kludge to write the JSON to a file at all -- but there it is.
Seems the only way to move JSON data from Twitter4J to BaseX, or at least most pragmatic. Other solutions appreciated.
Related
When I call my batch program I get a null pointer exception on this line:
String fileName = ((Map<String, MccFtpFile>) ec.get(AbstractSetupTasklet.BATCH_FTP_FILES)).get("scacFile").getLocalFile();
Here is the entire class
// Writer
#Bean(name = "clf010Writer")
#StepScope
public FlatFileItemWriter<Clf010Item> clfWriter(#Value("#{jobExecutionContext}") Map<Object, Object> ec, //
#Qualifier("clf010LineAggregator") FormatterLineAggregator<Clf010Item> lineAgg) throws IOException {
#SuppressWarnings("unchecked")
String fileName = ((Map<String, MccFtpFile>) ec.get(AbstractSetupTasklet.BATCH_FTP_FILES)).get("scacFile").getLocalFile();
//Ensure the file can exist.
PrintWriter fos = getIoHarness().getFileOutputStream(fileName);
fos.close();
FlatFileItemWriter<Clf010Item> writer = new FlatFileItemWriter<>();
writer.setResource(new FileSystemResource(fileName));
writer.setLineAggregator(lineAgg);
return writer;
}
I can't seem to figure out the problem. My guess is it's looking for a file it can't find? If anyone has some insight or information I would appreciate it a lot.
I didn't realize, but in my setup tasklet I needed to declare a new ftp file :) thanks for the help
I know that there already are many questions on this topic, but none of them seem to help. In my Maven Project I am trying to access the src/main/resource files, and i know that during runtime when I want to read a file and write to it i must access its class path or something.
I am really not sure how that works, I managed to write code to read from a file, but I don't know how to write code that writes to a file. Here it is what i have so far:
private File userFile;
private ObjectMapper om;
private InputStream is;
public UserService() {
om = new ObjectMapper();
ClassLoader classLoader = Thread.currentThread().getContextClassLoader();
is = classLoader.getResourceAsStream("files/userJSON.json");
}
This constructor represents the UserService class which methods I use to access User Resoruces located in userJSON.json file. Here is a method I use for reading a file:
public List<User> readJSON() throws JsonParseException, JsonMappingException, IOException{
BufferedReader br = new BufferedReader(new InputStreamReader(is));
List<User> list = om.readValue(br,new TypeReference<List<User>>(){});
return list;
}
How can I write to a file, and if possible to explain to me how this works. By that I mean how does Maven work with files during runtime and so on.
Just call this method it will read the file and convert it to JSONObject:
private static JSONObject readUserJSON() throws FileNotFoundException, IOException, ParseException {
String jarPath = (new File("")).getAbsolutePath();
JSONObject config = null;
JSONParser parser = new JSONParser();
config = (JSONObject) parser.parse(new FileReader(jarPath + "resource/files/userJSON.json"));
return config;
}
ADD this dependency in .pom file:
<dependency>
<groupId>com.googlecode.json-simple</groupId>
<artifactId>json-simple example</artifactId>
<version>1.1.1</version>
</dependency>
Using below statement you get Absolute path of file and you can write it using file writer :
String jarPath = (new File("")).getAbsolutePath();
In my Android app I'm downloading Video file using Quickblox API, on successful download I'm getting file content in the form of InputStream object now, using that InputStream object I wants to create android supported Video file and stored it on to the SDCard but I don't know how to create Video file using InputStream object. Please see the following code where I'm getting InputStream object.
QBContent.downloadFileTask(fileId, new QBEntityCallbackImpl<InputStream>()
{
#Override
public void onSuccess(InputStream inputStreamObject, Bundle params)
{
// TODO Auto-generated method stub
super.onSuccess(inputStreamObject, params);
});
}
Please help. Thank you..!
If the inputStreamObject is the content of an mp4 file, you could simply save the input stream to a file. That's your mp4 .
public static final String PREFIX = "myMusicfile";
public static final String SUFFIX = ".mp4";
public static File stream2file (InputStream in) throws IOException {
final File tempFile = File.createTempFile(PREFIX, SUFFIX);
tempFile.deleteOnExit();
try (FileOutputStream out = new FileOutputStream(tempFile)) {
IOUtils.copy(in, out);
}
return tempFile;
}
Hope it help :)
Thanks
I have a utility class I created for my Spring controller to invoke to generate a CSV from a collection of beans using the SuperCSV library ( http://supercsv.sourceforge.net/ )
The utility class is pretty basic:
public static void export2CSV(HttpServletResponse response,
String[] header, String filePrefix, List<? extends Object> dataObjs) {
try{
response.setContentType("text/csv;charset=utf-8");
response.setHeader("Content-Disposition","attachment; filename="+filePrefix+"_Data.csv");
OutputStream fout= response.getOutputStream();
OutputStream bos = new BufferedOutputStream(fout);
OutputStreamWriter outputwriter = new OutputStreamWriter(bos);
ICsvBeanWriter writer = new CsvBeanWriter(outputwriter, CsvPreference.EXCEL_PREFERENCE);
// the actual writing
writer.writeHeader(header);
for(Object anObj : dataObjs){
writer.write(anObj, header);
}
}catch (Exception e){
e.printStackTrace();
}
};
The catch is, I'm getting different behaviors out of this operation and I don't know why. When I invoke it from one controller (we'll call it 'A'), I get the expected output of data.
When I invoke it from the other controller ('B'), I get a tiny blurb of unrecognizable binary data that cannot be opened by OO Calc. Opening it in Notepad++ yields an unreadable line of gibberish that I can only assume is an attempt by the reader to show me a binary stream.
Controller 'A' invocation (the one that works)
#RequestMapping(value="/getFullReqData.html", method = RequestMethod.GET)
public void getFullData(HttpSession session, HttpServletRequest request, HttpServletResponse response) throws IOException{
logger.info("INFO: ******************************Received request for full Req data dump");
String projName= (String)session.getAttribute("currentProject");
int projectID = ProjectService.getProjectID(projName);
List<Requirement> allRecords = reqService.getFullDataSet(projectID);
final String[] header = new String[] {
"ColumnA",
"ColumnB",
"ColumnC",
"ColumnD",
"ColumnE"
};
CSVExporter.export2CSV(response, header, projName+"_reqs_", allRecords);
};
...and here's the Controller 'B' invocation (the one that fails):
#RequestMapping(value="/getFullTCData.html", method = RequestMethod.GET)
public void getFullData(HttpSession session, HttpServletRequest request, HttpServletResponse response) throws IOException{
logger.info("INFO: Received request for full TCD data dump");
String projName= (String)session.getAttribute("currentProject");
int projectID = ProjectService.getProjectID(projName);
List<TestCase> allRecords = testService.getFullTestCaseList(projectID);
final String[] header = new String[] {
"ColumnW",
"ColumnX",
"ColumnY",
"ColumnZ"
};
CSVExporter.export2CSV(response, header, projName+"_tcs_", allRecords);
}
Observations:
Which controller I invoke first is irrelevant. 'A' always works and 'B' always produces gibberish
Both calls to this function have a list of header columns that are a subset of the total set of operations defined in the bean being passed in to CSVWriter
The simple Exception printStackTrace is working to detect when a bean's reflection field doesn't match the definition (i.e., can't find get() to get the value programmatically) suggesting that all column/variable matchups are succeeding.
In the debugger, I've verified the writer.write(Object, header) call is being hit the expected number of times based on the number of objects being passed and that these objects have the expected data
Any suggestions or insights would be greatly appreciated. I'm really stumped how to better isolate the issue...
You aren't closing the writer. Also, CsvBeanWriter will wrap the writer in a BufferedWriter, so you can probably simplify your outputwriter as well.
public static void export2CSV(HttpServletResponse response,
String[] header, String filePrefix, List<? extends Object> dataObjs) {
ICsvBeanWriter writer;
try{
response.setContentType("text/csv;charset=utf-8");
response.setHeader("Content-Disposition",
"attachment; filename="+filePrefix+"_Data.csv");
OutputStreamWriter outputwriter =
new OutputStreamWriter(response.getOutputStream());
writer = new CsvBeanWriter(outputwriter, CsvPreference.EXCEL_PREFERENCE);
// the actual writing
writer.writeHeader(header);
for(Object anObj : dataObjs){
writer.write(anObj, header);
}
} catch (Exception e) {
e.printStackTrace();
} finally {
try {
writer.close(); // closes writer and underlying stream
} catch (Exception e){}
}
};
Super CSV 2.0.0-beta-1 is out now! As well as adding numerous other features (including Maven support and a new Dozer extension), CSV writers now expose a flush() method as well.
The FileUtils.writeStringToFile(fileName, text) function of Apache Commons I/O overwrites previous text in a file. I would like to append data to my file. Is there any way I could use Commons I/O for the same? I can do it using normal BufferedWriter from Java but I'm curious regarding the same using Commons I/O.
It has been implemented in 2.1 version of Apache IO.
To append string to the file just pass true as an additional parameter in functions:
FileUtils.writeStringToFile
FileUtils.openOutputStream
FileUtils.write
FileUtils.writeByteArrayToFile
FileUtils.writeLines
ex:
FileUtils.writeStringToFile(file, "String to append", true);
Download the latest version Commons-io 2.1
FileUtils.writeStringToFile(File,Data,append)
set append to true....
Careful. That implementation seems to be leaking a file handle...
public final class AppendUtils {
public static void appendToFile(final InputStream in, final File f) throws IOException {
OutputStream stream = null;
try {
stream = outStream(f);
IOUtils.copy(in, stream);
} finally {
IOUtils.closeQuietly(stream);
}
}
public static void appendToFile(final String in, final File f) throws IOException {
InputStream stream = null;
try {
stream = IOUtils.toInputStream(in);
appendToFile(stream, f);
} finally {
IOUtils.closeQuietly(stream);
}
}
private static OutputStream outStream(final File f) throws IOException {
return new BufferedOutputStream(new FileOutputStream(f, true));
}
private AppendUtils() {}
}
this little thingy should do the trick:
package com.yourpackage;
// you're gonna want to optimize these imports
import java.io.*;
import org.apache.commons.io.*;
public final class AppendUtils {
public static void appendToFile(final InputStream in, final File f)
throws IOException {
IOUtils.copy(in, outStream(f));
}
public static void appendToFile(final String in, final File f)
throws IOException {
appendToFile(IOUtils.toInputStream(in), f);
}
private static OutputStream outStream(final File f) throws IOException {
return new BufferedOutputStream(new FileOutputStream(f, true));
}
private AppendUtils() {
}
}
edit: my eclipse was broken, so it didn't show me the errors earlier. fixed errors
Actually, version 2.4 of apache-commons-io FileUtils now has append mode for collections as well.
Here's the Javadoc
And the maven dependency:
<dependency>
<groupId>commons-io</groupId>
<artifactId>commons-io</artifactId>
<version>2.4</version>
<type>jar</type>
</dependency>
in version 2.5 you need to pass one extra parameter i.e, encoding.
FileUtils.writeStringToFile(file, "line to append", "UTF-8", true);
public static void writeStringToFile(File file,
String data,
boolean append)
throws IOException
Writes the toString() value of each item in a collection to the specified File line by line. The default VM encoding and the default line ending will be used.
Parameters:
file - the file to write to
lines - the lines to write, null entries produce blank lines
append - if true, then the lines will be added to the end of the file rather than overwriting
Throws:
IOException - in case of an I/O error
Since:
Commons IO 2.1