I have to do a HTTP post in java every second after building a json object.
The json object is built from reading a CSV file which is huge (200Mbs+), so my problem is
how do I read x number of lines build x objects and post that every second(as it is not possible to parse the whole 200mb file in less than a second) and continue reading the next x lines.
Please let me know your thoughts..
Can I use Java timer class, and keep reading the CSV file and at the same time post the json object to the server every second with the formed json?
It is hardly possible to read, parse, convert and send a 200 MB file once per second.
So you need to change your design:
My suggestion would be to only send changed lines, something like this:
{
"1" : {"field1":"value1","field2":"value2"},
"17" : {"field1":"value1","field2":"value2"}
}
Which of course gives you new problems:
The client needs to figure out which lines have changed, and the server needs to integrate the changed lines with the existing data.
I would make it depending on the file size and not depending on time.
BufferedReader fin = null; //create it
Gson gson=new Gson(); //Google code open source library for JSON in Java
ArrayList<JSONObject> jsonList=new ArrayList<JSONObject>();
while (((line = fin.readLine()) != null)) {
if ( line.length()==0 ){
//"Blank line;
}else{
currJSON=loadJSON(line);//You have to load it in a Java Object
if ( jsonList.size()<MAX_JSON){
jsonList.add(currJSON);
}
if (JsonList.size()==MAX_JSON){ //Define the maximum size of the list you want to post
gson.toJson(jsonList); //Convert to JSON
//You should post your Json with some Http Connection to your server
jsonList.clear();
Related
I'm trying to parse a huge CSV (56595 lines) into a JSONArray but it's taking a considerable amount of time. This is what my code looks like and it takes ~17 seconds to complete. I'm limiting my results based on one of the columns but the code still has to go through the entire CSV file.
Is there a more efficient way to do this? I've excluded the catch's, finally's and throws to save space.
File
Code
...
BufferedReader reader = null;
String line = "";
//jArray is retrieved by an ajax call and used in a graph
JSONArray jArray = new JSONArray();
HttpClient httpClient = new DefaultHttpClient();
try {
//url = CSV file
HttpGet httpGet = new HttpGet(url);
HttpResponse response = httpClient.execute(httpGet);
int responseCode = response.getStatusLine().getStatusCode();
if (responseCode == 200) {
try {
reader = new BufferedReader(new InputStreamReader(response.getEntity().getContent()));
while (((line = reader.readLine()) != null)) {
JSONObject json = new JSONObject();
String[] row = line.split(",");
//skips first three rows
if(row.length > 2){
//map = 4011
if(row[1].equals(map)) {
json.put("col0", row[0]);
json.put("col1", row[1]);
json.put("col2", row[2]);
json.put("col3", row[3]);
json.put("col4", row[4]);
json.put("col5", row[5]);
json.put("col6", row[6]);
jArray.put(json);
}
}
return jArray;
}
...
Unfortunately, the main delay will predictably be at downloading the file from HTTP, so all your chances will rest upon optimizing your code. So, based upon the info you provided, I can suggest some enhancements to optimize your algorithm:
It was a good idea to process the input file in streaming mode, reading line by line with a with BufferedReader. Usually it is a good practice to set an explicit buffer size (BufferedReader's default size is 8Kb), but being the source a network connection, I doubt it will be any better in this case. Anyway, you should try 16Kb, for instance.
Since the number of output items is very low (49, you said), it doesn't matter to store it in an array (for a higher amount, I would have recommend you to chose another collection, like LinkedList), but it is always useful to pre-size it with an estimated size. In JSONArray, I suppose it would be enough to put a null item at position 100 (for example) at the beginning of your method.
The biggest deal I think of is the line line.split(","), because that makes the program go through the whole line, duplicate its contents character by character into an array, and the worst of it all, for eventually use it only in a 0.1% of cases.
And there might be even a worse drawback: Merely splitting by comma might be not a good way to properly parse a JSON line. I mean: Are you sure the json values cannot contain a comma as part of user data?
Well, to solve this problem I suggest you to code your own json custom parsing algorithm, which might be a little hard, but it will be worth the effort. You must code a state machine in which you detect the second value and, if the key coincides with the filtering value ("4011"), continue parsing the rest of the line. In this way, you will save a big amount of time and memory.
I am writing json data into one file. It works fine for small data but for large data it is writing some amount of data and remaining data gets skipped. So Please guide me how to create new file if one gets fulled in java?
Currently i am using below code to write json object into a file.
Below is the sample which is quite similar to my working code.
String json=null;
for(int counter=0;counter<2;counter++){
JSONObject obj=new JSONObject();
JSONObject second=new JSONObject();
obj.put("ClassName", "sample.com");
obj.put("Query", "query");
obj.put("Message", "Successfylly");
second.put("Number"+sequence++, obj);
company.add(second);
//file.write(obj.toJSONString());
Gson gson = new GsonBuilder().setPrettyPrinting().create();
json = gson.toJson(company);
file.write(json);
file.close();
}
Please guide me how to create new file if existing file gets fulled while writing json object into a text file.
It's difficult to tell, as your code is not formatted well, but it looks like every iteration of your loop is overwriting the variable json. Only the value of json at the last iteration of the loop will be written to the file.
Move the call to file.write(json) inside the loop.
Is there a way (any jmeter plugin) by which we can have the JMeter script read all the contents(String) from external text file ?
I have a utility in java which uses Jackson ObjectMapper to convert a arraylist to string and puts it to a text file in the desktop. The file has the JSON info that i need to send in the jmeter Post Body.
I tried using ${__FileToString()} but it was unable to deserialize the instance of java.util.ArrayList. It was also not reading all the values properly.
I am looking for something like csv reader where i just give the file location. I need all the json info present in the file. Need to extract it and assign to the post body.
Thanks for your help !!!
If your question is about how to deserialize ArrayList in JMeter and dynamically build request body, you can use i.e. Beanshell PreProcessor for it.
Add a Beanshell PreProcessor as a child of your request
Put the following code into the PreProcessor's "Script" area:
FileInputStream in = new FileInputStream("/path/to/your/serialized/file.ser");
ObjectInput oin = new ObjectInputStream(in);
ArrayList list = (ArrayList) oin.readObject();
oin.close();
in.close();
for (int i = 0; i < list.size(); i++) {
sampler.addArgument("param" + i, list.get(i).toString());
}
The code will read file as ArrayList, iterate through it and add request parameter like:
param1=foo
param2=bar
etc.
This is the closest answer I'm able to provide, if you need more exact advice - please elaborate your question. In the meantime I recommend you to get familiarized with How to use BeanShell: JMeter's favorite built-in component guide to learn about scripting in JMeter and what do pre-defined variables like "sampler" in above code snippet mean.
I am trying to read RSS feeds from CNN in an Android project. So far everything is going right. I successfully made the connection and retrieved the whole XML file as one string. Then I tried to create a JSON Object and parse it. However some part of it couldn't be read. The XML I tried to read is this
view-source:http://rss.cnn.com/rss/edition_world.rss
In order to simplify and make everything more clear, I attached a picture of the RSS feed viewed in a JSON Object Editor:
http://tinypic.com/view.php?pic=2iiaezb&s=8#.U7ax0vmGFmw
So the code is like this,
//It successfully converts the text to JSON
JSONObject jObj = new JSONObject(responseText);
String respTime = jObj.getString("responseTime");
//It successfully prints the responseTime
System.out.println("Response time is: " + respTime);
JSONObject respHeader = jObj.getJSONObject("responseHeaders");
String date = respHeader.getString("date");
//It successfully prints the date as well
System.out.println("Date is: "+ date);
//However it says no value for rss found
JSONObject rssObj = jObj.getJSONObject("rss");
JSONObject channelObj = rssObj.getJSONObject("channel");
JSONArray itemArr = channelObj.getJSONArray("item");
"responseTime", "responseHeaders" and "rss" are all equivalent in terms of hierarchical structure of the XML file, as you can see from the image I referenced. So while I am able to read "responseTime" and "responseHeaders", why does it say that no value found for "rss", and therefore I am unable to reach any of it's sub-items ?
I found out the answer. The actual JSON object string, namely the responseText, is not identical to what appears in the XML file. It is something different and it took time for me to realize it. I couldn't copy it from the Eclipse's logcat since it has a limited buffer, so I had to study it by writing to a text file. Then I could parse it correctly.
I have really big JSON file for parsing and managing. My JSON file contains structure like this
[
{"id": "11040548","key1":"keyValue1","key2":"keyValue2","key3":"keyValue3","key4":"keyValue4","key5":"keyValue5","key6":"keyValue6","key7":"keyValue7","key8":"keyValue8","key9":"keyValue9","key10":"keyValue10","key11":"keyValue11","key12":"keyValue12","key13":"keyValue13","key14":"keyValue14","key15":"keyValue15"
},
{"id": "11040549","key1":"keyValue1","key2":"keyValue2","key3":"keyValue3","key4":"keyValue4","key5":"keyValue5","key6":"keyValue6","key7":"keyValue7","key8":"keyValue8","key9":"keyValue9","key10":"keyValue10","key11":"keyValue11","key12":"keyValue12","key13":"keyValue13","key14":"keyValue14","key15":"keyValue15"
},
....
{"id": "11040548","key1":"keyValue1","key2":"keyValue2","key3":"keyValue3","key4":"keyValue4","key5":"keyValue5","key6":"keyValue6","key7":"keyValue7","key8":"keyValue8","key9":"keyValue9","key10":"keyValue10","key11":"keyValue11","key12":"keyValue12","key13":"keyValue13","key14":"keyValue14","key15":"keyValue15"
}
]
My JSON file contains data about topics from news website and practically every day this JSON file will be increased dramatically.
For parsing of that file I use
URL urlLinkSource = new URL(OUTBOX_URL);
urlLinkSourceReader = new BufferedReader(new InputStreamReader(
urlLinkSource.openStream(), "UTF-8"));
ObjectMapper mapper = new ObjectMapper();
List<DataContainerList> DataContainerListData = mapper.readValue(urlLinkSourceReader,new TypeReference<List<DataContainerList>>() { }); //DataContainerList contains id, key1, key2, key3..key15
My problem is that I want to load in this line
List<DataContainerList> DataContainerListData = mapper.readValue(urlLinkSourceReader,new TypeReference<List<DataContainerList>>() { });
only range of JSON object - just first ten object, just second ten object - because I need to display in my app just 10 news in paging mode (all the time I know the index of which 10 I need to display). It totally stuped to load 10 000 objects and to iterate just first 10 of them. So my question is how I can load
in similar way like this one:
List<DataContainerList> DataContainerListData = mapper.readValue(urlLinkSourceReader,new TypeReference<List<DataContainerList>>() { });
only objects with indexes FROM -TO (for example from 30 to 40) without loading of all objects in the entire JSON file?
Regards
It depends of what you mean by "load object with indexes from to", if you want to
Read everything but bind only a sublist
The solution in that case is to read the full stream and only bind values within those indexes.
You can use jacksons streaming api and do it yourself. Parse the stream use a counter to keep track of actual index and then bind to POJOs only what you need.
However this is not a good solution if your file is large and its done in real time.
Read only the data between those indexes
You should do that if your file is big and performance matters. Instead of having a single big file, do the pagination by splitting your json array into multiple files matching your ranges, and then just deserialize the specific file content into your array.
Hope this helps...