We Keep Coding

Randomly changing the JSON Values for every "Post" Request Body using Java

This could be a duplicate question, but I couldn't find my solution anywhere. Hence, posting it.
I am trying to simply POST a request for a Student account Creation Scenario. I do have a JSON file which comprises all the "Keys:Values", required for Student account creation.
This is how the file student_Profile.json looks like:
{
"FirstName":"APi1-Stud-FN",
"MiddleInitial":"Q",
"LastName":"APi1-Stud-LN",
"UserAlternateEmail":"",
"SecretQuestionId":12,
"SecretQuestionAnswer":"Scot",
"UserName":"APi1-stud#xyz.com",
"VerifyUserName":"APi1-stud#xyz.com",
"Password":"A123456",
"VerifyPassword":"A123456",
"YKey":"123xyz",
"YId":6,
"Status":false,
"KeyCode":"",
"SsoUserName":"APi1-stud#xyz.com",
"SsoPassword":"",
"BirthYear":2001
}
So everything on Posting the request from "Rest Assured" point of view looks fine, it's just that I want to update a few values from the above JSON body using JAVA so that I can create a new Student profile every time I run my function and don't have to manually change the Body.
For Every POST Student Account Creation scenario, I need to update the value for
the following keys so that a new test student user account can be created:
First Name
Last Name and
Username // "VerifyUserName" and "SSO UserName" will remain same as user name

I modified the answer to get random values and pass them to json body. random value generation was taken from the accepted answer of this question.
public void testMethod() {
List<String> randomValueList = new ArrayList<>();
for (int i = 0; i < 3; i++) {
String SALTCHARS = "ABCDEFGHIJKLMNOPQRSTUVWXYZ1234567890";
StringBuilder salt = new StringBuilder();
Random rnd = new Random();
while (salt.length() < 18) { // length of the random string.
int index = (int) (rnd.nextFloat() * SALTCHARS.length());
salt.append(SALTCHARS.charAt(index));
}
randomValueList.add(salt.toString());
}
String jsonBody = "{\n" +
" \"FirstName\":\"" + randomValueList.remove(0) + "\",\n" +
" \"MiddleInitial\":\"Q\",\n" +
" \"LastName\":\"" + randomValueList.remove(0) + "\",\n" +
" \"UserAlternateEmail\":\"\",\n" +
" \"SecretQuestionId\":12,\n" +
" \"SecretQuestionAnswer\":\"Scot\",\n" +
" \"UserName\":\"" + randomValueList.remove(0) + " \",\n" +
" \"VerifyUserName\":\"APi1-stud#xyz.com\",\n" +
" \"Password\":\"A123456\",\n" +
" \"VerifyPassword\":\"A123456\",\n" +
" \"YKey\":\"123xyz\",\n" +
" \"YId\":6,\n" +
" \"Status\":false,\n" +
" \"KeyCode\":\"\",\n" +
" \"SsoUserName\":\"APi1-stud#xyz.com\",\n" +
" \"SsoPassword\":\"\",\n" +
" \"BirthYear\":2001\n" +
"}";
Response response = RestAssured
.given()
.body(jsonBody)
.when()
.post("api_url")
.then()
.extract()
.response();
// Do what you need to do with the response body
}

We can used pojo based approach to do certain things very easily . No matter how complex is the payload , serialization and dieselization is the best answer . I have created a framework template for api automation that can we used by putting required POJO's in path :
https://github.com/tanuj-vishnoi/pojo_api_automation
To create pojo, I also have ready to eat food for you :
https://github.com/tanuj-vishnoi/pojo_generator_using_jsonschema2pojo
for the above problem you can refer to the JsonPath lib https://github.com/json-path/JsonPath and use this code:
String mypayload = "{\n" +
" \"FirstName\":\"APi1-Stud-FN\",\n" +
" \"MiddleInitial\":\"Q\",\n" +
" \"LastName\":\"APi1-Stud-LN\"}";
Map map = JsonPath.parse(mypayload).read("$",Map.class);
System.out.println(list);
once the payload converted into map you can change only required values as per the requirement
To generate random strings you can refer to lib org.apache.commons.lang3.RandomStringUtils;
public static String generateUniqueString(int lenghtOfString){
return
RandomStringUtils.randomAlphabetic(lenghtOfString).toLowerCase();
}
I recommend to store payload in a separate file and load it at runtime.

Read and write to different Mongo collections using Spark with Java

I am a relative newbie to Spark. I need to read from a Mongo collection in Java using Spark, change some field values, let's say I am appending "123" to one field value and write into another collection. Accordingly I had 2 separate Mongo URIs as the input and output URIs configured in Spark. I am then proceeding to read from the input collection. However, what I am not understanding is how would I make the same RDD of documents as output to another collection. This is the input code:
String inputUri = "mongodb://" + kp.getProperty("source.mongo.userid") + ":"
+ Encryptor.decrypt(kp.getProperty("source.mongo.cache")) + "#"
+ kp.getProperty("source.mongo.bootstrap-servers") + "/" + kp.getProperty("source.mongo.database")
+ "." + kp.getProperty("source.mongo.inputCollection") + "?ssl=true&connectTimeoutMS="
+ kp.getProperty("source.mongo.connectTimeoutMS") + "&socketTimeoutMS="
+ kp.getProperty("source.mongo.socketTimeoutMS") + "&maxIdleTimeMS="
+ kp.getProperty("source.mongo.maxIdleTimeMS");
String outputUri = "mongodb://" + kp.getProperty("source.mongo.userid") + ":"
+ Encryptor.decrypt(kp.getProperty("source.mongo.cache")) + "#"
+ kp.getProperty("source.mongo.bootstrap-servers") + "/" + kp.getProperty("source.mongo.database")
+ "." + kp.getProperty("source.mongo.outputCollection") + "?ssl=true&connectTimeoutMS="
+ kp.getProperty("source.mongo.connectTimeoutMS") + "&socketTimeoutMS="
+ kp.getProperty("source.mongo.socketTimeoutMS") + "&maxIdleTimeMS="
+ kp.getProperty("source.mongo.maxIdleTimeMS");
SparkSession spark = SparkSession.builder().master("local[3]").appName(kp.getProperty("spark.app.name"))
.config("spark.mongodb.input.uri", inputUri)
.config("spark.mongodb.output.uri", outputUri)
...;
JavaSparkContext sc = new JavaSparkContext(spark.sparkContext());
JavaMongoRDD<Document> rdd = MongoSpark.load(sc);
System.out.println("Count: " + rdd.count());
System.out.println(rdd.first().toJson());
Please help me in this regard.

I have got the answer myself. I went the Dataset route instead of RDDs which made the modification simpler. So, to load the Mongo colection, I use
Dataset<Row> df = MongoSpark.load(sc).toDF();
Then I create a temporary view upon it in orcder to be able to use Spark SQL:
df.createOrReplaceTempView("Customer");
I register an UDF for operating upon each column value:
spark.udf().register("Test", new TestUDF(), DataTypes.StringType);
the UDF definition is as follows:
public class TestUDF implements UDF1<String, String> {
#Override
public String call(String customer) throws Exception {
return customer + "123";
}
}
Then I call the UDF using the same column name as the original so that the values in the original dataset are replaced:
df = df.withColumn("CustomerName", functions.callUDF("Test", functions.col("CustomerName")));
Then I write it back to Mongo in a separate collection:
MongoSpark.write(df).option("collection", "myCollection").save();

Convert HOCON string into Java object

One of my webservice return below Java string:
[
{
id=5d93532e77490b00013d8862,
app=null,
manufacturer=pearsonEducation,
bookUid=bookIsbn,
model=2019,
firmware=[1.0],
bookName=devotional,
accountLinking=mandatory
}
]
I have the equivalent Java object for the above string. I would like to typecast or convert the above java string into Java Object.
I couldn't type-cast it since it's a String, not an object. So, I was trying to convert the Java string to JSON string then I can write that string into Java object but no luck getting invalid character "=" exception.
Can you change the web service to return JSON?
That's not possible. They are not changing their contracts. It would be super easy if they returned JSON.

The format your web-service returns has it's own name HOCON. (You can read more about it here)
You do not need your custom parser. Do not try to reinvent the wheel.
Use an existing one instead.
Add this maven dependency to your project:
<dependency>
<groupId>com.typesafe</groupId>
<artifactId>config</artifactId>
<version>1.3.0</version>
</dependency>
Then parse the response as follows:
Config config = ConfigFactory.parseString(text);
String id = config.getString("id");
Long model = config.getLong("model");
There is also an option to parse the whole string into a POJO:
MyResponsePojo response = ConfigBeanFactory.create(config, MyResponsePojo.class);
Unfortunately this parser does not allow null values. So you'll need to handle exceptions of type com.typesafe.config.ConfigException.Null.
Another option is to convert the HOCON string into JSON:
String hoconString = "...";
String jsonString = ConfigFactory.parseString(hoconString)
.root()
.render(ConfigRenderOptions.concise());
Then you can use any JSON-to-POJO mapper.

Well, this is definitely not the best answer to be given here, but it is possible, at least…
Manipulate the String in small steps like this in order to get a Map<String, String> which can be processed. See this example, it's very basic:
public static void main(String[] args) {
String data = "[\r\n"
+ " {\r\n"
+ " id=5d93532e77490b00013d8862, \r\n"
+ " app=null,\r\n"
+ " manufacturer=pearsonEducation, \r\n"
+ " bookUid=bookIsbn, \r\n"
+ " model=2019,\r\n"
+ " firmware=[1.0], \r\n"
+ " bookName=devotional, \r\n"
+ " accountLinking=mandatory\r\n"
+ " }\r\n"
+ "]";
// manipulate the String in order to have
String[] splitData = data
// no leading and trailing [ ] - cut the first and last char
.substring(1, data.length() - 1)
// no linebreaks
.replace("\n", "")
// no windows linebreaks
.replace("\r", "")
// no opening curly brackets
.replace("{", "")
// and no closing curly brackets.
.replace("}", "")
// Then split it by comma
.split(",");
// create a map to store the keys and values
Map<String, String> dataMap = new HashMap<>();
// iterate the key-value pairs connected with '='
for (String s : splitData) {
// split them by the equality symbol
String[] keyVal = s.trim().split("=");
// then take the key
String key = keyVal[0];
// and the value
String val = keyVal[1];
// and store them in the map ——> could be done directly, of course
dataMap.put(key, val);
}
// print the map content
dataMap.forEach((key, value) -> System.out.println(key + " ——> " + value));
}
Please note that I just copied your example String which may have caused the line breaks and I think it is not smart to just replace() all square brackets because the value firmware seems to include those as content.

In my opinion, we split the parse process in two step.
Format the output data to JSON.
Parse text by JSON utils.
In this demo code, i choose regex as format method, and fastjson as JSON tool. you can choose jackson or gson. Furthermore, I remove the [ ], you can put it back, then parse it into array.
import com.alibaba.fastjson.JSON;
import java.util.regex.Matcher;
import java.util.regex.Pattern;
public class SerializedObject {
private String id;
private String app;
static Pattern compile = Pattern.compile("([a-zA-Z0-9.]+)");
public static void main(String[] args) {
String str =
" {\n" +
" id=5d93532e77490b00013d8862, \n" +
" app=null,\n" +
" manufacturer=pearsonEducation, \n" +
" bookUid=bookIsbn, \n" +
" model=2019,\n" +
" firmware=[1.0], \n" +
" bookName=devotional, \n" +
" accountLinking=mandatory\n" +
" }\n";
String s1 = str.replaceAll("=", ":");
StringBuffer sb = new StringBuffer();
Matcher matcher = compile.matcher(s1);
while (matcher.find()) {
matcher.appendReplacement(sb, "\"" + matcher.group(1) + "\"");
}
matcher.appendTail(sb);
System.out.println(sb.toString());
SerializedObject serializedObject = JSON.parseObject(sb.toString(), SerializedObject.class);
System.out.println(serializedObject);
}
}

JSOUP parsing for multiple rows

I am trying to parse information from a particular website using JSOUP.
So far I can parse and display a single row, as the website has a lot of html and I am quite new to this I was wondering is there a way to parse all table rows on the page containing the word "fixturerow".
Here is my parser code:
Document doc =Jsoup.connect("http://www.irishrugby.ie/club/ulsterbankleagueandcup/fixtures.php").get();
Elements kelime = doc.select("tr#fixturerow0");
for(Element sectd:kelime){
Elements tds = sectd.select("td");
String result = tds.get(0).text();
String result1 = tds.get(1).text();
String result2 = tds.get(2).text();
String result3 = tds.get(3).text();
String result4 = tds.get(4).text();
String result5 = tds.get(5).text();
String result6 = tds.get(6).text();
String result7 = tds.get(7).text();
System.out.println("Date: " + result);
System.out.println("Time: " + result1);
System.out.println("League: " + result2);
System.out.println("Home Team: " + result3);
System.out.println("Score: " + result4);
System.out.println("Away Team: " + result5);
System.out.println("Venue: " + result6);
System.out.println("Ref: " + result7);
}`
Thanks for your time!

You can use the ^= (starts-with) selector:
Elements kelime = doc.select("tr[id^=fixturerow]");
This will return all elements with an id that starts with fixturerow.

You may have better luck if you use a selector that looks for id's that start-with the text of interest. So try changing
Elements kelime = doc.select("tr#fixturerow0");
to
Elements kelime = doc.select("tr[id^=fixturerow]");
Where ^= means that the text of interest starts with the text that follows.

How to parse custom objects from a list within a jsonstring?

I got the following json:
{
"ID": "1234567",
"dangereousCargo": true,
"numberOfPassangers": 164,
"cargo": [
{
"type": "Oil",
"amount": 8556
},
{
"type": "Chemicals",
"amount": 5593
}
]
}
From this question, I understood that it is possible to get the cargoList out of the jsonObject (if that list contains a certain type of object). But how do I get the seperate cargoObjects out of that list?
+Do the variable names of the jsonstring have to correspond with the variable names in my CargoClass? What if the jsonObject only contains type and amount and my CargoClass has more attributes?

You can iterate throws the JSONArray which represents your cargo list doing (not tested)
JSONObject json = new JSONObject(jsonString);
JSONArray cargoList = json.getJSONArray("cargo");
for(int i=0; i< cargoList.length(); i++) {
JSONObject cargo = cargoList.getJSONObject(i);
//Do something with cargo
}

But how do I get the seperate cargoObjects out of that list?
String jsonString = "{ ... }";
JSONObject json = new JSONObject(jsonString);
JSONArray cargoList = json.getJSONArray("cargo");
for(JSONObject cargo : cargoList)
{
//do something with your cargo element
}
Do the variable names of the jsonstring have to correspond with the
variable names in my CargoClass?
If you use the the get method from the JSONObject, you have to specify the exact name of the attribute in your jsonString. Following the example above:
String cargoType = cargo.getString("type");
By the way, if you want to use your already defined CargoClass, you need a Deserializer and all the attributes on your JSON must be the all present and all the same on your CargoClass: I suggest you to take a look at other SOs questions like this one.
What if the jsonObject only contains type and amount and my CargoClass
has more attributes?
The other attributes will be initialize in base of your class declaration

Using JSON Simple it is very easy to parse and read your data from your JSON. As other users have said there are a plethora of libraries out there that can accomplish this. Below is a tested example of your code.
public static void main(String[] args) throws JSONException {
String json = "{\n"
+ " ID: 1234567,\n"
+ " dangereousCargo: true,\n"
+ " numberOfPassangers: 164,\n"
+ " cargo: [\n"
+ " {\n"
+ " type: Oil,\n"
+ " amount: 8556\n"
+ " },\n"
+ " {\n"
+ " type: Chemicals,\n"
+ " amount: 5593\n"
+ " }\n"
+ " ]\n"
+ "}";
JSONObject data = new JSONObject(json);
int id = data.getInt("ID");
boolean danger = data.getBoolean("dangereousCargo");
int numOfPassengers = data.getInt("numberOfPassangers");
System.out.println("Current ID: " + id + "\n"
+ "Is Dangerous: " + danger + "\n"
+ "Number of Passengers: " + numOfPassengers + "\n");
JSONArray cargo = data.getJSONArray("cargo");
NumberFormat currency = NumberFormat.getCurrencyInstance();
for (int i = 0; i < cargo.length(); i++) {
JSONObject cargoObject = cargo.getJSONObject(i);
String type = cargoObject.getString("type");
double amount = cargoObject.getDouble("amount");
System.out.println("Current Type: " + type);
System.out.println("Current Amount: " + currency.format(amount));
}
}
}
The above code gives us the following output:
Once you have your data you can really do whatever you want with it.
Libraries
JSONSimple-https://code.google.com/p/json-simple/
GSON-https://github.com/google/gson