We Keep Coding

How to get .har file or network request using selenium4

As we know One of the features added in the new version of Selenium (4.0.0-alpha-2) is a very nice Interface for Chrome DevTools API in Java.DevTools API offers great capabilities for controlling the Browser and the Web Traffic
As per documentation using the latest version of selenium we can capture the network request from the session.
Before I used browsermob for getting the network request but unfortunately they didn't update it a couple of years.
I am looking for someone who used this selenium4 dev tools API for getting all the internal request.
Can anyone suggest me how can I start to get all the requests? Thanks, advance

You can find #adiohana's example in the selenium-chrome-devtools-examples repo on gitHub.
I think youed find this test example helpful:
public class ChromeDevToolsTest {
private static ChromeDriver chromeDriver;
private static DevTools chromeDevTools;
#BeforeClass
public static void initDriverAndDevTools() {
chromeDriver = new ChromeDriver();
// dev-tools handler
chromeDevTools = chromeDriver.getDevTools();
chromeDevTools.createSession();
}
#Test
public void interceptRequestAndContinue() {
//enable Network
chromeDevTools.send(Network.enable(Optional.empty(), Optional.empty(), Optional.empty()));
//add listener to intercept request and continue
chromeDevTools.addListener(Network.requestIntercepted(),
requestIntercepted -> chromeDevTools.send(
Network.continueInterceptedRequest(requestIntercepted.getInterceptionId(),
Optional.empty(),
Optional.empty(),
Optional.empty(), Optional.empty(),
Optional.empty(),
Optional.empty(), Optional.empty())));
//set request interception only for css requests
RequestPattern requestPattern = new RequestPattern("*.css", ResourceType.Stylesheet, InterceptionStage.HeadersReceived);
chromeDevTools.send(Network.setRequestInterception(ImmutableList.of(requestPattern)));
chromeDriver.get("https://apache.org");
}
You need to add the following imports:
import com.google.common.collect.ImmutableList;
import com.google.common.collect.ImmutableMap;
import org.junit.AfterClass;
import org.junit.Assert;
import org.junit.BeforeClass;
import org.junit.Test;
import org.openqa.selenium.chrome.ChromeDriver;
import org.openqa.selenium.devtools.Command;
import org.openqa.selenium.devtools.Console;
import org.openqa.selenium.devtools.DevTools;
import org.openqa.selenium.devtools.network.Network;
import org.openqa.selenium.devtools.network.model.BlockedReason;
import org.openqa.selenium.devtools.network.model.InterceptionStage;
import org.openqa.selenium.devtools.network.model.RequestPattern;
import org.openqa.selenium.devtools.network.model.ResourceType;
import org.openqa.selenium.devtools.security.Security;
import java.util.Optional;

Glassfish & MongoDB connection error : NoClassDefFoundError

I am running a Glassfish server that is trying to connect to MongoDB. At first I created seperate projects for the server and MongoDB. So now I am trying to merge those projects but it appears anything I try to do it results in a faliure.
The current error I am getting is:
2018-07-05T19:54:36.249+0200|Severe: java.lang.NoClassDefFoundError: org/bson/conversions/Bson
I am well aware that the error happens in runtime and that the possible cause is my classpath.
Currently I copied all of my code from one project to another, added Maven dependencies and the following happens:
if I create a separate .java file for my MongoDB and run it in the same folder that the Glassfish server is, it works perfectly fine.
if I run the server and try to call methods from the other class (a little bit modified) the upper error appears
Simplified code example withouth error:
import org.bson.Document;
import org.bson.conversions.Bson;
import com.mongodb.BasicDBObject;
import com.mongodb.MongoClient;
import com.mongodb.client.FindIterable;
import com.mongodb.client.MongoCollection;
import com.mongodb.client.MongoDatabase;
import com.mongodb.client.MongoIterable;
import com.mongodb.client.model.Filters;
import com.mongodb.client.model.Updates;
public class MyClass{
public static void main(String[]args){
String ip = "127.0.0.1";
int port = 27017;
MongoClient mongoClient = new MongoClient(ip,port);
/* Remaining code */
}
}
With error:
import org.bson.Document;
import org.bson.conversions.Bson;
import com.mongodb.BasicDBObject;
import com.mongodb.MongoClient;
import com.mongodb.client.FindIterable;
import com.mongodb.client.MongoCollection;
import com.mongodb.client.MongoDatabase;
import com.mongodb.client.MongoIterable;
import com.mongodb.client.model.Filters;
import com.mongodb.client.model.Updates;
public class MyClass{
private MongoClient mongoClient;
public MyClass(String ip, int port){
mongoClient = new MongoClient(ip, port); // Error called here
}
/* Remaining code */
}
Called from the server.java file:
MyClass mc = new MyClass("127.0.0.1",27017);
I also tried to download all of the bson jar files separately and add them to the project but that had no effect...

The working solution for me was to delete the whole project and create it once more. Apparently there was a problem with Eclipse or I made a mistake before and forgot about it.

Apache Spark with elasticsearch V5.X

I am a novice in Java and I am looking for some examples of connector between elasticsearch V5.X and Spark in order to see some use cases.
At the moment here is my code :
package Spark;
import org.apache.hadoop.conf.Configuration;
import org.apache.log4j.Level;
import org.apache.spark.SparkConf;
import org.apache.spark.api.java.JavaSparkContext;
import org.junit.Test;
import com.google.common.collect.ImmutableList;
import com.google.common.collect.ImmutableMap;
import scala.collection.immutable.Map;
import twitter4j.Status;
import org.apache.spark.api.java.JavaSparkContext;
import org.apache.spark.api.java.JavaRDD;
import org.apache.spark.SparkConf;
import org.elasticsearch.spark.rdd.api.java.JavaEsSpark;
public class EsSpark {
public EsSpark(){
SparkConf conf = new SparkConf().setAppName("MyApp1").setMaster("localhost");
conf.set("es.index.auto.create", "true");
JavaSparkContext jsc = new JavaSparkContext(conf);
Map<String, ?> numbers = (Map<String, ?>) ImmutableMap.of("one", 1, "two", 2);
Map<String, ?> airports = (Map<String, ?>) ImmutableMap.of("OTP", "Otopeni", "SFO", "San Fran");
JavaRDD<Map<String, ?>> javaRDD = jsc.parallelize(ImmutableList.of(numbers, airports));
JavaEsSpark.saveToEs(javaRDD, "spark/docs");
}
}
Thanks.

Except if you are working with a local instance of Elasticsearch, there are some important settings to be provided, notably es.nodes.
You can do it using
conf.set("es.nodes", "eshost:9200");
You can even specify multiple instances, prefer master nodes, but not all nodes are required.
Please refer to official documentation.
People at discussion forum at elastic often publish some code you can use as example.
Ensure to provide several documents as the EsSpark or EsSparkStreaming objects. Do not send 1 document each time, prefer multiple documents.
EsSpark or EsSparkStreaming connect to the nodes you provide, they check for the cluster topology (number of nodes, types of nodes) and they will send data directly to data nodes and to the correct shard (avoiding hops).
It is possible to prevent to push data directly to the data nodes (using the settings specified in this section of the documentation), but you will introduce bottlenecks.

Why is my import of the containsString method not working?

I've written the simple Java script below in order to learn more about TDD, IntelliJ and Java itself.
import java.util.ArrayList;
import java.util.List;
import org.junit.Before;
import org.junit.Test;
import static org.hamcrest.CoreMatchers.containsString;
import static org.hamcrest.CoreMatchers.is;
import static org.junit.Assert.assertThat;
import static org.junit.matchers.JUnitMatchers.both;
public class JUnit_Dummy {
private StringJoiner joiner;
private List<String> strings;
#Before
public void setUp() throws Exception {
strings = new ArrayList<String>();
joiner = new StringJoiner();
}
....
#Test
public void shouldContainBothStringsWhenListIsTwoStrings() {
strings.add("one");
strings.add("two");
assertThat(joiner.join(strings),
both(containsString("A")).
and(containsString("B")));
}
}
_____________
import java.util.List;
public class StringJoiner {
public String join(List<String> strings) {
if(strings.size() > 0) {
return (strings.get(0);
}
return "";
}
}
I'm trying to use the "containsString" method inside an assertion, but IntelliJ keeps telling me that it "cannot resolve method 'containsString(java.lang.String)". This despite the fact that the jUnit docs (http://junit.sourceforge.net/javadoc/org/junit/matchers/JUnitMatchers.html#containsString(java.lang.String)) tell me that this method does accept a String parameter.
I've tried swapping out various import statements, including the following:
import static org.hamcrest.Matcher.containsString;
import static org.hamcrest.Matcher.*;
import static org.hamcrest.CoreMatchers.*;
The best that I get is a greyed-out import statement telling me that the import statement is unused. Not sure what the problem is, any help would be appreciated.
UPDATE:
Here is the exact compiler error:
java: cannot find symbol
symbol: method containsString(java.lang.String)
location: class JUnit_Dummy

I thought I had tried every worthwhile import statement already, but this one did the trick:
import static org.junit.matchers.JUnitMatchers.*;

I faced the same issue with a Spring Boot app.
Seems like this is a dependency ordering issue.. one of the dependencies mentioned in pom.xml before the "spring-boot-starter-test" artifact was overriding the hamcrest version.
So all I did was change the order (moved this dependency up):
<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-test</artifactId>
<scope>test</scope>
</dependency>
I'm using Spring Boot 1.5.7.RELEASE.

We are supposed to use containsString method of hamcrest library.
My suggestion would be to stick to Junit 4 and import hamcrest library 1.3 in your build path. This would do the trick.
This will allow you to access other features of hamcrest library as well.
The solution can also be found by adding the required static imports manually. Or you can configure the required static imports in favorites tab of eclipse.

try this instead
import static org.hamcrest.CoreMatchers.*;

I'm working with MAVEN - doing a tutorial and I ran into this same issue.
I used the "import static org.hamcrest.CoreMatchers.*;" solution and that failed.
So I then moved JUNIT to be first on the list in the POM file - and that solved it.

Play Framework 2.0 and EBean, wrapping INFORMATION_SCHEMA

Using Play! Framework 2.0.4 and EBean as the persistence layer I am attempting to wrap a databases "meta" information to Java classes.
I map the classes in application.conf:
db.myData.url="jdbc:sqlserver://localhost:2301;databaseName=myData;"
ebean.myData="models.database.JavaTable"
I then created a class as follows:
package models.database;
import javax.persistence.Column;
import javax.persistence.Entity;
import javax.persistence.Table;
import play.db.ebean.Model;
#Entity
#Table(name="tables",schema="INFORMATION_SCHEMA",catalog="myData")
public class JavaTable extends Model{
#Column(name="TABLE_NAME")
public String table_name;
public static Finder<String, JavaTable> find = new Finder<String, JavaTable>(
String.class, JavaTable.class
);
}
When I fire up Play!, it tells me that I need to run an evolution on the database to create the table "myData.INFORMATION_SCHEMA.tables". I then tried to test the connection via a unit test...
#Test
public void testGetTables(){
running(fakeApplication(), new Runnable() {
#Override
public void run() {
EbeanServer server = Ebean.getServer("myData");
List<SqlRow> rows = server.createSqlQuery("select * from myData.Information_Schema.Tables").findList();
for(SqlRow row: rows)
System.out.println("====>Row: " + row.toString());
}
});
}
The unit test executes correctly and the table names are printed out successfully.
Edit: per #nico_ekito I removed the evolution plugin in the configuration file and started getting:
RuntimeException DataSource user is null
So, researching a bit I decided to disable other datasources in the config file and moved the database I'm attempting to communicate with to "db.default" and "ebean.default" and the models started working. Is there a way to tell a model which datasource it should use if Play has multiple databases defined? Setting the classes via "ebean.default="myData.JavaTable" did not seem to work.

You should try to disable the evolutions by adding the following in your application.conf file:
evolutionplugin=disabled
The tests are ok because since you're not starting a real application, but using a fakeApplication() which does not use the evolutions plugin.

Anecdotally, I've only had success defining my eBean server using package level values (so, "myData.*" instead of "myData.JavaTable"). To pull this off, you may have to move all classes for a particular eBean server to their own package.