Is there a library that would parse fragment and query from String to URI even if the string has the fragment and query in a wrong order?
import java.net.URI;
public class UriTest {
public static void main(String[] args){
String validUriString = "http://localhost:4200/?name=value#/fragment";
String invalidUriString = "http://localhost:4200/#/fragment?name=value";
URI validUri = URI.create(validUriString);
URI invalidUri = URI.create(invalidUriString);
System.out.println("OK: "+validUri.getFragment());
System.out.println("OK: "+validUri.getQuery()+"\n");
System.out.println("NOT OK: "+invalidUri.getFragment());
System.out.println("NOT OK: "+invalidUri.getQuery());
}
}
Output:
OK: /fragment
OK: name=value
NOT OK: /fragment?name=value
NOT OK: null
Would be nice to get the same result in both cases.
No guarantees if this will work for all URLs, but you can try to check if the url is broken and fix it like this:
...
public static String fixUrl(String url) {
int fragmentIndex = url.indexOf("#");
int queryIndex = url.indexOf("?");
if(fragmentIndex < queryIndex && fragmentIndex > -1)
return url.substring(0,fragmentIndex) + url.substring(queryIndex) + url.substring(fragmentIndex, queryIndex);
else
return url;
}
public static void main (String[] args) {
String validUriString = "http://localhost:4200/?name=value#/fragment";
String invalidUriString = "http://localhost:4200/#/fragment?name=value";
URI validUri = URI.create(fixUrl(validUriString));
URI invalidUri = URI.create(fixUrl(invalidUriString));
System.out.println("OK: "+validUri.getFragment());
System.out.println("OK: "+validUri.getQuery()+"\n");
System.out.println("OK: "+invalidUri.getFragment());
System.out.println("OK: "+invalidUri.getQuery());
}
Related
Currently I'm doing a program which retrieve the Azure Client ID and Secret Value through key-vault.
Below is the logic that my friend and I make to get the value, my question is how can I take the value that I got in the static void main and pass to another class for use? I had no idea how to reuse the value I get in another class. Please teach me.
public class SecretReceiver {
private static SecretClient secretClient;
public static final String AZURE_CLIENT_ID="AZURE_CLIENT_ID";
public static final String AZURE_CLIENT_SECRET="AZURE_CLIENT_SECRET";
public static final String AZURE_TENANT_ID="AZURE_TENANT_ID";
public static final String AZURE_KEY_VAULT_NAME="AZURE_KEY_VAULT_NAME";
private static final String KEY_VAULT_URL = "https://%s.vault.azure.net";
private static void secretReceiverBuilder() {
if (secretClient == null) {
String keyVaultUrl = String.format(KEY_VAULT_URL, getProperty(AZURE_KEY_VAULT_NAME, ""));
secretClient = new SecretClientBuilder()
.vaultUrl(keyVaultUrl)
.credential(new ClientSecretCredentialBuilder()
.clientId(getProperty(AZURE_CLIENT_ID, ""))
.clientSecret(getProperty(AZURE_CLIENT_SECRET, ""))
.tenantId(getProperty(AZURE_TENANT_ID, ""))
.build())
.buildClient();
}
}
public static void loadConfigFileAndSetEnv(String filePath) {
try (BufferedReader reader = new BufferedReader(new InputStreamReader(new FileInputStream(new File(filePath))))) {
String line = null;
while ((line = reader.readLine()) != null) {
String[] split = line.split("=");
if (split.length > 1) {
String key = split[0].trim();
String value = split[1].trim();
if (key.contains(AZURE_CLIENT_ID)) {
System.setProperty(AZURE_CLIENT_ID, value);
continue;
}
if (key.contains(AZURE_CLIENT_SECRET)) {
System.setProperty(AZURE_CLIENT_SECRET, value);
continue;
}
if (key.contains(AZURE_TENANT_ID)) {
System.setProperty(AZURE_TENANT_ID, value);
continue;
}
if (key.contains(AZURE_KEY_VAULT_NAME)) {
System.setProperty(AZURE_KEY_VAULT_NAME, value);
}
}
}
} catch (Exception e) {
log.error("Load file : {} error.", filePath, e);
}
}
public static String getProperty(String key, String defaultValue) {
String value = System.getProperty(key);
if (StringUtils.isBlank(value)) {
value = System.getenv(key);
}
return StringUtils.isBlank(value) ? defaultValue : value;
}
public static void main() throws Exception {
//getUatAKV();
loadConfigFileAndSetEnv("C:script\\key_vault.conf");
String username = getSecretByKey("client-secret-name");
String secret = getSecretByKey("client-secret");
System.out.println("This is client id: " + username);
System.out.println("This is client secret: " + secret);
}
public static String getSecretByKey(String name) {
if (secretClient == null) {
secretReceiverBuilder();
}
return secretClient.getSecret(name).getValue();
}
Try the following code:
// Create static variable in a Class
public class Global {
public static String USER_NAME ="";
public static String SECRET ="";
}
// Set the value in your main function
String username = getSecretByKey("client-secret-name");
String secret = getSecretByKey("client-secret");
Global.USER_NAME = username;
Global.SECRET = secret;
// Get value in another class
System.out.println("This is client id: " + Global.USER_NAME);
System.out.println("This is client secret: " + Global.SECRET);
I am new to the HTTP request in java. I have been trying to send an HTTP Post request to my NODE.JS server with the parameter key:12345. However, it doesn't send anything to my server. I tried tested my NOEDJS server to see if it worked in POSTMAN, and it did. So I am sure that this is something with the java that I made. I think a look at my code would help. Here it is down below.
import java.io.*;
import java.net.HttpURLConnection;
import java.net.URL;
import java.util.HashMap;
import java.util.Map;
public class ConnectionFactory {
private double API_VERSION = 0;
private String API = "";
private String METHOD = "POST";
private String USER_AGENT = "Mozilla/5.0";
private String TYPE = "application/x-www-form-urlencoded";
private String data = "";
private URL connection;
private HttpURLConnection finalConnection;
private HashMap<String, String> fields = new HashMap<String, String>();
public ConnectionFactory(String[] endpoint, String url, double version) {
this.API_VERSION = version;
this.API = url;
fields.put("version", String.valueOf(version));
for (int i = 0; i < endpoint.length; i++) {
String[] points = endpoint[i].split(";");
for (int f = 0; f < points.length; f++) {
fields.put(points[f].split(":")[0], points[f].split(":")[1]);
}
}
}
public String buildConnection() {
StringBuilder content = new StringBuilder();
if (!this.getEndpoints().equalsIgnoreCase("") && !this.getEndpoints().isEmpty()) {
String vars = "";
String vals = "";
try {
for (Map.Entry<String, String> entry: fields.entrySet()) {
vars = entry.getKey();
vals = entry.getValue();
data += ("&" + vars + "=" + vals);
}
if (data.startsWith("&")) {
data = data.replaceFirst("&", "");
}
connection = new URL(API);
BufferedReader reader = new BufferedReader(new InputStreamReader(readWithAccess(connection, data)));
String line;
while ((line = reader.readLine()) != null) {
content.append(line + "\n");
}
reader.close();
return content.toString();
} catch (Exception e) {
System.err.println(e.getMessage());
}
} else {
return null;
}
return null;
}
private InputStream readWithAccess(URL url, String data) {
try {
byte[] out = data.toString().getBytes();
finalConnection = (HttpURLConnection) url.openConnection();
finalConnection.setRequestMethod(METHOD);
finalConnection.setDoOutput(true);
finalConnection.addRequestProperty("User-Agent", USER_AGENT);
finalConnection.addRequestProperty("Content-Type", TYPE);
finalConnection.connect();
try {
OutputStream os = finalConnection.getOutputStream();
os.write(out);
} catch (Exception e) {
System.err.println(e.getMessage());
}
return finalConnection.getInputStream();
} catch (Exception e) {
System.err.println(e.getMessage());
return null;
}
}
public String getApiVersion() {
return String.valueOf(API_VERSION);
}
public String getEndpoints() {
return fields.toString();
}
public String getEndpointValue(String key) {
return fields.get(key);
}
public void setUserAgent(String userAgent) {
this.USER_AGENT = userAgent;
}
public void setMethod(String method) {
this.METHOD = method;
}
public void setSubmissionType(String type) {
this.TYPE = type;
}
}
public class example {
public static void main(String[] args) {
double version = 0.1;
String url = "http://localhost:3000";
String[] fields = {
"key:12345"
};
ConnectionFactory connection = new ConnectionFactory(fields, url, version);
connection.setUserAgent("Mozilla/5.0");
String response = connection.buildConnection();
System.out.println(response);
}
}
Here is the code for my node.js server
var http = require('http');
var url = require('url');
var queryString = require('querystring')
var StringDecoder = require('string_decoder').StringDecoder;
var server = http.createServer(function(req, res) {
//parse the URL
var parsedURL = url.parse(req.url, true);
//get the path
var path = parsedURL.pathname;
var trimmedPath = path.replace(/^\/+|\/+$/g, '');
//queryString
var queryStringObject = parsedURL.query;
console.log(queryStringObject);
if (queryStringObject.key == 12345) {
console.log("true")
res.end("true")
} else {
console.log("failed")
res.end("false")
}
// var query = queryStringObject.split()
});
server.listen(3000, function() {
console.log("Listening on port 3000");
});
The is no problem with your java client
The problem is that you are sending the content of your POST request as ""application/x-www-form-urlencoded" and then in your nodeJS server you are reading it as a query string
Here is a correct example using ExpressJS :
const express = require('express')
const app = express()
app.get('/', function (req, res) {
res.send('Hello World!')
})
var bodyParser = require('body-parser');
app.use(bodyParser.json()); // support json encoded bodies
app.use(bodyParser.urlencoded({ extended: true })); // support encoded bodies
app.post('/test', function(req, res) {
var key = req.body.key;
if (key==12345)
res.send(true );
else
res.send(false);
});
app.listen(3000, function () {
console.log('Example app listening on port 3000!')
})
I am new to Jsoup parsing and I want to get the list of all the companies on this page: https://angel.co/companies?company_types[]=Startup
Now, a way to do this is actually to inspect the page with the div tags relevant to what I need.
However, when I call the method :
Document doc = Jsoup.connect("https://angel.co/companies?company_types[]=Startup").get();
System.out.println(doc.html());
Firstly I cannot even find those DIV tags in my consol html output, (the ones which are supposed to give a list of the companies)
Secondly, even if I did find it, how can I find a certain Div element with class name :
div class=" dc59 frw44 _a _jm"
Pardon the jargon, I have no idea how to go through this.
The data are not embedded in the page but they are retrieved using subsequent API calls :
a POST https://angel.co/company_filters/search_data to get an ids array & a token named hexdigest
a GET https://angel.co/companies/startups to retrieve company data using the output from the previous request
The above is repeated for each page (thus a new token & a list of ids are needed for each page). This process can be seen using Chrome dev console in Network tabs.
The first POST request gives JSON output but the second request (GET) gives HTML data in a property of a JSON object.
The following extracts the company filter :
private static CompanyFilter getCompanyFilter(final String filter, final int page) throws IOException {
String response = Jsoup.connect("https://angel.co/company_filters/search_data")
.header("Content-Type", "application/x-www-form-urlencoded;charset=UTF-8")
.header("X-Requested-With", "XMLHttpRequest")
.data("filter_data[company_types][]=", filter)
.data("sort", "signal")
.data("page", String.valueOf(page))
.userAgent("Mozilla")
.ignoreContentType(true)
.post().body().text();
GsonBuilder gsonBuilder = new GsonBuilder();
Gson gson = gsonBuilder.create();
return gson.fromJson(response, CompanyFilter.class);
}
Then the following extracts companies :
private static List<Company> getCompanies(final CompanyFilter companyFilter) throws IOException {
List<Company> companies = new ArrayList<>();
URLConnection urlConn = new URL("https://angel.co/companies/startups?" + companyFilter.buildRequest()).openConnection();
urlConn.setRequestProperty("User-Agent", "Mozilla");
urlConn.connect();
BufferedReader reader = new BufferedReader(new InputStreamReader(urlConn.getInputStream(), "UTF-8"));
HtmlContainer htmlObj = new Gson().fromJson(reader, HtmlContainer.class);
Element doc = Jsoup.parse(htmlObj.getHtml());
Elements data = doc.select("div[data-_tn]");
if (data.size() > 0) {
for (int i = 2; i < data.size(); i++) {
companies.add(new Company(data.get(i).select("a").first().attr("title"),
data.get(i).select("a").first().attr("href"),
data.get(i).select("div.pitch").first().text()));
}
} else {
System.out.println("no data");
}
return companies;
}
The main function :
public static void main(String[] args) throws IOException {
int pageCount = 1;
List<Company> companies = new ArrayList<>();
for (int i = 0; i < 10; i++) {
System.out.println("get page n°" + pageCount);
CompanyFilter companyFilter = getCompanyFilter("Startup", pageCount);
pageCount++;
System.out.println("digest : " + companyFilter.getDigest());
System.out.println("count : " + companyFilter.getTotalCount());
System.out.println("array size : " + companyFilter.getIds().size());
System.out.println("page : " + companyFilter.getpage());
companies.addAll(getCompanies(companyFilter));
if (companies.size() == 0) {
break;
} else {
System.out.println("size : " + companies.size());
}
}
}
Company, CompanyFilter & HtmlContainer are model class :
class CompanyFilter {
#SerializedName("ids")
private List<Integer> mIds;
#SerializedName("hexdigest")
private String mDigest;
#SerializedName("total")
private String mTotalCount;
#SerializedName("page")
private int mPage;
#SerializedName("sort")
private String mSort;
#SerializedName("new")
private boolean mNew;
public List<Integer> getIds() {
return mIds;
}
public String getDigest() {
return mDigest;
}
public String getTotalCount() {
return mTotalCount;
}
public int getpage() {
return mPage;
}
private String buildRequest() {
String out = "total=" + mTotalCount + "&";
out += "sort=" + mSort + "&";
out += "page=" + mPage + "&";
out += "new=" + mNew + "&";
for (int i = 0; i < mIds.size(); i++) {
out += "ids[]=" + mIds.get(i) + "&";
}
out += "hexdigest=" + mDigest + "&";
return out;
}
}
private static class Company {
private String mLink;
private String mName;
private String mDescription;
public Company(String name, String link, String description) {
mLink = link;
mName = name;
mDescription = description;
}
public String getLink() {
return mLink;
}
public String getName() {
return mName;
}
public String getDescription() {
return mDescription;
}
}
private static class HtmlContainer {
#SerializedName("html")
private String mHtml;
public String getHtml() {
return mHtml;
}
}
The full code is also available here
i trie to search in google with JSoup. The problem that i have is, that the variable query shows not the URL i want when i start searching.
Also, how does Jsoup search ? Looking for Title or URL or what ?
public class Start {
public static void main(String[] args) {
try {
new Google().Searching("Möbel Beck GmbH & Co.KG");
} catch (Exception e) {
System.out.println(e.getMessage());
}
}
}
public class Google implements Serializable {
private static final long serialVersionUID = 1L;
private static Pattern patternDomainName;
private Matcher matcher;
private static final String DOMAIN_NAME_PATTERN = "([a-zA-Z0-9]([a-zA-Z0-9\\-]{0,61}[a-zA-Z0-9])?\\.)+[a-zA-Z]{2,6}";
static {
patternDomainName = Pattern.compile(DOMAIN_NAME_PATTERN);
}
public void Searching(String searchstring) throws IOException {
Google obj = new Google();
Set<String> result = obj.getDataFromGoogle(searchstring);
for (String temp : result) {
if (temp.contains(searchstring)) {
System.out.println(temp + " ----> CONTAINS");
} else {
System.out.println(temp);
}
}
System.out.println(result.size());
}
public String getDomainName(String url) {
String domainName = "";
matcher = patternDomainName.matcher(url);
if (matcher.find()) {
domainName = matcher.group(0).toLowerCase().trim();
}
return domainName;
}
private Set<String> getDataFromGoogle(String query) {
Set<String> result = new HashSet<String>();
String request = "https://www.google.com/search?q=" + query;
System.out.println("Sending request..." + request);
try {
// need http protocol, set this as a Google bot agent :)
Document doc = Jsoup.connect(request)
.userAgent("Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)").timeout(6000)
.get();
// get all links
Elements links = doc.select("a[href]");
for (Element link : links) {
String temp = link.attr("href");
if (temp.startsWith("/url?q=")) {
// use regex to get domain name
result.add(getDomainName(temp));
}
}
} catch (IOException e) {
e.printStackTrace();
}
return result;
}
}
Parsing google sites directly is not a good idea. You can try Google API
https://developers.google.com/web-search/docs/#java-access
I need to retrieve the host name from the address bar using java code. For example, if "www.stackoverflow.com/questions", is there in the address bar, I have to get "www.stackoverflow.com". Please help me.
If you know a bit about format, it can be done very easily with this code :
String http = "www.stackoverflow.com/questions";
String url = http.substring(0, http.indexOf("/"));
System.out.println(url);
http = "http://www.stackoverflow.com/questions";
String nohttp = http.substring(7, http.length());
url = nohttp.substring(0, nohttp.indexOf("/"));
System.out.println(url);
Or you can use some "nicer" approach with methods :
public static void main(String[] args) {
System.out.println(getHostname("www.stackoverflow.com/questions"));
System.out.println(getHostname("http://www.stackoverflow.com/questions"));
}
public static boolean isHttp(String s){
if (s.indexOf("http://") == 0){
return true;
} else {
return false;
}
}
public static String getHostname(String url){
String nativeUrl = url;
if (isHttp(nativeUrl)){
url = url.substring(7);
}
url = url.substring(0, url.indexOf("/"));
if (isHttp(nativeUrl)){
url = "http://" + url;
}
return url;
}
Output
www.stackoverflow.com
http://www.stackoverflow.com