Recently, I decided to try and create a java program that would retrieve the source code of a webpage after logging into the site. I've searched SO threads fairly tediously and found many (helpful) answers, but am having trouble understanding why my code will only return a non-logged-in Document for the webpage. I assume there is an error with using cookies from the log in, but am not sure. All help is appreciated!
import java.io.*;
import java.net.MalformedURLException;
import java.util.Map;
import java.net.*;
import org.jsoup.Connection;
import org.jsoup.Connection.Method;
import org.jsoup.Jsoup;
import org.jsoup.nodes.Document;
public class PageLogin {
private static Map<String, String> loginCookies;
public PageLogin() {
login();
}
private static void login() {
try {
Connection.Response res = Jsoup.connect("https://connection.naviance.com/family-connection/auth/login/")
.data("e-mail", "myEmail")
.data("password", "myPass")
.method(Method.POST)
.execute();
loginCookies = res.cookies();
} catch (MalformedURLException ex) {
System.out.println("The URL specified was unable to be parsed or uses an invalid protocol. Please try again.");
System.exit(1);
} catch (Exception ex) {
System.out.println(ex.getMessage() + "\nAn exception occurred.");
System.exit(1);
}
}
public Document getDoc(String url){
try {
return Jsoup.connect(url)
.cookies(loginCookies)
.get();
} catch (IOException e) {}
return null;
}
public static void main(String[] args) throws IOException {
PageLogin test1 = new PageLogin();
Document doc = test1.getDoc("https://connection.naviance.com/family-connection/main/");
System.out.println(doc);
}}
Related
I am trying to retrieve all groups and respective users from Active Directory LDAP servers.
Can anyone please help with sample code?
package test;
import java.util.List;
import com.unboundid.ldap.sdk.LDAPConnection;
import com.unboundid.ldap.sdk.LDAPException;
import com.unboundid.ldap.sdk.LDAPSearchException;
import com.unboundid.ldap.sdk.SearchRequest;
import com.unboundid.ldap.sdk.SearchResult;
import com.unboundid.ldap.sdk.SearchResultEntry;
import com.unboundid.ldap.sdk.SearchScope;
public class Test2 {
public static void main(String[] args) {
try {
List<SearchResultEntry> usersFromLdap = getUsersFromLdap();
System.out.println(usersFromLdap);
} catch (LDAPException e) {
//TODO handle exception
}
}
public static List<SearchResultEntry> getUsersFromLdap() throws LDAPException, LDAPSearchException{
String searchBaseDN = "dc=your-domain,dc=com"; //your-domain.com
String searchFilter = "(&(objectClass=user)(sn=*))"; //see e.g. https://confluence.atlassian.com/kb/how-to-write-ldap-search-filters-792496933.html
LDAPConnection connection = new LDAPConnection("host.your-domain.com", 389);
try {
connection.bind("yourLdapUser", "YourLdapPassword");
SearchRequest request = new SearchRequest(searchBaseDN, SearchScope.SUB, searchFilter);
request.setSizeLimit(0);
SearchResult searchResult = connection.search(request);
List<SearchResultEntry> result = searchResult.getSearchEntries();
return result;
} finally {
if (connection != null) {
connection.close();
}
}
}
}
I am trying to get some data from a website. I make copy paste from my old program. But its not working. My code is below.
import java.io.IOException;
import javax.swing.JOptionPane;
import org.jsoup.Jsoup;
import org.jsoup.Connection.Response;
import org.jsoup.nodes.Document;
import org.jsoup.nodes.Element;
public class Veri {
public static void main(String[] args) {
Veri();
}
public static void Veri() {
try {
String url = "https://www.isyatirim.com.tr/tr-tr/analiz/hisse/Sayfalar/default.aspx";
Response res = Jsoup.connect(url).timeout(6000).execute();
Document doc = res.parse();
Element ele = doc.select("table[class=dataTable hover nowrap excelexport data-tables no-footer]").first();
for (int i = 0; i < 100; i++) {
System.out.println(ele.select("td").iterator().next().text());
}
} catch (IOException c) {
JOptionPane.showMessageDialog(null, "Veriler Alınırken Bir Harta Oluştu!");
c.printStackTrace();
}
}
}
I got the below error
Exception in thread "main" java.lang.NullPointerException at
Veri.Veri(Veri.java:37) at Veri.main(Veri.java:20)
The page has probably changed a little bit since you last used your program.
Try this:
import org.jsoup.Jsoup;
import org.jsoup.Connection.Response;
import org.jsoup.nodes.Document;
import org.jsoup.nodes.Element;
import org.jsoup.select.Elements;
public class Veri {
public static void main(String[] args) {
Veri();
}
public static void Veri() {
try {
String url = "https://www.isyatirim.com.tr/tr-tr/analiz/hisse/Sayfalar/default.aspx";
Response res = Jsoup.connect(url).timeout(6000).execute();
Document doc = res.parse();
Element ele = doc.select("table[class=dataTable hover nowrap excelexport]").first();
Elements lines = ele.select("tr");
for (Element elt : lines) {
System.out.println(elt.text());
System.out.println("------------------------");
}
} catch (IOException c) {
JOptionPane.showMessageDialog(null, "Veriler Alınırken Bir Harta Oluştu!");
c.printStackTrace();
}
}
}
I think you get all the information needed this way.
Can you give me an example for create DKIM with jDKIM?
I don't see any example for this.
http://james.apache.org/jdkim/
Thanks!
Use: james-jdkim
Have a look at this code, This is the answer you are looking for:
lima here is the selector and domain is yahoogroups.com
PublicKeyRecordRetriever publicKeyRecordRetriever = new DNSPublicKeyRecordRetriever();
PublicKeyRecord keys = new DKIMVerifier()
.publicKeySelector(publicKeyRecordRetriever.getRecords("dns/txt", "lima", "yahoogroups.com"));
String signedMIME = "DKIM-Signature: a=rsa-sha256; b=xxxxxxxxxxxxxxxxxx; s=lima; d=yahoogroups.com; v=1; bh=xxxxxxxxx; h=from:to;"
+ "From: test#yahoogroups.com" + "To: test#yahoogroups.com" + "body";
try {
// Verify message against public key
new DKIMVerifier(publicKeyRecordRetriever).verify(new ByteArrayInputStream(signedMIME.getBytes()));
} catch (Exception e) {
throw e;
}
Have a look if this helps. I am also starting with jDKIM. I just wrote a method to list the signature records for the input message. Puzzled how to parse the signature records.
import java.io.IOException;
import java.io.InputStream;
import java.util.List;
import org.apache.james.jdkim.DKIMVerifier;
import org.apache.james.jdkim.api.SignatureRecord;
import org.apache.james.jdkim.exceptions.FailException;
import org.apache.mailet.Mail;
import org.apache.mailet.Mailet;
public class AlgorithmDkimVerification {
public List<SignatureRecord> verifyDkim(InputStream messageStream)
{
DKIMVerifier verifier = new DKIMVerifier();
List<SignatureRecord> records = null;
try {
records = verifier.verify(messageStream);
} catch (IOException | FailException e) {
e.printStackTrace();
}
System.out.println(records);
return records;
}
}
Once the websocket session has been established, is it possible to pass data other than using session.getAsyncRemote().sendText(**)) across the session?
I am using
javax.websocket.* as ClientEndpoint
and spring-websocket for server.
I have tried using:
session.getRequestParameterMap().putAll(bMap); //This gives me exception
session.getPathParameters().putAll(aMap); //This gives me exception
session.getUserProperties().put("dataE", "dataF");; //This goes through fine
And I am not able to see the data on server side:
Map<String, Object> aMap = stStandard.getNativeSession().getUserProperties();
This map comes as blank on server side.Please let me know if more information is needed. Here is my code:
import java.io.IOException;
import java.util.ArrayList;
import java.util.HashMap;
import java.util.List;
import java.util.Map;
import javax.websocket.ClientEndpoint;
import javax.websocket.OnClose;
import javax.websocket.OnMessage;
import javax.websocket.OnOpen;
import javax.websocket.Session;
#ClientEndpoint
public class TestSpring {
#OnMessage
public void onMessage(String message, Session session) throws IOException,
InterruptedException {
Map aMap = new HashMap();
aMap.put("dataA", "dataB");
try {
session.getPathParameters().putAll(aMap);
} catch (Exception e) {
e.printStackTrace();
}
Map bMap = new HashMap();
List<String> bList = new ArrayList<String>();
bList.add("dataC");
try {
bMap.put("dataD", bList);
session.getRequestParameterMap().putAll(bMap);
} catch (Exception e) {
e.printStackTrace();
}
try {
session.getUserProperties().put("dataE", "dataF");
} catch (Exception e) {
e.printStackTrace();
}
session.getAsyncRemote().sendText("Data to be sent");
}
#OnOpen
public void onOpen() {
System.out.println("Client connected");
}
#OnClose
public void onClose() {
System.out.println("Connection closed");
}
}
import java.util.List;
import java.util.Map;
import org.springframework.web.socket.BinaryMessage;
import org.springframework.web.socket.TextMessage;
import org.springframework.web.socket.WebSocketExtension;
import org.springframework.web.socket.WebSocketSession;
import org.springframework.web.socket.adapter.standard.StandardWebSocketSession;
import org.springframework.web.socket.handler.AbstractWebSocketHandler;
import com.sap.websocket.inMemoryWebSockets.KeepInMemoryWebSockets;
public class WebsocketEndPoint extends AbstractWebSocketHandler {
#Override
protected void handleTextMessage(WebSocketSession session,
TextMessage message) throws Exception {
session.getHandshakeHeaders();
StandardWebSocketSession stStandard = (StandardWebSocketSession) session;
try {
Map<String, Object> aMap = stStandard.getNativeSession()
.getUserProperties();
} catch (Exception e) {
e.printStackTrace();
}
TextMessage binaryMessage = new TextMessage(new String(
"Hello Client. This is message sent from server"));
session.sendMessage(binaryMessage);
}
#Override
protected void handleBinaryMessage(WebSocketSession session,
BinaryMessage message) throws Exception {
}
}
I have the following code to my website that I want to login to as shown below. There are no error messages when I run it, yet the HTML it returns at the end is of the original login page, suggesting that I have not logged in. I have tried it with multiple websites and the same thing keeps happening, so I am not sure whether it is not logging me in when I want it to, or whether it has something to do with cookies not being remembered when I go to the other page.
Any help would be greatly appreciated.
import java.util.logging.Level;
import org.jsoup.*;
import org.jsoup.nodes.Document;
import com.gargoylesoftware.htmlunit.WebClient;
import com.gargoylesoftware.htmlunit.html.HtmlElement;
import com.gargoylesoftware.htmlunit.html.HtmlOption;
import com.gargoylesoftware.htmlunit.html.HtmlPage;
import com.gargoylesoftware.htmlunit.html.HtmlSelect;
public class testing {
public static void main(String[] args) throws Exception {
java.util.logging.Logger.getLogger("com.gargoylesoftware").setLevel(Level.OFF);
final WebClient webClient = new WebClient();
final HtmlPage page = webClient.getPage("www.websiteneededtologinto.com");
HtmlElement usrname = page.getElementByName("Email");
usrname.click();
usrname.type("MYUSERNAME");
HtmlElement psswrd = page.getElementByName("Passwd");
psswrd.click();
psswrd.type("MYPASSWORD");
HtmlElement signin = page.getElementByName("signIn");
signin.click();
String html = Jsoup.connect("pagerequiringlogintoaccess.com").get().html();
System.out.println(html);
}
}
I have run into this issue before, and to fix it I created methods to run all the functions:setText, getEid, and click. They are:
public void click(HtmlElement el) throws Exception{
page = el.click();
}
public void setText(HtmlElement el, String text){
HtmlTextInput input = (HtmlTextInput)el;
input.setText(text);
}
public HtmlElement getEid(String id){
HtmlElement rval = null;
boolean looking = true;
int counter = 0;
while(counter < elementRetryCount && looking){
try{
rval = page.getHtmlElementById(id);
looking = false;
} catch (Exception e){
try {
Thread.sleep(elementRetryTimer * 1000);
} catch (InterruptedException e1) {
e1.printStackTrace();
}
}
counter ++;
}
return rval;
}
To run them the code would be something like:
scraper.setText(scraper.getEid("txtFromDate"), startingDate);
scraper.click(scraper.getEid("btnSubmit"));
Hope these changes help