for my master thesis I need to keep track of google searches that the users perform. It should be a web project. At first I want to setup a server (that acts like a proxy) to monitor all actions (search queries) performed by the user.
The server should deliver the google search page. I need to keep track of the input the users make and the corresponding results returned from google as well.
My questen is now.. How should I start?
I am not really sure which webserver to use. Should I use tomcat / jetty / or something else?
What about Java Server Faces or Servlets? I worked with JSF long time ago but I am not sure whether its a good decision to use it.
The server should deliver the google search page. Here is my idea: The user connects to my server. The server "reads" the google page and returns it (the source code) to the user. I think of using a listener on the default search field to monitor the search queries of the users. But how is it possible to monitor the results returned from google? The google site itself uses JavaScript I guess. So when the user makes some input it is directly sent to google and the results are directly shown on the webpage thus my webserver does not see anything from the connection between the client and the google services.
The main idea is to monitor a search query and the corresponding search results.
I just need some help and ideas to get started. This is just a small part of my thesis. I do not want to start from zero during my editing time thus I want to make sure that I make the right choices before I start.
Thank you in advance... best regards.
You could use netty as a proxy and just filter and log all search queries. But then again why even use java? From what you describe you wan't to play man-in-the-middle and manipulate/log http traffic.
A quick google search came up with this for python: https://code.google.com/p/proxpy/
Edit: and similar in java https://github.com/adamfisk/LittleProxy
Related
I am trying to get the country where the user is sending the request to my server.
So far I have found those solutions:
https://github.com/fiorix/freegeoip
https://github.com/maxmind/GeoIP2-java
These solutions are using MaxMind database GeoIP2 Country Database which has a license and for this specific project I have restriction with Libraries (legal aspects because of the business)
I have found some topics mentioning that Google API can provide this service, but that doesn't interest me as I want a solution that I can host myself and it is implemented using Java or Kotlin.
Does someone know a solution different from the one mentioned (that runs on the backend)?
I've coded a little service that returns the country based on data I've acquired from http://ipverse.net/
It should be license free but correct me if I am wrong. According to the website the data is provided by the five big RIRs.
The website updates its data ~5am. The Cron job that updates the IP Blocks is scheduled at 6am. The data should be accurate I guess.
*edit
I've replicated the algo and created a github action to keep it updated, check it out: https://github.com/jaecktec/ncc-inetnum-parser
Not sure if, from an architecture point of view, this is a good solution, but you could install a command line tool in your backend server and use that.
For example: http://manpages.ubuntu.com/manpages/zesty/en/man1/geoiplookup.1.html
There are also some free web API you could use, have a look here
I've made a simple mailing app that takes in email credentials and uses it to send emails of certain kinds to selected addresses. Problem is, I've had to input the credentials right into the code, so anyone who uses dex2jar can get the source code and get the email used for forwarding and easily make the app obsolete.
I imagine I'm not the only one facing this issue, so what are some ways to make my code secure?
No matter how good of a technique you use to hide the credentials, if it's in the code then it can always be found.
Instead of hard coding them in, you could perhaps let the user specify them when he starts the app? If that can't be avoided you could instead have a remote service that will do the sending and forward your request to that.
You can not both connect to an e-mail account and keep those same users out of said e-mail account. Consider using a hosted server as part of the project to securely connect to the e-mail account from the server level and process these e-mails remotely.
I am writing a simple turn game in Android for two players. I already developed an offline version (when two persons exchange the phone) and now want to create an on-line one.
They way I wanted to do it was to use Tomcat or my own PHP server for my website. My question would be whether it is possible and how to store data?
I think that the only thing I need to send to the server is the current score and an array representing the board. Then I'll be able to retrieve these pieces of information on client's side. Am I thinking properly?
I gotta say, I am completely new to all this stuff. I talked to my friend who programs in Java and he told me that I should use Tomcat. But how? It is a local server so the only way it could work would be when two mobile phones are connected to the same network or what?
The simplest way would be to implement a set of RESTful services on an online server.
You can use Tomcat if you want, but the cheapest approach would be to have a simple web hosting with i.e. PHP.
Create REST API's like /get/board, /get/move, /update/move, /authenticated and so on.
The simplest way to start out is probably not even with any real service, but just a couple of PHP scripts and an MySQL database.
You'll need some server, per example just a simple FTP server where you can put some on-the-internet reachable scripts on, and host the (MySQL) database.
Then per example for keeping the score you'll just need 2 scripts : 1 that saves and 1 that retrieves the score of a player.
Saving the score will just demand you to call on a link as per example:
www.server.com/saveScore.php?playerId=111&score=60
In the PHP you could just take this out with $_GET['playerId'] and $_GET['score'] and save those values in to your database with a query.
Returning the score to your app could work out with calling a link as:
www.server.com/retrieveScore.php?playerId=111
Which will simply query your database for the info on that player, and then print it as XML or JSON.
Then your app can just parse the response (xml/json) again.
A good idea would be to move this from $_GET to $_POST (POST is not visible in the link, but you'll need a bit more complicated code to transmit the data through that) once it works.
Once you've got that going you can work on actually building a service, REST would be a good choice, in order to increase maintainability, extendability and security.
I'm working with Apache Tomcat 7 (JSP and Servlets). In my application, I need to send some messages from server to client. Bellow, I'll explain a little bit what I'm working on.
Brief explanation: The application will bring up a login page if the user isn't logged in every time when he wants to connect to internet. After the user logged in successfully and his time is going to end, I will need to send to client a message with remained time (for example in last few minutes). It can also be another requirement to open advertising popup at a specific time.
I now about JMS but I don't know how fit is that for my scenario. I also read in other posts, the WebSocket can be also an option.
I'm running the server on CentOS 6.2.
Question: For this scenario, do you have some thoughts on how to treat it with Java technologies? If you have some other ideas, feel free to expose!
N.B. Related to JavaScript and PHP I found good answers on SO's questions. I'm interested on how to solve this issue with Java technologies especially.
http://jwebsocket.org/
Maybe this fits your needs.
You will not be able to initiate an HTTP connection from the server to the client. One solution will be to use WebSocket/Comet Framework. Unfortunately websockets are not really wide spread (server+browser) for now. I will suggest you to use a framework to fill the "gap": https://github.com/Atmosphere/atmosphere
I don't understand your obsession with us implementing the solution in Java - any valid solution should be portable across different serverside languages. However if the termination is to occur without synchronous user-driven interaction, then you're just creating load on your server by trying to handle it here. If you want somebody to write the code for you then this isn't the right forum.
I now about JMS....CentOS 6.2.
Not much help here.
The thing we really need to know is what you mean by:
After the user logged in successfully and his time is going to end
(I assume you mean the session time is going to end, unless you've written some software which predicts when people will die).
How do you determine when the session will be ended?
Is it a time limit per page?
Is it a fixed time from when they login?
Is it when the session is garbage collected by the Java?
In the case of 1, then the easiest way to achieve this would be to use javascript to set a timeout on the page (when the user navigates to a new screen the timeout will be discarded), e.g.
setTimeout(function() {
alert('5 minutes has expired since you got here - about to be logged out');
}, (300000)); // 5 minutes
In the 2nd case then you'd still use the method above, but reduce the timeout on the javascript by the time already spent on the server (write the javascript using java, or drop a cookie containing the javascript timestamp at login).
In the 3rd case.....you don't really have any way of knowing when the user will be logged out.
I'm working on a browser game with the play framework, and I definitely need longpolling, but I don't quite understand how to use it. WebSockets would be perfect for this, but it's not supported by that many browsers yet.
Here's what I want to do: When the user logs in, and navigates to the play game controller, I want to start a connection, and keep it open. I want to do this for all users that are online, so I can show a list of them on the site, so they can play with each other. I've looked at the documentation, but I don't understand how I could implement it in my case. Because there simply isn't anything I want to calculate (in the example they're generating a pdf) I just want the connection to stay open.
What I'm also wondering is, how I should keep track of all these open connections? Right now, I just have an online column in my users table in the database, which I update. SO everytime someone connects I have to update the database. Are there better ways to do this, or is this fine?
And lastly, assuming all of the above works. When player A, selects player B to play with: how do I notify player B of this? Do I just send some JSON code, and change the page with javascript, on player B's side, or do I send him to a totally different page? I'm not sure how to communicate when the two connections are established and the game has started.
Firstly, I think you need to appreciate the difference between Websockets and Long Polling.
Websockets creates a connection and keeps it open until the browser terminates the session, via some javascript or the user moving on from the page. This would give you the desired nature of what you are requesting. Looking at the Chat example in the Play download will show you how an entire Chat application is handled using Websockets.
Further to Pere's answer regarding Play's statelessness. The Play creators have suggested that a single Websocket connection, regardless of how long it is open for and how many requests are sent back and forther, is considered to be a single transaction. Therefore, saving to the database in between each Websocket request is not needed (again, you can see that nothing is saved in the Chat example). Using this method, you would be expected to save the details when the Websocket is finally closed, or indeed all Websockets, depending on your use-case.
Long Polling on the other hand opens a connection to the server, and the server simply waits until there is something to send back to the client. If you need to push any data to the server, you would do this as a separate AJAX request, so you would effectively have two requests open at once. You don't necessarily know when a user logs off, unless you send a request just as they leave the page, to let the server know they have gone, but this is not always successful. Long Polling can work, but it is not as neat a solution as Websockets, but as you say, this is not widely supported yet.
My suggestion would be to study the Chat example (as it has a Long Polling and Websockets version). This will be the most effective way to get up and running with your requirements.
As for your final query regarding how to notify the other player. In Long Polling, you would simply respond to the suspended request with some JSON. With websockets, you would send an event back to the client. Again, both approaches can be pretty clearly figured out from the Chat example.
I have also written a Blog post on Websockets, which may help you understand this process a little better.
On the Websocket part, as you can see here (1st answer) the support is not so bad, and you have a Javascript fallback if there is some problem with the browser. This would simplify your scenario, as long polling may be more complicated to manage.
On the issue of keeping track, as Play is stateless you have to store the flag in the database and remove it when they close the connection. Otherwise you are breaking the statelessness.
About the notification, you have to send some message to B, but don't move them to another page as it may be confusing and cause bad user experience. Use Json to pop some message (in a div) alerting them of the game starting or the request to play.
I'm not using the "play" framework.
But I've been lately researching and tinkering with http-based long polling. Websockets, if available, is much more appropriate for realtime messages!
As for long-polling, I found that using a "cargo truck" analogy helped me reason about long-polling quite effectively. Here's a little note I wrote on the subject:
http://dvb.omino.com/blog/2011/http-comet-realtime-messages/
Perhaps you or future greppers may find it useful.
You might also want to take a look at the Juggernaut project which is based on node.js and Redis and gives you a "realtime connection between your servers and your client browsers". When using a Java Redis client like Jedis, you should easily be able to integrate the whole thing with the Play framework!