Closed. This question needs to be more focused. It is not currently accepting answers.
Want to improve this question? Update the question so it focuses on one problem only by editing this post.
Closed 8 years ago.
Improve this question
I was wondering if there is a way to pull specific data from a website using java (eclipse). For example, stock information from Yahoo Finances or from Bloomberg. I've looked around and have found some resources, but I haven't been able to get them to work, perhaps I'm missing something or they're outdated. If possible, I also want to avoid downloading any external resources, I've read up on JSoup and will consider it more seriously if all else fails.
Thanks for the help.
The answer is: yes there are many different ways to pull data from websites.
There are essentially 2 alternatives no matter the programming language (Java, .NET, Perl...):
the website has an API: in this case it will be a REST or SOAP API or perhaps a custom one (REST and SOAP probably account for the vast majority). Check out that website's API documentation if any. Also check out Programmable Web for references.
the website doesn't have an API. You then need to do what you call here as screen-scraping. Essentially you will send a series of HTTP GET or HTTP POST requests as your browser would. The server replies with a response which contains HTML code. From there on, you need to "parse" the HTML to extract the information you need. This will require heavy duty XPath (if the content is XML) or regular expressions (if the content is HTML or text).
Look at Apache HTTP Components to get you started.
If all you want is Finance information, Google has a JSON/REST API for that and there's a question on SO that will help you: How can I get stock quotes using Google Finance API?.
Yahoo also has one and there is also already an question on it in SO: Yahoo Finance All Currencies quote API Documentation
Related
Closed. This question needs to be more focused. It is not currently accepting answers.
Want to improve this question? Update the question so it focuses on one problem only by editing this post.
Closed 7 years ago.
Improve this question
I'm not sure if the title says what I needed. But that's the best way I can put it.
My requirement: I want to check if a series of engine numbers (of vehicles) are registered with the local transportation authority. They have provide a web-interface for this task, but I can only check for a single engine number at once. Usually I need to check over 200 numbers. So, its a humongous task to check each one individually.
A couple of years ago, I have created a small standalone python script to do this automatically using web scraping, but now I want to do this on a server.
A user gives all the numbers in the text file which they upload/paste the contents in to a text field. Then I'll have to submit the form on the transportation website (using web scraping) for each number and display a final status for all of them.
What I want to know is how to do this on a server? What technologies could be helpful. I'm comfortable with Java & JavaScript. I don't know PHP (But i can learn if needed). I don't have slightest Idea how to do this on server side. Any Ideas and Help is greatly appreciated.
Thanks.
Java has a library called JSoup, which provides a mostly-familiar api that uses css selectors.
And obviously there are built-in functions that can get you the html from a given URL.
Put those together and you've got a server-side scraper
[edit]
Your question, on a re-read, isn't just about scraping -- it's about how to automatically submit an html form from within Java to an external server. This is an interesting question, one I've wondered myself.
this may be an answer: How to send post form with java?
Closed. This question needs details or clarity. It is not currently accepting answers.
Want to improve this question? Add details and clarify the problem by editing this post.
Closed 8 years ago.
Improve this question
I have searched for some hours now, and can't find an answer. I have a homepage coded in php that needs to frequently send and receive data to a java program.
So far I have done it via POST, but I wonder if it's a better way to do it. I have tried with sockets. But only managed to find tutorials for php client connecting to a java server. But I need the reverse.
All modern languages allow you to send and receive data, no matter from which language is sent or received. Many large applications (such as Facebook API), work with simple methods such as POST, GET, PUT, DELETE, etc., that is a REST API, returning the data in a specific format (JSON, XML, RAW, etc.). This is innecessary if your application is a little piece of code, but you could take this idea and make something simple. So, there is nothing wrong if you are using the POST method, but just in case, I can name you other resources that you could take advantage of.
You can use WebClient.
Also, you can comunicate your PHP code with your Java code using sockets.
TCP sockets: for PHP read this, and this guide for Java.
WebSockets: Java WebSockets for Java and Ratchet for PHP.
Of course, you can use cURL too. For Java you can read this question and connect with php. If your application is very large and complex in the PHP side, I guggest you to use Laravel.
Closed. This question is opinion-based. It is not currently accepting answers.
Want to improve this question? Update the question so it can be answered with facts and citations by editing this post.
Closed 8 years ago.
Improve this question
I'am writing an android app that needs some data from the server. I am also writing the server side in Java.
What is the best way to pass data from the server to the android device: with REST or Sockets (like Kryonet)?
In what format: XML/JSON (for REST) or plain Java objects?
Thanks in advance.
"Best" is very subjective, I think a very good way to communicate with a RESTful api is via Square's Retrofit library, which can be found here:
http://square.github.io/retrofit/
There is also Volley from Google,
http://developer.android.com/training/volley/index.html
Agree with nPn, "Best" depends on lot of app and user considerations. That said,
REST is preferred as it is most widely used and you have access to stable and optimized client libraries. Most of the these libraries support all kinds of use-cases and customizations. Web Sockets are well suited for real time or live content. If you have a different use-case , REST is strongly recommended.
With Android, JSON is well supported. There is a core JSON API included with Android that you can use without any client libraries. XML can be helpful if you plan to expose your APIs for public consumption (some platforms eg: JAVA, windows have strong XML legacy).
REST + JSON seems to be most commonly used combination in recent times, and lot of client libraries usually enable this use-case.
Closed. This question is opinion-based. It is not currently accepting answers.
Want to improve this question? Update the question so it can be answered with facts and citations by editing this post.
Closed 8 years ago.
Improve this question
I'd like to create some kind of location search during user input. The webservice should be contacted during typing of the user input, eg a location name.
So it's clear that there might be lot's of queries one after the other.
The response should only contain a couple of data for each found location, like an id, a fullname and geoCoordinates.
Would you create a SOAP or REST service for this purpose, and why?
Compared to XML, JSON is light weight. We have the flexibility of choosing responses in REST API's from normal string, XML to JSON, where as in case of SOAP always it will be SOAP XML. So i would prefer to go with REST.
Coming to differences between REST and SOAP, REST supports only HTTP where as SOAP supports other protocols also apart from HTTP. It always better to go with REST if your protocol is HTTP.
Sounds like this service is going to be used on some mobile devices. Also since it is a relatively straight forward request response I would go for REST here with JSON as the protocol.
JSON is much more lightweight than XML thus saving on the bandwidth required to request and respond. Data usage is still expensive in some countries on mobile.
Its a simple service so it really does not warrant the overhead of creating a WSDL and modelling the XSD in the case of a top down approach. Keep it simple so less things can break.
However without really understanding your requirements these are just hunches.
Closed. This question needs to be more focused. It is not currently accepting answers.
Want to improve this question? Update the question so it focuses on one problem only by editing this post.
Closed 9 years ago.
Improve this question
Java with Google maps api
I'd like to begin a project with Java that needs a map. I've been thinking about either Google Maps or another Open Source map. How would I implement this?
First of all, I would offer you to use google maps api, they are using REST services and they are very good.
Here is a short explantation:
You need to make an HTTP request with a URL with the parameters you wish (such as the location - longtitude, latitude, the address, and more), this code should be made in java. If you did everything correct, you will recieve a response of type (xaml or json, depends on the URL you sent, it is preferable to use JSON).
After you receive the response you need to deserialize it, which seems to be hard, but is very simple, for example JSON has many ways and built in methods to do it.
And then you can do whatever you wish with the objects you received.
So here are some links:
Google API - https://developers.google.com/maps/documentation/geocoding/?hl=en and https://developers.google.com/maps/documentation/staticmaps/?hl=iw&csw=1
JSON Deserialize - http://james.newtonking.com/json/help/index.html?topic=html/LINQtoJSON.htm or a better one (if you want to make many requests) - http://blogs.msdn.com/b/webdev/archive/2012/12/18/paste-json-as-classes-in-asp-net-and-web-tools-2012-2-rc.aspx
How about this?
The Google Geocoding API
The Google Directions API
The Google Distance Matrix API