Closed. This question needs to be more focused. It is not currently accepting answers.
Want to improve this question? Update the question so it focuses on one problem only by editing this post.
Closed 7 years ago.
Improve this question
I'm not sure if the title says what I needed. But that's the best way I can put it.
My requirement: I want to check if a series of engine numbers (of vehicles) are registered with the local transportation authority. They have provide a web-interface for this task, but I can only check for a single engine number at once. Usually I need to check over 200 numbers. So, its a humongous task to check each one individually.
A couple of years ago, I have created a small standalone python script to do this automatically using web scraping, but now I want to do this on a server.
A user gives all the numbers in the text file which they upload/paste the contents in to a text field. Then I'll have to submit the form on the transportation website (using web scraping) for each number and display a final status for all of them.
What I want to know is how to do this on a server? What technologies could be helpful. I'm comfortable with Java & JavaScript. I don't know PHP (But i can learn if needed). I don't have slightest Idea how to do this on server side. Any Ideas and Help is greatly appreciated.
Thanks.
Java has a library called JSoup, which provides a mostly-familiar api that uses css selectors.
And obviously there are built-in functions that can get you the html from a given URL.
Put those together and you've got a server-side scraper
[edit]
Your question, on a re-read, isn't just about scraping -- it's about how to automatically submit an html form from within Java to an external server. This is an interesting question, one I've wondered myself.
this may be an answer: How to send post form with java?
Related
Closed. This question needs to be more focused. It is not currently accepting answers.
Want to improve this question? Update the question so it focuses on one problem only by editing this post.
Closed 5 years ago.
Improve this question
I'm creating a simple java application made in swing that communicates with a database. Multiple people will be using the application at the same time. As they are able to change things at the same time, if someone for example, adds a new user, and a different person is on that same page, I want the person who did not make the change to be notified that changes were made to the database since they last loaded it.
My lecturer in college advised me that WebSockets would be the way to go to achieve this, however after some reading about WebSockets in Java, it seems it is based to work with web browsers instead of between Java applications.
Can using WebSockets achieve what I am trying? Or, if not, what would be a way to achieve this?
Simple answer is Yes you can achieve what you needed
WebSocket is a communication protocol(#see RFC 6455) & it is not a must to use a Web browser.
You can achieve what you want to do with your app, it is just a matter of writing a custom WebSocket server to facilitate your requirements in your case sending database changes to the other clients(Which is called Server push)
There are several java libraries to get the work done,
netty WebSocket (My favorite)
jWebsocket
Atmosphere
Webbit
Netty WebSocket is a good one to start with and you can find examples in its project to write a custom client and a server
Closed. This question needs details or clarity. It is not currently accepting answers.
Want to improve this question? Add details and clarify the problem by editing this post.
Closed 5 years ago.
Improve this question
I am starting to build my first website with Angular&Bootstrap. I am using the Spring Tool Suite. So far my website looks pretty boring but learning CSS takes me a while.
On my website I have an input (style=file) and a button to submit. So far I managed to press a button to upload a file. I can see is as a String in the console when I log it, its working.
Now to the point where I need help:
I got a java-program that I have to use which gets the path to an XML-File and parses it. It checks if it has the right special format and reads it in. This is only a part of the whole program.
How can I use this in my Code to upload an XML-File and then check it and read it with this program? How do I use the Java-Code? So far I am only using HTML, CSS and Javascript.
Thank you for your help!
There are two options precisely available for you:
1) convert the logic which is in Java to Java script
Or
2) make it a web application and run it on tomcat(or a similar web server)
There are others too but there are a bit heavy to understand and implement.
Closed. This question needs to be more focused. It is not currently accepting answers.
Want to improve this question? Update the question so it focuses on one problem only by editing this post.
Closed 7 years ago.
Improve this question
I was given the task - to create a simple JAVA EE web-application that have the following functionality:
It had an index page
I choose an algorithm and enter parameters for it on index page
Depending on chosen algorithm a statistical .csv file will be processed on its own way.
After processing the .csv file in compliance with entered parameters and chosen algorithm I get a table where I can find processed data.
I know that it's a very simple application but I don't know what to do - architecture, what kind of web layer and etc
If you install Apache Tomcat on your computer, and then run it, you will find a lot of simple examples, many of which will come close to what your are asking. Plus, once you install it, you have a server on which to run your new code. I would recommend you start there.
Apache Tomcat
I would say that if you really want simplicity, you go with just Servlets for the server-side processing, and JSP pages for the HTML views. I know, it's "vintage" almost, but will get you running in no time and without the need for any special "architecture" or framework besides the servlets API.
You can build your app as a War file with the tool of your choice and deploy it in a Tomcat server and that's it.
Closed. This question needs to be more focused. It is not currently accepting answers.
Want to improve this question? Update the question so it focuses on one problem only by editing this post.
Closed 7 years ago.
Improve this question
I am not a Java pro. I need to enable my completed java applications to ONLINE-Versions. Kindly give me the simplest way.
I have tried to go the JSP and Servlet way, but seems exceedingly tiresome,there surely ought to be a direct way.
When you use a typical web application what do you think is happening? You're in a browser and it's displaying some pages. You click around and some server does some work, for example Amazon shows you a list of books you could buy, the server is passing data to the browser for it to display.
That's a big difference from your current Java Application where everything is happening in the same computer.
This split between the UI part of the application, in the Browser, and the server part that's doing the real business is one reason why Web programming is so different from simple Java programming.
Servlets and JSPs are the traditional Java way of doing things, but modern web apps make much greater use of JavaScript for all the UI and all the Java (or any other server technology) does is provide the data displayed by JavaScript.
Bottom line: Sorry, but there is a whole lot more to learn. My recommendation would be to grit your teeth and forget about what you've done so far and start studying JavaScript and HTML.
Closed. This question needs to be more focused. It is not currently accepting answers.
Want to improve this question? Update the question so it focuses on one problem only by editing this post.
Closed 8 years ago.
Improve this question
I was wondering if there is a way to pull specific data from a website using java (eclipse). For example, stock information from Yahoo Finances or from Bloomberg. I've looked around and have found some resources, but I haven't been able to get them to work, perhaps I'm missing something or they're outdated. If possible, I also want to avoid downloading any external resources, I've read up on JSoup and will consider it more seriously if all else fails.
Thanks for the help.
The answer is: yes there are many different ways to pull data from websites.
There are essentially 2 alternatives no matter the programming language (Java, .NET, Perl...):
the website has an API: in this case it will be a REST or SOAP API or perhaps a custom one (REST and SOAP probably account for the vast majority). Check out that website's API documentation if any. Also check out Programmable Web for references.
the website doesn't have an API. You then need to do what you call here as screen-scraping. Essentially you will send a series of HTTP GET or HTTP POST requests as your browser would. The server replies with a response which contains HTML code. From there on, you need to "parse" the HTML to extract the information you need. This will require heavy duty XPath (if the content is XML) or regular expressions (if the content is HTML or text).
Look at Apache HTTP Components to get you started.
If all you want is Finance information, Google has a JSON/REST API for that and there's a question on SO that will help you: How can I get stock quotes using Google Finance API?.
Yahoo also has one and there is also already an question on it in SO: Yahoo Finance All Currencies quote API Documentation