When debugging I can confirm 6 out of 9 url connections work. One is dead, and others are sending a exceptions. Question 1:I want it to show either alive or dead, nothing else. Question 2 : when running the test, it tests too fast and it shows all are "dead" links when running (even though I know 6 work). Any suggestions?
try{
//Read File
FileReader file = new FileReader("c:\\blocked\\domains1.txt");
BufferedReader buff = new BufferedReader(file);
BufferedReader rd;
OutputStreamWriter wr;
while (true)
{
String line = buff.readLine();
if (line == null)
break;
else
try
{// Test URL Connection
URL url = new URL("http://www."+line);
URLConnection conn = url.openConnection();
conn.setDoOutput(true);
wr = new OutputStreamWriter(conn.getOutputStream());
wr.flush();
rd = new BufferedReader(new InputStreamReader(conn.getInputStream()));
if( rd.ready())
{
//write to output file
System.out.println("Good URL: " + line);
urlAlive ++;
//Write to file
Path filePath = Paths.get("c:\\RevisedUrls\\revised.txt");
try
{
BufferedWriter myWriter = new BufferedWriter(new FileWriter(filePath.toFile()));
// write the user supplied data to the file plus a line separator.
myWriter.write(line + System.getProperty("line.separator"));
myWriter.close(); // immediately close the file when finished with it.
}
catch(Exception e)
{
System.out.println(e);
}
}
else// increment dead url
urlDead++;
}
catch (Exception e)
{
System.out.println(e.toString());
}
}
buff.close();
}
finally
{
int totalSites = urlAlive + urlDead;
JOptionPane.showMessageDialog(null,"Total Number of Alive Sites: " + urlAlive);
JOptionPane.showMessageDialog(null,"Total Number of Dead Sites: " + urlDead);
JOptionPane.showMessageDialog(null,"Total Number of Sites: " + totalSites);
}
}
}
Related
Hey I am having a file nearly 110MB size at apache. I am reading that file into input stream and then converting that input stream to List of String based on all suggestion i find on stack overflow. But still i am facing out of memory issue.
Below is my code.
private List<String> readFromHttp(String url, PlainDiff diff) throws Exception {
HttpUrlConnection con = new HttpUrlConnection();
con.setGetUrl(url);
List<String> lines = new ArrayList<String>();
final String PREFIX = "stream2file";
final String SUFFIX = ".tmp";
final File tempFile = File.createTempFile(PREFIX, SUFFIX);
tempFile.deleteOnExit();
StringBuilder sb = new StringBuilder();
try {
InputStream data = con.sendGetInputStream();
if(data==null)
throw new UserAuthException("diff is not available at the location");
else {
try (FileOutputStream out = new FileOutputStream(tempFile)) {
IOUtils.copy(data, out);
LineIterator it = FileUtils.lineIterator(tempFile, "UTF-8");
try {
while (it.hasNext()) {
String line = it.nextLine();
lines.add(line);
sb.append(line);
}
} finally {
LineIterator.closeQuietly(it);
}
}
data.close();
diff.setLineAsString(sb.toString());
}
} catch (IOException e) {
// TODO Auto-generated catch block
e.printStackTrace();
}
//System.out.println(lines);
return lines;
}
public InputStream sendGetInputStream() throws IOException {
String encoding = Base64.getEncoder().encodeToString(("abc:$xyz$").getBytes("UTF-8"));
URL obj = new URL(getGetUrl());
// Setup the connection
HttpURLConnection con = (HttpURLConnection) obj.openConnection();
// Set the parameters from the headers
con.setRequestMethod("GET");
con.setDoOutput(true);
con.setRequestProperty ("Authorization", "Basic " + encoding);
InputStream is;
int responseCode = con.getResponseCode();
logger.info("GET Response Code :: " + responseCode);
if (responseCode == HttpURLConnection.HTTP_OK) {
is = con.getInputStream();
}
else {
is = null;
}
return is;
}
Is something in memory i am doing that is consuming lot of heap? Is there a better way to do it?
Your code has multiple issues. I am not going to solve each and every issue but point that out so that you can review your code and learn to write better code.
In method readFromHttp(..):
There is no need to create a new file by IOUtils.copy(data, out);
No use of String Builder StringBuilder sb = new StringBuilder();
No use of line iterator LineIterator
And there are multiple other memory-related issues but for the time being correct these points and test with the below-mentioned code.
Change your reading lines from file to very simple way after correcting the above mistakes:
try(BufferedReader reader = new BufferedReader(new InputStreamReader(data, StandardCharsets.UTF_8))) {
for (String line; (line = reader.readLine()) != null;) {
lines.add(line);
}
}
This code loads my html but it will not load my jpgs in the html, when I run the code it seems like it is working and all the bytes are being sent for each image but they do not show up in my browser. Im supposed to use the DataOutputStream to send the bytes but maybe thats the problem? im really lost and brand new to network programming.
public void run(){
System.out.println("CLIENT THREAD STARTING");
String req = "";
if(clientIn.hasNextLine()){
req = clientIn.nextLine();
System.out.println("Header: " + request);
}
PrintWriter toClient = null;
try {
toClient = new PrintWriter(sock.getOutputStream());
} catch (IOException e2) {
e2.printStackTrace();
}
//only sends files if client requests them
if(request.contains("GET")){
req = req.substring(request.indexOf("/") + 1);
String name = req.substring(0, req.indexOf(" "));
String type = "";
if(name.contains(".jpg")){
type = "text/html";
}
else if(name.contains(".html")){
type = "image/jpeg";
}
File theFile = new File(fileName);
FileInputStream fileIn = null;
try{
fileIn = new FileInputStream(fileName);
}catch(FileNotFoundException e){
toClientText.println("404");
toClientText.flush();
System.exit(-1);
}
System.out.println("File name " + name);
toClientText.println("HTTP/1.1 200 OK");
toClientText.println("Connection: closed");
toClientText.println("Content-Length: " + theFile.length());
toClientText.println("Content-Type: " + type);
toClientText.println("\n");
toClientText.flush();
DataOutputStream dataStream = null;
try {
dataStream = new DataOutputStream(sock.getOutputStream());
} catch (IOException e1) {
e1.printStackTrace();
}
byte[] send = new byte[1024] ;
try {
while ((fin.read(send)) != -1 ) {
toClientData.write(send);
toClientData.flush();
}
toClientData.close();
fin.close();
} catch (IOException e) {
e.printStackTrace();
}
}
else{
toClientText.println("Bad Request 400");
toClientText.flush();
}
}
toClientText.println("Content-Type: " + fileType);
toClientText.println("\n");
This will create 3 line breaks between your header and the file data, so your browser will think the third line break is part of the image, which will break the image.
Also, even though all browsers can handle \n linebreaks, be aware that the standard demands \r\n linebreaks, not \n.
This should work:
toClientText.println("Content-Type: " + fileType);
toClientText.println();
I am trying to make a web proxy for HTTP communication between server and client. The GET method is working fine but I am POST method part is not working. I am sure I have missed out something. I want to know what have I missed or not implemented.
// request from client is handle from here
while ((inputLine = in.readLine()) != null) {
try {
StringTokenizer tok = new StringTokenizer(inputLine);
tok.nextToken();
} catch (Exception e) {
break;
}
if (cnt == 0) {
System.out.println("inputLine "+inputLine);
String[] tokens = inputLine.split(" ");
urlToCall = tokens[1];
//hum inputline sy URL nikaal rahay hai
if(tokens[0]=="POST")
{
f=1;
}
System.out.println("Request for : " + urlToCall);
}
cnt++;
}
BufferedReader rd = null;
try {
//yaha sy hum ab server ko request send karay gy
URL url = new URL(urlToCall);
URLConnection conn = url.openConnection();
HttpURLConnection huc = (HttpURLConnection) conn;
conn.setDoInput(true);
conn.setDoOutput(false);
// now we will get the response from the server
if (f == 1) {
huc.setDoOutput(true);
huc.setInstanceFollowRedirects(false);
huc.setRequestMethod("POST");
huc.setRequestProperty("Content-Type", "application/x-www-form-urlencoded");
huc.setRequestProperty("charset", "utf-8");
}
InputStream is = null;
if (conn.getContentLength() > 0)
{
try {
is = conn.getInputStream();
rd = new BufferedReader(new InputStreamReader(is));
} catch (IOException ioe) {
System.out.println(
"********* IO EXCEPTION **********: " + ioe);
}
}
What is the error you are getting on the post? And what is a sample GET request that is being passed into your code?
When I use a simple GET request, the code fails because the urlToCall does not have the host or protocol. The below code worked for me, but I would highly suggest you change your code to not hide the exceptions that are being thrown because they will have important information about what is going wrong with your code.
if (cnt == 1) {
System.out.println("host: " + inputLine);
String[] tokens = inputLine.split(" ");
urlToCall = "HTTP://" + tokens[1] + urlToCall;
}
I'm struggling against the uncompleted download of a file.
For example, I upload some data on github :https://gist.githubusercontent.com/rdanniau/3b7f26bb1101b28400bf24f2f9664828/raw/980d6ff511404bf14d3efc56be3dfb081541991f/LEDirium.hex
and on pasteBin : http://pastebin.com/raw/FcVfLf5b
I want to retrieve them and save them into a file "filename".
I've watch a lot of example on internet and it must be working.
Here is the code :
private void download(final URL myUrl){
new Thread(new Runnable() {
//InputStream is = null;
//FileOutputStream fos = null;
public void run() {
try {
URLConnection connection = myURLL.openConnection();
//HttpURLConnection connection = (HttpURLConnection) myUrl.openConnection();
//connection.setRequestMethod("GET");
connection.setReadTimeout(5000);
connection.setConnectTimeout(10000);
connection.connect();
//is = myUrl.openStream();
is = connection.getInputStream();
File file = new File(context.getFilesDir(),fileName);
file.delete();
file = new File(context.getFilesDir(),fileName);
fos = new FileOutputStream(file);
byte data[] = new byte[1024];
String str ="";
int count = 0;
while ((count = is.read(data)) != -1) {
fos.write(data, 0, count);
}
is.close();
fos.close();
}
catch (Exception e) {
downloadedFileCallback.onError(e);
Log.e("DownloadedFile", "Unable to download : " + e.getMessage() + " cause :" + e.getCause());
return;
}
downloadedFileCallback.onDownloadedFinished();
readFile(context);
}
}).start();
}
public void readFile(Context context){
// read
try {
BufferedReader br = new BufferedReader(new FileReader(new File(context.getFilesDir(),fileName)));
String line;
while ((line = br.readLine()) != null) {
Log.v("DL", line);
}
br.close();
}
catch (Exception e) {
Log.e("DownloadedFile", "Unable to read : " + e.getMessage() + " cause :" + e.getCause());
}
//Log.v("DownloadedFile", text.toString());
}
where myURL are called like
URL myURL = new URL("http://pastebin.com/raw/FcVfLf5b");
In the Log.v, I can see that I downloaded only a part of the file which is never the same (it could be the entire file, the half, the quarter, we don' know...)
It's probably the inputStream connection which is closed too fast. But why ?
Last question, instead of using Log.v to check if the file is correctly downloaded. Where can I found it on my phone ? I searched in many folders but I never seen my File.
Thanks a lot for any advice
EDIT : It seems to be the same here InputStream returns -1 before end of file but no one answered.
This is the method I have in my java application. It is reading the bytes correctly, I have logged to see if it was. The problem is that the php is not realizing the data is there. I have tested and the .php reads that $_POST is set, but is empty.
public void screenshot(BufferedImage screenshot) {
try {
ImageIO.write(screenshot, "png",
new File(Environment.getStorageDirectory().toString()
.concat(File.separator + SCRIPT_NAME + ".png")));
HttpURLConnection httpUrlConnection;
OutputStream outputStream;
BufferedInputStream fileInputStream;
BufferedReader serverReader;
int totalBytes;
String response = "";
String serverResponse = "";
String localFileName = Environment.getStorageDirectory().toString()
.concat(File.separator + SCRIPT_NAME + ".png");
// Establish a connection
httpUrlConnection = (HttpURLConnection) new URL(
"http://www.scripted.it/scriptoptions/utils/saveScreenshot.php?user="
+ SupraCrafter.statHandler.getUser())
.openConnection();
httpUrlConnection.setDoOutput(true);
httpUrlConnection.setDoInput(true);
httpUrlConnection.setRequestMethod("POST");
httpUrlConnection.setRequestProperty("Content-type",
"application/x-www-form-urlencoded");
outputStream = httpUrlConnection.getOutputStream();
// Buffered input stream
fileInputStream = new BufferedInputStream(new FileInputStream(
localFileName));
// Get the size of the image
totalBytes = fileInputStream.available();
// Loop through the files data
for (int i = 0; i < totalBytes; i++) {
// Write the data to the output stream
outputStream.write(fileInputStream.read());
}
// Close the output stream
outputStream.close();
// New reader to get server response
serverReader = new BufferedReader(new InputStreamReader(
httpUrlConnection.getInputStream()));
// Read the servers response
serverResponse = "";
while ((response = serverReader.readLine()) != null) {
serverResponse = serverResponse + response;
}
System.out.println(serverResponse);
// Close the buffered reader
serverReader.close();
} catch (Exception e) {
e.printStackTrace();
}
try {
URL url = new URL(
"http://scripted.it/scriptoptions/utils/setScreenshotStatus.php?user="
+ SupraCrafter.statHandler.getUser() + "&pass="
+ SupraCrafter.statHandler.getPass() + "&script="
+ SCRIPT_NAME + "&status=1");
BufferedReader in = new BufferedReader(new InputStreamReader(
url.openStream()));
in.close();
} catch (MalformedURLException e) {
} catch (IOException e) {
}
}
Here is the .php file:
<?
// Config
$uploadBase = "../screenshots/";
$uploadFilename = $_GET['user'] . ".png";
$uploadPath = $uploadBase . $uploadFilename;
// Upload directory
if(!is_dir($uploadBase))
mkdir($uploadBase);
// Grab the data
$incomingData = file_get_contents('php://input');
// Valid data?
if(!$incomingData)
die("No input data");
// Write to disk
$fh = fopen($uploadPath, 'w') or die("Error opening file");
fwrite($fh, $incomingData) or die("Error writing to file");
fclose($fh) or die("Error closing file");
echo "Success";
?>
It always echos 'no input data.'
You are not encoding the content with application/x-www-form-urlencoded. You should not simply copy the bytes into the HTTP payload, but instead encode it correctly.
application/x-www-form-urlencoded is not the only possible way of encoding it, multipart/form-data is another common choice. Both are supported by almost all webservers, and as a consequence by PHP.
A tutorial on how to encode using Java is here : http://www.devx.com/Java/Article/17679
Why don't you use Apache's HttpClient or similar library that already do that tedious work for you?
Apache HttpClient : http://hc.apache.org/httpcomponents-client-ga/