Hello dear Stack community,
I am looking for solution for my test execution. I am using Java & Selenium & TestNG tests kept on AzureDevOps.
Each execution needs to load input data from file stored in external source - where client have an access and can easily drag-drop files.
By now files are stored on my companys Share Drive. It works all fine when I run tests from local machine, but when using pipeline, I get FileNotFoundException in line where file is loaded.
Does anyone have an idea why does it happen and how can it be overcome?
Of course if there is any other solution, I would love to hear about it.
It works all fine when I run tests from local machine, but when using pipeline, I get FileNotFoundException in line where file is loaded.
If you use Microsoft-hosted agents to access to files, which are stored on company's Share Drive, you should check whether there is correct connection. If your organization is secured with a firewall or proxy server, you need to add certain IP addresses and domain URLs to the allowlist. See: Allowed address lists and network connections for details.
Also you could use self-hosted agents to access to this shared files so it will communicate with company's Share Drive via your internal network.
I am totally new to AWS. I have been given a official task to explore the AWS api for java So that we can launch a Virtual Machine ON EC2. I have gone through the official docs at sdk-for-java but having few doubts:
Would I need to deploy the java code on EC2 ?
Can Java code create and launch the Virtual Server or we can only start it through code and need to create the virtual machine through EC2 dashboard?
Thanks
First off, since you're very unfamiliar with AWS, have a read through their documentation. I know there's a lot there but the following links should directly apply to your situation:
Concepts
Getting started with AWS SDK for Java
Java examples
To answer your question, you want to start a virtual machine. Within AWS, these are referred to as "instances". To launch an instance using the API, you could execute the code that makes the API call locally and as long as you have valid credentials etc, the code should work. Your code does not need to be run inside of EC2.
The following code will run an EC2 instance for you as long as you have valid credentials and the AWS-SDK included in your java project.
RunInstancesRequest runInstancesRequest =
new RunInstancesRequest();
runInstancesRequest.withImageId("ami-4b814f22") // This is a base image, for example a ubuntu or linux instance
.withInstanceType("m1.small") // The size of the instance, these cost more the more powerful they are
.withMinCount(1) // Minimum amount of instances you want to launch
.withMaxCount(1); // Maximum amount of instances you want to launch
after creating a instance in amazon cloud using webservice in java i need to transfer a executable file or war file via program from my local machine to the newly created instance in amazon and i want to execute that excetuable,i tried and found that there is something called createbucket in ec2 api and using that we can upload the file to that and we can transfer that reference using PutObjectRequest i can transfer the reference to a remote computer in amazon do it is possible or if it is wrong please suggest me the correct way to proceed for file transfer from my local machine to the amazon ec2.
The basic suggestion is, you shouldn't transfer the file(s) with CreateBucket, which is actually an S3 API. Use scp may be a better solution.
Amazon S3, which you are trying to use with CreateBucket, is a data storage service mainly for flexible, public (with authentication) file sharing. You can use REST or SOAP APIs to access the data, but cannot really read/write it in EC2 instances as if it's in local harddisk.
To access file system in EC2 instances, that really depends on your operating system (on EC2). If it's running Linux, scp is a mature choice. You can use Java to directly invoke scp, if you are using Linux locally, or pscp if you are using Windows. If the EC2 instance is running Windows, one choice is to host an SSH/SFTP environment with FreeSSHD, and then proceed like Linux. Another option is use Shared Folder and regular file copy.
I want to place a server program written in Java on the cloud. It would accept TCP socket connections from clients (clients are android phones using 3G), do some computations, save stuff to a MySQL database (also on EC2), and send stuff back to the clients over the TCP connections. It may even be necessary to create several instances of the server (i.e. a process group).
Is this easy to do? I think I can make a AMI, but I'm not sure how to upload Java files, compile and run them, and create a MySQL database etc
Any help would be much appreciated.
Take a look at using Amazon Elastic Beanstalk. Beanstalk is Amazon's PaaS offering and it will alleviate a lot of the system administration burden. Here's a quick description from their docs:
AWS Elastic Beanstalk is an even
easier way for you to quickly deploy
and manage applications in the AWS
cloud. You simply upload your
application, and Elastic Beanstalk
automatically handles the deployment
details of capacity provisioning, load
balancing, auto-scaling, and
application health monitoring.
Also, if you're interested in using MySQL then you should look at Amazon RDS. Again, this will alleviate the system administration burden for your database tier. Here's a quick description from their docs:
Amazon Relational Database Service
(Amazon RDS) is a web service that
makes it easy to set up, operate, and
scale a relational database in the
cloud. It provides cost-efficient and
resizable capacity while managing
time-consuming database administration
tasks, freeing you up to focus on your
applications and business.
Is this easy to do? I think I can make
a AMI, ...
I think the answer depends on how comfortable you are with system administration in general. Creating a AMI to run in EC2 is really pretty much the same as creating a physical server or a VM image. You'll need to install an operating system, and then install tools, libraries and programs you need (like mysql, the jdk, ssh, etc).
You can save yourself a little work by using one of Amazon's pre-built AMI's http://aws.amazon.com/amis/. But, ultimately, you'll be responsible for all system administration of the server. If you've never built a server from the ground up, you have a pretty big learning curve ahead of you. It's not insurmountable, but just be warned that the devil is in the details; there's a ton of stuff you'll need to learn ;-)
... but I'm not sure how to upload Java
files, compile and run them, ...
Once the server is setup and running in EC2, compiling them and running java files is just the same as compiling and running on your local. Normally, you probably want to compile and package your java app into a jar or war and then transfer that up to your EC2 server. If you install linux os on your EC2 server, you can use scp or a FTP client to transfer your files over sftp to move the files from your local up to the server. Once the latest files are up on your server, you can ssh to the server and start your app.
... and create a MySQL database etc ...
Installing mysql is going to be specific to the OS you choose to install on your server. For example, you can install mysql easily on Ubuntu with a command like:
sudo aptitude install mysql
Again, there will be more system-admin-type stuff to learn here specific to mysql databases.
So, it's definitely doable. An experienced sys admin could build a AMI instance pretty easily/quickly. If this is your first experience with system administration, I'd suggest finding an old Desktop you have lying around and try installing Ubuntu and all the required libraries and tools you need (mysql, jdk, ssh, etc..). Get your java program working on the old desktop and then it should be pretty easy to create an AMI from that. Then you can run your custom AMI on EC2 and will be set up.
If you don't have a spare desktop lying around, you can use one of the Virtual Machine products like VMWare Player or Sun's VirtualBox and build a server instance on one of those.
If you want to avoid the hassle of managing the entire install of the Operating system, you might want to look at services like slicehost and/or linode instead of EC2. They give you ssh access to a pre-installed server. And it's as easy as clicking a button to install programs like mysql, etc.
Hope this answer is relevant and helpful, good luck.
- Dave
If you can use Tomcat as your server - you might want to try Amazon Web Services Elastic Beanstalk It will greatly simplify your task by providing an easy instance of Tomcat to deploy.
EDIT: AWS has a full section on how to develop using Java here: http://aws.amazon.com/java/
How comfortable are you with remote administering a server and solving problems most people have never heard of? That's what you're talking about.
You'll create your EC2 instance, log into it and configure it like you would any other server you're working with. You can download JDKs to it, dbs etc. You might consider using a tool like Chef to help you. You'll use ftp and scp to copy files to the server.
You're probably going to want your Java server on one box and have it talk to a separate db server since you say you may want multiple servers.
Once the server is working the way you like it you can create an image of it to use to launch multiple instances, then configure a load balancer to point at your servers.
If you can create a MySql db on your local box you can create it in the cloud. If not?....
Running Applications needs app. s/w to run them.
Apps such as Cyber Duck helps to upload files via SFTP.
I am successful in doing the same problem addressed.. Trust me, YOU CAN DO IT. All u need is interfaces for server services such as MySQL (use WorkBench to connect using key pair), terminal access (using PUTTY/SSH via MAC) and well you are good to go
I think to get started is to use a a whole unit packed into one archive.
Think of it as you have your Java files compiled, a embedded http server say tomcat. Now all of this packed in a jar ready to be deployed on Amazon's EC2. Use this link for more on embedded servers and relevant code.
As far as database is concerned, you can use Amazon's RDS. You can configure mysql on RDS and obtain a link to it for connection. Amazon RDS would help you to get started with database.
Now that you have everything ready to start and work with application. Now is the time to deploy on Amazon AMI
Perform Following steps to deploy you app on Amazon EC2:
Create a server instance on Amazon EC2 refer EC2 getting started.
Receive Private key file i.e. .pem(receive from Step 1), this file would help you to login to your server instance and perform SFTP.
Use Putty or similar SSH client to login to your EC2 server instance created on Step-1 using .pem file and server's public DNS, refer Accessing Instance for SSH login.
Transfer the archived files packed into jar to server using SFTP client, WinnSCP being one such client, refer SFTP for more on it.
Run application. Note in case of jar file you can simply fire java -jar TomcatApp.jar.
If everything is configured properly you would be to access the app using EC2 instance's public DNS or public IP, which would be of the form:
http://<public_dns_address>:<port number>/servlet
Hope it helps you to get started and provide you an overall view.
I am developing a java based application; its pertinent requirements are listed below
Large datasets exist on several machines on network. my program needs to (remotely) execute a java program to process these data sets and fetch the results
A user on a windows desktop will need to process datasets (several gigs) on machine A. My program can reside on the user's machine. He will execute my program from his machine and initiate the dataset processing on remote machine(s)
Instead of getting the dataset over the network from the remote machine to his machine, he will execute the program on the remote machine and fetch results
The user may have open access to the other machines but ftp is the requirement
Data should not be brought through network to the user's machine.
Users have windows OS
My question(s)
How can I perform this kind of remote process execution ? Any ideas?
I am looking at hadoop; I am working on Windows XP. I was unable to get hadoop working for a single node cluster; I am unable to find good documentation. I therefore haven't quite tested hadoop. Any comments on if I am on the right track?
Any links any of you has found useful for installation of hadoop and trouble shooting?
Thanks in advance for any responses. Do please let me know if I should provide any more/specific details.
-jv
Java has a RMI API that you could use, assuming that you can have a JAVA VM running on your remote machines. That's the lightest weight solution. The next lightest weight would be straight socket communication. After that you're getting into EJB servers or Web Servers, which is probably overkill.
Have a look at how to write web services with Java 6. That allows you to publish a method as a web service with an annotation. A web service client is small and does not require additional software. I found the Idea IntelliJ IDE easy to use, and generated a pure Java 6 client.
Then it essentially boils down to making a "normal" method call, and processing the result.
Keep it simple. Grid software is most likely not what you want.