I wanted to delete AWS EC2 volumes which were attached to instances.
But, I could not find any implementation in AWS's java sdk to do so.
Its equivalent cli command is :
/usr/bin/aws ec2 delete-volume --volume-id $volumeId
But I want to do it via java sdk.
Any suggestions?
Use deleteVolume(DeleteVolumeRequest request):
Deletes the specified EBS volume. The volume must be in the available state (not attached to an instance).
See also: DeleteVolume documentation
How can I automatically transfer a log file which is present in amazon ec2 instance to my local machine. Can anyone give an idea about it. I would like to do it by a java API.
Currently I was using Chef "flume cookbook", but i was stuck at the part of automating it. And also that chef can be used in java APIs, so the process can be automated.
I have some Virtual machine running with same configuration. On those VMs I have installed my application.
I want to set some custom unique identifier in my VM , so that I know that if particular (from my application) request is coming from which VM that request is coming(or that application is running in which VM). As same application is running inside all VM, so to differentiate each request , I want to setup some unique custom identifier at the time of creation of Virtual machine.
I want to setup some system properties/Environment variable at the time of creation of virtual machine.
I am using Windows Azure Rest API and Java to create and manage Virtual machine on Windows Azure Portal. I want to achieve this by using Java programing language.
Please let me know if you have nay information about same.
Each instance should have a different SID because the machines are sysprepped with the generalize option (the same applies when you want to provide a custom image). You can get the SID using the code in this question, and use that as identifier:
string domainName;
Helper.SID_NAME_USE accountType;
SecurityIdentifier machineSID =
Helper.LookupAccountName("", Environment.MachineName, out domainName, out accountType);
Note: even though it says domainName in the code, your machine doesn't need to be in a domain for this to work.
To set a "custom identifier" on your VM you can use environment variables. Since you're using Java the easiest way to set these is probably the setx.exe command line tool (download). You can call the tool like this:
setx.exe VmIdentifier 123 -m
Then you should be able to read it using System:
System.getenv("VmIdentifier")
It's not really clear to me if you're using Virtual Machines (IaaS) or Cloud Services (PaaS). If you're using Virtual Machines, it will be up to you to deploy the application, and while doing so, you can also execute setx.exe.
Now, if you're using Cloud Services, you actually upload a package which gets deployed on your instances. And you get support for startup tasks (which run before your application starts): How to Define Startup Tasks for a Role. You can use a startup tasks to run setx.exe, but make sure you run it with executionContext set to elevated .
You can connect to your VM with remote desktop services, so it should be trivial to connect and add a setting to your application's web.config file which identifies the deployment, or, if you want your web.config files to be identical, then you could upload a small text file.
Alternatively, your server must already have a unique name. I assume you're running a .Net application? If so then you should be able to use Server.MachineName (http://msdn.microsoft.com/en-us/library/system.web.httpserverutility.machinename%28v=vs.71%29.aspx) to get the unique name of the box your code is running on.
I want to place a server program written in Java on the cloud. It would accept TCP socket connections from clients (clients are android phones using 3G), do some computations, save stuff to a MySQL database (also on EC2), and send stuff back to the clients over the TCP connections. It may even be necessary to create several instances of the server (i.e. a process group).
Is this easy to do? I think I can make a AMI, but I'm not sure how to upload Java files, compile and run them, and create a MySQL database etc
Any help would be much appreciated.
Take a look at using Amazon Elastic Beanstalk. Beanstalk is Amazon's PaaS offering and it will alleviate a lot of the system administration burden. Here's a quick description from their docs:
AWS Elastic Beanstalk is an even
easier way for you to quickly deploy
and manage applications in the AWS
cloud. You simply upload your
application, and Elastic Beanstalk
automatically handles the deployment
details of capacity provisioning, load
balancing, auto-scaling, and
application health monitoring.
Also, if you're interested in using MySQL then you should look at Amazon RDS. Again, this will alleviate the system administration burden for your database tier. Here's a quick description from their docs:
Amazon Relational Database Service
(Amazon RDS) is a web service that
makes it easy to set up, operate, and
scale a relational database in the
cloud. It provides cost-efficient and
resizable capacity while managing
time-consuming database administration
tasks, freeing you up to focus on your
applications and business.
Is this easy to do? I think I can make
a AMI, ...
I think the answer depends on how comfortable you are with system administration in general. Creating a AMI to run in EC2 is really pretty much the same as creating a physical server or a VM image. You'll need to install an operating system, and then install tools, libraries and programs you need (like mysql, the jdk, ssh, etc).
You can save yourself a little work by using one of Amazon's pre-built AMI's http://aws.amazon.com/amis/. But, ultimately, you'll be responsible for all system administration of the server. If you've never built a server from the ground up, you have a pretty big learning curve ahead of you. It's not insurmountable, but just be warned that the devil is in the details; there's a ton of stuff you'll need to learn ;-)
... but I'm not sure how to upload Java
files, compile and run them, ...
Once the server is setup and running in EC2, compiling them and running java files is just the same as compiling and running on your local. Normally, you probably want to compile and package your java app into a jar or war and then transfer that up to your EC2 server. If you install linux os on your EC2 server, you can use scp or a FTP client to transfer your files over sftp to move the files from your local up to the server. Once the latest files are up on your server, you can ssh to the server and start your app.
... and create a MySQL database etc ...
Installing mysql is going to be specific to the OS you choose to install on your server. For example, you can install mysql easily on Ubuntu with a command like:
sudo aptitude install mysql
Again, there will be more system-admin-type stuff to learn here specific to mysql databases.
So, it's definitely doable. An experienced sys admin could build a AMI instance pretty easily/quickly. If this is your first experience with system administration, I'd suggest finding an old Desktop you have lying around and try installing Ubuntu and all the required libraries and tools you need (mysql, jdk, ssh, etc..). Get your java program working on the old desktop and then it should be pretty easy to create an AMI from that. Then you can run your custom AMI on EC2 and will be set up.
If you don't have a spare desktop lying around, you can use one of the Virtual Machine products like VMWare Player or Sun's VirtualBox and build a server instance on one of those.
If you want to avoid the hassle of managing the entire install of the Operating system, you might want to look at services like slicehost and/or linode instead of EC2. They give you ssh access to a pre-installed server. And it's as easy as clicking a button to install programs like mysql, etc.
Hope this answer is relevant and helpful, good luck.
- Dave
If you can use Tomcat as your server - you might want to try Amazon Web Services Elastic Beanstalk It will greatly simplify your task by providing an easy instance of Tomcat to deploy.
EDIT: AWS has a full section on how to develop using Java here: http://aws.amazon.com/java/
How comfortable are you with remote administering a server and solving problems most people have never heard of? That's what you're talking about.
You'll create your EC2 instance, log into it and configure it like you would any other server you're working with. You can download JDKs to it, dbs etc. You might consider using a tool like Chef to help you. You'll use ftp and scp to copy files to the server.
You're probably going to want your Java server on one box and have it talk to a separate db server since you say you may want multiple servers.
Once the server is working the way you like it you can create an image of it to use to launch multiple instances, then configure a load balancer to point at your servers.
If you can create a MySql db on your local box you can create it in the cloud. If not?....
Running Applications needs app. s/w to run them.
Apps such as Cyber Duck helps to upload files via SFTP.
I am successful in doing the same problem addressed.. Trust me, YOU CAN DO IT. All u need is interfaces for server services such as MySQL (use WorkBench to connect using key pair), terminal access (using PUTTY/SSH via MAC) and well you are good to go
I think to get started is to use a a whole unit packed into one archive.
Think of it as you have your Java files compiled, a embedded http server say tomcat. Now all of this packed in a jar ready to be deployed on Amazon's EC2. Use this link for more on embedded servers and relevant code.
As far as database is concerned, you can use Amazon's RDS. You can configure mysql on RDS and obtain a link to it for connection. Amazon RDS would help you to get started with database.
Now that you have everything ready to start and work with application. Now is the time to deploy on Amazon AMI
Perform Following steps to deploy you app on Amazon EC2:
Create a server instance on Amazon EC2 refer EC2 getting started.
Receive Private key file i.e. .pem(receive from Step 1), this file would help you to login to your server instance and perform SFTP.
Use Putty or similar SSH client to login to your EC2 server instance created on Step-1 using .pem file and server's public DNS, refer Accessing Instance for SSH login.
Transfer the archived files packed into jar to server using SFTP client, WinnSCP being one such client, refer SFTP for more on it.
Run application. Note in case of jar file you can simply fire java -jar TomcatApp.jar.
If everything is configured properly you would be to access the app using EC2 instance's public DNS or public IP, which would be of the form:
http://<public_dns_address>:<port number>/servlet
Hope it helps you to get started and provide you an overall view.
I am developing a java based application; its pertinent requirements are listed below
Large datasets exist on several machines on network. my program needs to (remotely) execute a java program to process these data sets and fetch the results
A user on a windows desktop will need to process datasets (several gigs) on machine A. My program can reside on the user's machine. He will execute my program from his machine and initiate the dataset processing on remote machine(s)
Instead of getting the dataset over the network from the remote machine to his machine, he will execute the program on the remote machine and fetch results
The user may have open access to the other machines but ftp is the requirement
Data should not be brought through network to the user's machine.
Users have windows OS
My question(s)
How can I perform this kind of remote process execution ? Any ideas?
I am looking at hadoop; I am working on Windows XP. I was unable to get hadoop working for a single node cluster; I am unable to find good documentation. I therefore haven't quite tested hadoop. Any comments on if I am on the right track?
Any links any of you has found useful for installation of hadoop and trouble shooting?
Thanks in advance for any responses. Do please let me know if I should provide any more/specific details.
-jv
Java has a RMI API that you could use, assuming that you can have a JAVA VM running on your remote machines. That's the lightest weight solution. The next lightest weight would be straight socket communication. After that you're getting into EJB servers or Web Servers, which is probably overkill.
Have a look at how to write web services with Java 6. That allows you to publish a method as a web service with an annotation. A web service client is small and does not require additional software. I found the Idea IntelliJ IDE easy to use, and generated a pure Java 6 client.
Then it essentially boils down to making a "normal" method call, and processing the result.
Keep it simple. Grid software is most likely not what you want.