I'm building a Git client in Java and would like to write automated tests that verify if Git commands running on an HTTP(s) server are running correctly. I've seen any number of local small Java HTTP servers, but it's unclear to me what would be involved in making them respond correctly to Git commands. I'm trying to avoid installing a local HTTP server because I'd like these automated tests to run anywhere. Can anyone point me in the right direction?
FYI, for ssh remote testing, I'm having fantastic success with the sshd mina system, and using its built-in GitPackCommandFactory. Something similar to HTTP(s) would be wonderful.
The smart HTTP protocol of git is implemented in git-http-backend. This is a CGI binary that can be used in combination with some web-servers. Unfortunately it ignores the content-length of the request and relies on the server to close its input, which makes it incompatible with some CGI servers.
If you have python available, an easy way to run it would be using the CGIHTTPServer. This one is affected by the mentioned incompatibility. With a rather ugly workaround you can nevertheless make it work:
$ python -c 'import CGIHTTPServer; CGIHTTPServer.CGIHTTPRequestHandler.have_fork = False; CGIHTTPServer.test()' 8000
Or with python 3:
$ python3 -c 'import http.server; http.server.CGIHTTPRequestHandler.have_fork = False; http.server.test(HandlerClass=http.server.CGIHTTPRequestHandler, port=8000)'
These set the internal have_fork of CGIHTTPRequestHandler to False which causes the implementation to use subprocesses and pipes instead of fork to run the CGI binary. With this method, the incoming requests are buffered and written to a pipe, which is then closed. This fits the expected model of git-http-backend and therefore makes it work.
If you are on a system without fork in the first place (i.e. Windows) you don't need the workaround and can directly use:
$ python -m CGIHTTPServer 8000
$ python3 -m http.server --cgi 8000
Both setups require that git-http-backend is made available under a cgi-bin directory. The easiest setup is to create the cgi-bin directory inside a bare git repository, symlink or copy the git-http-backend binary into it and start the server from the bare repository:
$ git clone --bare <repo> bare.git
$ cd bare.git
$ touch git-daemon-export-ok
$ mkdir cgi-bin
$ ln -s /usr/lib/git-core/git-http-backend cgi-bin/git
$ python -c 'import CGIHTTPServer; CGIHTTPServer.CGIHTTPRequestHandler.have_fork = False; CGIHTTPServer.test()' 8000
With that setup, the repository is served at http://localhost:8000/cgi-bin/git because git-http-backend was symlinked to cgi-bin/git. So the following should then work:
$ git clone http://localhost:8000/cgi-bin/git clonedRepoName
To allow pushing into the bare repository use:
$ git config http.receivepack true
More advanced setups, with more than one repository served, can be made using the appropriate environment variables. Since the environment is inherited by the server and passed to the CGI binary, simply exporting the desired environment variables should work. See the git-http-backend documentation for all possible configuration and environment variables.
Based on Gustave's suggestion, I've discovered SimpleHTTPServer within JGit. Works like a charm. Thank you!
Related
I'm trying to install Kafka message queue on Windows for testing purposes (not for production).
I found this article on how to install Apache Kafka 0.8 on Windows: http://janschulte.wordpress.com/2013/10/13/apache-kafka-0-8-on-windows/
It's a good article, but it's outdated, unfortunately.
Does someone know a way to achieve that?
Ok, it's finally not complicated :)
The only steps are:
Edit (2021/10) Latest versions of Kafka end up with an AccessDeniedException upon startup, and even when successfully started, throw the same error during log rotation. Kafka utilizes Java methods designed for Unix filesystem libraries, so installing Kafka in WSL2 is recommended (link to Confluent blog, but steps also applicable to Apache Kafka direct installation).
Old answer
Download Kafka and uncompress it somewhere nice (let's say C:/Kafka)
Install Cygwin
If using Cygwin, edit \bin\kafka-run-class.sh and at the end of the file, change
`exec $JAVA $KAFKA_HEAP_OPTS $KAFKA_JVM_PERFORMANCE_OPTS $KAFKA_GC_LOG_OPTS $KAFKA_JMX_OPTS $KAFKA_LOG4J_OPTS -cp $CLASSPATH $KAFKA_OPTS "$#"`
to
``exec java $KAFKA_HEAP_OPTS $KAFKA_JVM_PERFORMANCE_OPTS $KAFKA_GC_LOG_OPTS $KAFKA_JMX_OPTS $KAFKA_LOG4J_OPTS -cp `cygpath -wp $CLASSPATH` $KAFKA_OPTS "$#"``
In Environment Variables, Add java to your Path System Variable:
That's it.. you can now run ZooKeeper and Kafka servers and start playing with topics and stuff..
These are the steps I followed to run kafka on Windows
Install Zookeeper first (I downloaded v3.3.6) zookeeper-3.3.6.tar.gz
Extract Zookeeper and run this command in powershell/cmd \zookeeper-3.3.6\bin> .\zkServer.cmd Now this should up a Zookeeper instance on localhost:2181
Download Kafka binary version (I downloaded v0.10.0.1)kafka_2.10-0.10.0.1.tgz
Extract Kafka, time to modify some configs
Inside Kafka extraction you can find .\config\server.properties
In .\config\server.properties replace log.dirs=c:/kafka/kafka-logs
Note: Make sure to create those folders in relevant paths
Happy news: Now Kafka ships with windows .bat scripts, You can find these files inside ./bin/windows folder
Start powershell/cmd and run this command to start Kafka broker .\bin\windows\kafka-server-start.bat .\config\server.properties
DONE!, Now you have a running Zookeeper instance and a Kafka broker.
You will probably find that Kafka comes with Windows .bat files under the bin folder to help you run Kafka under windows. However, those bat files are not updated since version 0.8.0 and some of Kafka's logic has changed (regarding topics, etc.).
I have tried two ways to solve this problem. One is to use Cygwin or MinGW to simulate Linux environment and run the Linux shell script, but there would be tons of other problems related to path names. The other solution, which is simpler and come with less trouble, is to use the corrected version of Windows bat files.
Please refer to this blog post.
Download any binary version(I've downloaded kafka_2.11-0.10.2.0) of Kafka from https://kafka.apache.org/downloads
Extract it to any folder(I've extracted to "C:\Kafka")
Open Command Prompt
Go to the folder that you extract Kafka(C:\Kafka\kafka_2.11-0.10.2.0)
Run this command to start Zookeeper
.\bin\windows\zookeeper-server-start.bat .\config\zookeeper.properties
Run this command to start Kafka
.\bin\windows\kafka-server-start.bat .\config\server.properties
Now it works!
In version 0.9.0. there are scripts in "bin" folder for windows. You can use them.
Running From A windows Shell
As of April 2019, downloading Kafka from their website worked on Windows almost right out of the box
Downloading and using the windows version .bat files is described here:
https://kafka.apache.org/quickstart
I ran into two problems when I did this:
1) JAVA_HOME was set to an unsupported JDK which led to this kind of error
Exception in thread "main" java.lang.VerifyError: Uninitialized object
exists on backward branch 209
Replacing with JDK 11 solved the problem.
2) The JAVA_HOME must not contain spaces which caused a 'cannot find the specified file' error. To fix this I used a shortened path like set JAVA_HOME=C:\Progra~1\Java\jdk-11.0.1
Running From Cygwin
If instead of the .bat files, you want to run the .sh files from cygwin, there is actually quite a bit that needs to be done, and even after that there may be problems that come up later. I can't really recommend this option, but I do use it as it's pretty convenient for some purposes.
If your JAVA_HOME path contains a space e.g. "C:\Program Files\Java\Jdk..." you will see something like this:
bin/kafka-run-class.sh: line 305: exec: C:\Program: not found
One solution is to copy the jdk to a path without spaces, and change the Java home accordingly.
If you don't want to change the JDK location, you can change the cygwin env variable as follows:
JAVA_HOME="/cygdrive/c/Program Files/Java/jdk-11.0.1"
and change the line
exec $JAVA $KAFKA_HEAP_OPTS $KAFKA_JVM_PERFORMANCE_OPTS $KAFKA_GC_LOG_OPTS $KAFKA_JMX_OPTS $KAFKA_LOG4J_OPTS -cp $CLASSPATH $KAFKA_OPTS "$#"
to
exec "$JAVA" $KAFKA_HEAP_OPTS $KAFKA_JVM_PERFORMANCE_OPTS $KAFKA_GC_LOG_OPTS $KAFKA_JMX_OPTS $KAFKA_LOG4J_OPTS -cp $CLASSPATH $KAFKA_OPTS "$#"
There was another problem with the log parameters in kafka-run-class.sh and I had to replace the line
KAFKA_GC_LOG_OPTS="-Xlog:gc*:file=$LOG_DIR/$GC_LOG_FILE_NAME:time,tags:filecount=10,filesize=102400"
with
KAFKA_GC_LOG_OPTS="-Xlog:gc*"
And even after all these changes I occasionally run into problems of Kafka shutting down because of an incompatible windows style path in the logs directories as described here:
Kafka 1.0 stops with FATAL SHUTDOWN error. Logs directory failed
In short, you may be better off running the Kafka .bat scripts from the windows directory...
With Chocolatey and Powershell:
chocolatey.exe install kafka -> installs Kafka into C:\ProgramData\chocolatey\lib\kafka\tools\kafka_2.11-1.0.0\ - your version might differ of course
add C:\ProgramData\chocolatey\lib\kafka\tools\kafka_2.11-1.0.0\ to your powershell path using notepad $PROFILE
After restarting powershell kafka-console-consumer should work as a command.
You also need to install ssl.properties with keystore and truststore. Put them e.g. into C:\ProgramData\kafka and put that into ssl.properties (please note the escaping of the backslashes):
security.protocol=SSL
ssl.truststore.location=C:\\ProgramData\\kafka\\kafka-truststore.jks
ssl.truststore.password=PASSWORD
ssl.keystore.location=C:\\ProgramData\\kafka\\kafka-keystore.jks
ssl.keystore.password=PASSWORD
ssl.key.password=PASSWORD
client.id=console-test
Now things like kafka-consumer-groups --bootstrap-server SERVERNAME:9094 --command-config C:\ProgramData\kafka\ssl.properties --list should first pop up a firewall warning which you'd need to accept and then output a list of groups.
Kafka & Zookeeper have inbuilt Windows specific scripts as verified in the latest version. As mentioned in other other answers, the directory for windows scripts will be under kafka_2.12-2.4.0\bin\windows for Kafka. Zookeeper scripts are under /bin only but ending with "*.cmd"
There will be few minor additional steps you will have to do with ZooKeeper & Kafka like
Creating zoo.cfg for ZooKeeper
Configuring windows specific data directory for ZeeKeeper
Kafka comes with default configuration file & it has very good OS specific handling i.e. log.dirs=/tmp/kafka-logs but it will create that directory structure under C:/ directory seamlessly without failing.
It is also easy to quickly setup a Kafka Cluster with multiple Kafka Brokers on Windows with minor configurations. You can also create topics, publish/consume messages, verify topics/partitions etc. all from Windows Command Prompt. Here is a good detailed reference for the windows specific setup Kafka Setup on Windows OS | Basic Installation, Setup, Verification, Cluster Setup, Storage
I had configured Zookeeper and Apache Kafka on my Windows machine.
For Zookeeper, simply configure ZOOKEEPER_HOME and set the PATH also and then rename the zoo.sample.cfg to zoo.cfg and also change the path for dataDir in the zoo.cfg
For Apache Kafka, simply unzip and start the server from bin folder and start the Producer/Consumer from windows folder under the Kafka directory
I'm trying to install Kafka message queue on Windows for testing purposes (not for production).
I found this article on how to install Apache Kafka 0.8 on Windows: http://janschulte.wordpress.com/2013/10/13/apache-kafka-0-8-on-windows/
It's a good article, but it's outdated, unfortunately.
Does someone know a way to achieve that?
Ok, it's finally not complicated :)
The only steps are:
Edit (2021/10) Latest versions of Kafka end up with an AccessDeniedException upon startup, and even when successfully started, throw the same error during log rotation. Kafka utilizes Java methods designed for Unix filesystem libraries, so installing Kafka in WSL2 is recommended (link to Confluent blog, but steps also applicable to Apache Kafka direct installation).
Old answer
Download Kafka and uncompress it somewhere nice (let's say C:/Kafka)
Install Cygwin
If using Cygwin, edit \bin\kafka-run-class.sh and at the end of the file, change
`exec $JAVA $KAFKA_HEAP_OPTS $KAFKA_JVM_PERFORMANCE_OPTS $KAFKA_GC_LOG_OPTS $KAFKA_JMX_OPTS $KAFKA_LOG4J_OPTS -cp $CLASSPATH $KAFKA_OPTS "$#"`
to
``exec java $KAFKA_HEAP_OPTS $KAFKA_JVM_PERFORMANCE_OPTS $KAFKA_GC_LOG_OPTS $KAFKA_JMX_OPTS $KAFKA_LOG4J_OPTS -cp `cygpath -wp $CLASSPATH` $KAFKA_OPTS "$#"``
In Environment Variables, Add java to your Path System Variable:
That's it.. you can now run ZooKeeper and Kafka servers and start playing with topics and stuff..
These are the steps I followed to run kafka on Windows
Install Zookeeper first (I downloaded v3.3.6) zookeeper-3.3.6.tar.gz
Extract Zookeeper and run this command in powershell/cmd \zookeeper-3.3.6\bin> .\zkServer.cmd Now this should up a Zookeeper instance on localhost:2181
Download Kafka binary version (I downloaded v0.10.0.1)kafka_2.10-0.10.0.1.tgz
Extract Kafka, time to modify some configs
Inside Kafka extraction you can find .\config\server.properties
In .\config\server.properties replace log.dirs=c:/kafka/kafka-logs
Note: Make sure to create those folders in relevant paths
Happy news: Now Kafka ships with windows .bat scripts, You can find these files inside ./bin/windows folder
Start powershell/cmd and run this command to start Kafka broker .\bin\windows\kafka-server-start.bat .\config\server.properties
DONE!, Now you have a running Zookeeper instance and a Kafka broker.
You will probably find that Kafka comes with Windows .bat files under the bin folder to help you run Kafka under windows. However, those bat files are not updated since version 0.8.0 and some of Kafka's logic has changed (regarding topics, etc.).
I have tried two ways to solve this problem. One is to use Cygwin or MinGW to simulate Linux environment and run the Linux shell script, but there would be tons of other problems related to path names. The other solution, which is simpler and come with less trouble, is to use the corrected version of Windows bat files.
Please refer to this blog post.
Download any binary version(I've downloaded kafka_2.11-0.10.2.0) of Kafka from https://kafka.apache.org/downloads
Extract it to any folder(I've extracted to "C:\Kafka")
Open Command Prompt
Go to the folder that you extract Kafka(C:\Kafka\kafka_2.11-0.10.2.0)
Run this command to start Zookeeper
.\bin\windows\zookeeper-server-start.bat .\config\zookeeper.properties
Run this command to start Kafka
.\bin\windows\kafka-server-start.bat .\config\server.properties
Now it works!
In version 0.9.0. there are scripts in "bin" folder for windows. You can use them.
Running From A windows Shell
As of April 2019, downloading Kafka from their website worked on Windows almost right out of the box
Downloading and using the windows version .bat files is described here:
https://kafka.apache.org/quickstart
I ran into two problems when I did this:
1) JAVA_HOME was set to an unsupported JDK which led to this kind of error
Exception in thread "main" java.lang.VerifyError: Uninitialized object
exists on backward branch 209
Replacing with JDK 11 solved the problem.
2) The JAVA_HOME must not contain spaces which caused a 'cannot find the specified file' error. To fix this I used a shortened path like set JAVA_HOME=C:\Progra~1\Java\jdk-11.0.1
Running From Cygwin
If instead of the .bat files, you want to run the .sh files from cygwin, there is actually quite a bit that needs to be done, and even after that there may be problems that come up later. I can't really recommend this option, but I do use it as it's pretty convenient for some purposes.
If your JAVA_HOME path contains a space e.g. "C:\Program Files\Java\Jdk..." you will see something like this:
bin/kafka-run-class.sh: line 305: exec: C:\Program: not found
One solution is to copy the jdk to a path without spaces, and change the Java home accordingly.
If you don't want to change the JDK location, you can change the cygwin env variable as follows:
JAVA_HOME="/cygdrive/c/Program Files/Java/jdk-11.0.1"
and change the line
exec $JAVA $KAFKA_HEAP_OPTS $KAFKA_JVM_PERFORMANCE_OPTS $KAFKA_GC_LOG_OPTS $KAFKA_JMX_OPTS $KAFKA_LOG4J_OPTS -cp $CLASSPATH $KAFKA_OPTS "$#"
to
exec "$JAVA" $KAFKA_HEAP_OPTS $KAFKA_JVM_PERFORMANCE_OPTS $KAFKA_GC_LOG_OPTS $KAFKA_JMX_OPTS $KAFKA_LOG4J_OPTS -cp $CLASSPATH $KAFKA_OPTS "$#"
There was another problem with the log parameters in kafka-run-class.sh and I had to replace the line
KAFKA_GC_LOG_OPTS="-Xlog:gc*:file=$LOG_DIR/$GC_LOG_FILE_NAME:time,tags:filecount=10,filesize=102400"
with
KAFKA_GC_LOG_OPTS="-Xlog:gc*"
And even after all these changes I occasionally run into problems of Kafka shutting down because of an incompatible windows style path in the logs directories as described here:
Kafka 1.0 stops with FATAL SHUTDOWN error. Logs directory failed
In short, you may be better off running the Kafka .bat scripts from the windows directory...
With Chocolatey and Powershell:
chocolatey.exe install kafka -> installs Kafka into C:\ProgramData\chocolatey\lib\kafka\tools\kafka_2.11-1.0.0\ - your version might differ of course
add C:\ProgramData\chocolatey\lib\kafka\tools\kafka_2.11-1.0.0\ to your powershell path using notepad $PROFILE
After restarting powershell kafka-console-consumer should work as a command.
You also need to install ssl.properties with keystore and truststore. Put them e.g. into C:\ProgramData\kafka and put that into ssl.properties (please note the escaping of the backslashes):
security.protocol=SSL
ssl.truststore.location=C:\\ProgramData\\kafka\\kafka-truststore.jks
ssl.truststore.password=PASSWORD
ssl.keystore.location=C:\\ProgramData\\kafka\\kafka-keystore.jks
ssl.keystore.password=PASSWORD
ssl.key.password=PASSWORD
client.id=console-test
Now things like kafka-consumer-groups --bootstrap-server SERVERNAME:9094 --command-config C:\ProgramData\kafka\ssl.properties --list should first pop up a firewall warning which you'd need to accept and then output a list of groups.
Kafka & Zookeeper have inbuilt Windows specific scripts as verified in the latest version. As mentioned in other other answers, the directory for windows scripts will be under kafka_2.12-2.4.0\bin\windows for Kafka. Zookeeper scripts are under /bin only but ending with "*.cmd"
There will be few minor additional steps you will have to do with ZooKeeper & Kafka like
Creating zoo.cfg for ZooKeeper
Configuring windows specific data directory for ZeeKeeper
Kafka comes with default configuration file & it has very good OS specific handling i.e. log.dirs=/tmp/kafka-logs but it will create that directory structure under C:/ directory seamlessly without failing.
It is also easy to quickly setup a Kafka Cluster with multiple Kafka Brokers on Windows with minor configurations. You can also create topics, publish/consume messages, verify topics/partitions etc. all from Windows Command Prompt. Here is a good detailed reference for the windows specific setup Kafka Setup on Windows OS | Basic Installation, Setup, Verification, Cluster Setup, Storage
I had configured Zookeeper and Apache Kafka on my Windows machine.
For Zookeeper, simply configure ZOOKEEPER_HOME and set the PATH also and then rename the zoo.sample.cfg to zoo.cfg and also change the path for dataDir in the zoo.cfg
For Apache Kafka, simply unzip and start the server from bin folder and start the Producer/Consumer from windows folder under the Kafka directory
I'm running in to an error when I try to run my server application from Eclipse. The error is java.net.BindException: Permission denied. I think this is because I am using port 443 to set up an SSL connection. I can get around this problem if I run my code on the command line using java and sudo. Is there a way to set up Eclipse so that when I hit the run button, my application is executed with sudo?
You can follow these steps to compile/debug applications as superuser.
Rename your java-application
sudo mv /usr/lib/jvm/java-6-openjdk/jre/bin/java /usr/lib/jvm/java-6-openjdk/jre/bin/java.ori
Create following script and store it as /usr/lib/jvm/java-6-openjdk/jre/bin/java
#!/bin/bash
# file: /usr/lib/jvm/java-6-openjdk/jre/bin/java
# descr: Starter for jdk. Runs jdk as root when
# cmd-line-arg "--run-as-root" is specified.
#
jre="/usr/lib/jvm/java-6-openjdk/jre/bin/java.ori"
run_as_root=false
args=
# Filter command-line argument
for arg in "$#"
do
case "$arg" in
--run-as-root) run_as_root=true
;;
*) args="$args $arg"
;;
esac
done
# Remove leading whitespaces
args=$(echo $args | sed -e 's/^[ \t]*//')
if $run_as_root
then
echo "WARNING: Running as root!"
gksu "$jre $args"
else
$jre $args
fi
Change the permissions to make it executable
sudo chmod 0755 /usr/lib/jvm/java-6-openjdk/jre/bin/java
Startup eclipse
Go to Window->Preferences->Java->Installed JREs
Duplicate java-6-openjdk to java-6-openjdk-root
Edit JRE and add "--run-as-root" as Default VM Argument
To run projects as root you need to follow these steps:
Go to Project->Properties->Java Build Path
Double-Click the JRE System Library and choose in Alternate JRE "java-6-openjdk-root"
Note: The idea is from http://www.eclipse.org/forums/index.php/mv/msg/87353/724852/#msg_724852
Assuming you are on Linux (*nix),
How about starting your eclipse session via a sudo command?
Such as
sudo ~/eclipse/eclipse
Now whatever you do from eclipse will have the sudo context?
As mentioned in this thread:
In order to open a port below 1024 on Unix/Linux systems you need to be
"root".
I also used the argument -Dorg.eclipse.equinox.http.jetty.port=8080 to change the listen port, but this seems to be ignored (according to the stacktrace)
Please use "-Dorg.osgi.service.http.port=8080".
As mentioned in HTTP Service:
org.osgi.service.http.port - specifies the port number to use for the http serving. The default value for this property is 80 (which requires root permission), as per the OSGi specification.
org.osgi.service.http.port.secure - specifies the port number to use for secure http serving. The default value for this property is 443 (which requires root permission), as per the OSGi specification.
Maybe if you try to modify that last property to a value above 1024 it could work without requiring any special privilege.
Another option would be to use iptables or ipfilter to forward port 80 to a port above 1024.
(Can someone contribute a link to a practical and easy-to-understand explanation ?)
A better answer, perhaps, if this serves your needs AND is possible, could be simple port redirection on your router.
Instead of trying to force your linux/unix to open a reserved port, when you are only developing this now (not installing) and you want to run it in a debugger,
set your router to redirect incoming (external) port 443 to a port that is more convenient for your current needs (say 4443).
I think most routers support this, and if yours doesn't it gives your mum a good christmas or birthday present idea!
I am writing C not Java but this should work in either case.
I use remote debug - define a "remote" connection to LOCALHOST which allows you to specify the user you will connect with, specify ROOT. Then define a Remote Application in debug configuration connection: LOCALHOST. Be sure to check "skip download to target path" at the bottom of the main tab as well as under the connection properties window.
You can use Remote Java Application mechanism for this.
Create Debug configuration for Remote Java Application
section in Run -> Debug configurations...
Set your project name
Choose Connection type as Standard (Socket Attach)
Configure Connection properties parameters for your binding
(for you it will be localhost and 443).
Set breakpoint in your app (e.g. at the beginning of the main method)
Run your app from terminal as superuser with following command: java -Xdebug -Xrunjdwp:transport=dt_socket,server=y,address=443 MyApp
Hit debug button in Eclipse for early created Remote Java Application
You code should be stopped on breakpoint in Eclipse!
If you use External tools (Run menu/External tools or an icon next to the Run/Debug icons on the toolbar), you can use any scripts or whatever you like. The scripts may give you elevated rights, or whatever.
On the other hand, this way debugging the application can become very hard, as neither the Run nor Debug commands get associated with this External tool configuration. Maybe it is possible to connect the Eclipse debugger of the application, but I don't know, how that is possible.
You may go this way
create a Makefile with javac calls
add the following line:
setcap 'cap_net_admin=+ep' Server
configure sudo to allow your Eclipse user to run setcap.
So you will have a transparent debugging (no sudo wrapper - gdb ok).
Cons: it is a local security breach.
Solution:
put this to /opt/my-stupid-eclipse
#!/bin/sh
setcap 'cap_net_admin=+ep cap_net_raw=+ep' $1
chmod +x this script and whitelist it on sudo config.
username ALL=(ALL) NOPASSWD: /opt/my-stupid-eclipse
Add it to your makefile, specify path to your Server binary.
Now you have pretty strange but secure script, that cannot be changed by other users... and still a little breach for replacing Server binary with any malicious code, that will gain caps, so no filename check/stricts will help.. can $1 be contaminated with bash commands, no? Guess, no.
I am developing web application using scala. Now it become rather big and I am going to split it to few separated services and glue it with nginx as reverse proxy.
While it was just a scala app it was able to distribute and run it as jar with few config files. But now there is nginx which usually installed as system app. What is the best way to work with nginx in portable manner, to have all app concentrated in one folder?
Should I have *.conf file with host and include reloading nginx in startup script?
Or maybe it will be usefull to have portable nginx binary?
This is for testing and experimental needs only. Production app deployed into server with preconfigured nginx.
Having used Nginx on Ubuntu quite a lot, I found that it is quite effective to keep .conf files with the project work, along with one or two shell scripts to install and configure them.
You didn't mention your OS, but I can share my experience with Ubuntu if that helps. This is more an OS admin topic than a Scala one per se.
For example, a script might symlink your scripts/myapp.conf file(s) something like this:
#!/bin/bash -e
cd $(dirname $0)
TGT=$PWD
cd /etc/nginx/sites-enabled
sudo rm -f myapp.conf
sudo ln -vs $TGT/scripts/myapp.conf myapp.conf
sudo service nginx reload
So you have separated services with nginx as reverse proxy in your production. And need to adapt nginx for test environment ?
If that is one or two machines setup, try not to overenginer setting, but just adapt nginx config manually (and scala server file) as needed (hostname, ports, path etc...).
Just my 2c.
I'm running in to an error when I try to run my server application from Eclipse. The error is java.net.BindException: Permission denied. I think this is because I am using port 443 to set up an SSL connection. I can get around this problem if I run my code on the command line using java and sudo. Is there a way to set up Eclipse so that when I hit the run button, my application is executed with sudo?
You can follow these steps to compile/debug applications as superuser.
Rename your java-application
sudo mv /usr/lib/jvm/java-6-openjdk/jre/bin/java /usr/lib/jvm/java-6-openjdk/jre/bin/java.ori
Create following script and store it as /usr/lib/jvm/java-6-openjdk/jre/bin/java
#!/bin/bash
# file: /usr/lib/jvm/java-6-openjdk/jre/bin/java
# descr: Starter for jdk. Runs jdk as root when
# cmd-line-arg "--run-as-root" is specified.
#
jre="/usr/lib/jvm/java-6-openjdk/jre/bin/java.ori"
run_as_root=false
args=
# Filter command-line argument
for arg in "$#"
do
case "$arg" in
--run-as-root) run_as_root=true
;;
*) args="$args $arg"
;;
esac
done
# Remove leading whitespaces
args=$(echo $args | sed -e 's/^[ \t]*//')
if $run_as_root
then
echo "WARNING: Running as root!"
gksu "$jre $args"
else
$jre $args
fi
Change the permissions to make it executable
sudo chmod 0755 /usr/lib/jvm/java-6-openjdk/jre/bin/java
Startup eclipse
Go to Window->Preferences->Java->Installed JREs
Duplicate java-6-openjdk to java-6-openjdk-root
Edit JRE and add "--run-as-root" as Default VM Argument
To run projects as root you need to follow these steps:
Go to Project->Properties->Java Build Path
Double-Click the JRE System Library and choose in Alternate JRE "java-6-openjdk-root"
Note: The idea is from http://www.eclipse.org/forums/index.php/mv/msg/87353/724852/#msg_724852
Assuming you are on Linux (*nix),
How about starting your eclipse session via a sudo command?
Such as
sudo ~/eclipse/eclipse
Now whatever you do from eclipse will have the sudo context?
As mentioned in this thread:
In order to open a port below 1024 on Unix/Linux systems you need to be
"root".
I also used the argument -Dorg.eclipse.equinox.http.jetty.port=8080 to change the listen port, but this seems to be ignored (according to the stacktrace)
Please use "-Dorg.osgi.service.http.port=8080".
As mentioned in HTTP Service:
org.osgi.service.http.port - specifies the port number to use for the http serving. The default value for this property is 80 (which requires root permission), as per the OSGi specification.
org.osgi.service.http.port.secure - specifies the port number to use for secure http serving. The default value for this property is 443 (which requires root permission), as per the OSGi specification.
Maybe if you try to modify that last property to a value above 1024 it could work without requiring any special privilege.
Another option would be to use iptables or ipfilter to forward port 80 to a port above 1024.
(Can someone contribute a link to a practical and easy-to-understand explanation ?)
A better answer, perhaps, if this serves your needs AND is possible, could be simple port redirection on your router.
Instead of trying to force your linux/unix to open a reserved port, when you are only developing this now (not installing) and you want to run it in a debugger,
set your router to redirect incoming (external) port 443 to a port that is more convenient for your current needs (say 4443).
I think most routers support this, and if yours doesn't it gives your mum a good christmas or birthday present idea!
I am writing C not Java but this should work in either case.
I use remote debug - define a "remote" connection to LOCALHOST which allows you to specify the user you will connect with, specify ROOT. Then define a Remote Application in debug configuration connection: LOCALHOST. Be sure to check "skip download to target path" at the bottom of the main tab as well as under the connection properties window.
You can use Remote Java Application mechanism for this.
Create Debug configuration for Remote Java Application
section in Run -> Debug configurations...
Set your project name
Choose Connection type as Standard (Socket Attach)
Configure Connection properties parameters for your binding
(for you it will be localhost and 443).
Set breakpoint in your app (e.g. at the beginning of the main method)
Run your app from terminal as superuser with following command: java -Xdebug -Xrunjdwp:transport=dt_socket,server=y,address=443 MyApp
Hit debug button in Eclipse for early created Remote Java Application
You code should be stopped on breakpoint in Eclipse!
If you use External tools (Run menu/External tools or an icon next to the Run/Debug icons on the toolbar), you can use any scripts or whatever you like. The scripts may give you elevated rights, or whatever.
On the other hand, this way debugging the application can become very hard, as neither the Run nor Debug commands get associated with this External tool configuration. Maybe it is possible to connect the Eclipse debugger of the application, but I don't know, how that is possible.
You may go this way
create a Makefile with javac calls
add the following line:
setcap 'cap_net_admin=+ep' Server
configure sudo to allow your Eclipse user to run setcap.
So you will have a transparent debugging (no sudo wrapper - gdb ok).
Cons: it is a local security breach.
Solution:
put this to /opt/my-stupid-eclipse
#!/bin/sh
setcap 'cap_net_admin=+ep cap_net_raw=+ep' $1
chmod +x this script and whitelist it on sudo config.
username ALL=(ALL) NOPASSWD: /opt/my-stupid-eclipse
Add it to your makefile, specify path to your Server binary.
Now you have pretty strange but secure script, that cannot be changed by other users... and still a little breach for replacing Server binary with any malicious code, that will gain caps, so no filename check/stricts will help.. can $1 be contaminated with bash commands, no? Guess, no.