I am developing web application using scala. Now it become rather big and I am going to split it to few separated services and glue it with nginx as reverse proxy.
While it was just a scala app it was able to distribute and run it as jar with few config files. But now there is nginx which usually installed as system app. What is the best way to work with nginx in portable manner, to have all app concentrated in one folder?
Should I have *.conf file with host and include reloading nginx in startup script?
Or maybe it will be usefull to have portable nginx binary?
This is for testing and experimental needs only. Production app deployed into server with preconfigured nginx.
Having used Nginx on Ubuntu quite a lot, I found that it is quite effective to keep .conf files with the project work, along with one or two shell scripts to install and configure them.
You didn't mention your OS, but I can share my experience with Ubuntu if that helps. This is more an OS admin topic than a Scala one per se.
For example, a script might symlink your scripts/myapp.conf file(s) something like this:
#!/bin/bash -e
cd $(dirname $0)
TGT=$PWD
cd /etc/nginx/sites-enabled
sudo rm -f myapp.conf
sudo ln -vs $TGT/scripts/myapp.conf myapp.conf
sudo service nginx reload
So you have separated services with nginx as reverse proxy in your production. And need to adapt nginx for test environment ?
If that is one or two machines setup, try not to overenginer setting, but just adapt nginx config manually (and scala server file) as needed (hostname, ports, path etc...).
Just my 2c.
Related
Built a new Windows Server 2016 to act as a target for jenkins orchestrated builds. We are not supposed to use oracle java anymore, so I got coretta openJDK. OpenJDK does not support javaws any more. The executable does not ship with it. I have seen some scripted workarounds, but they do not work and are poorly written to boot.
What are we supposed to use to launch slave agents on Windows machines?
There are several ways to implement this but since this is a windows server you can follow steps on https://github.com/kohsuke/winsw/blob/master/doc/installation.md#winsw-installation-guide to install jenkins slave as service without javaws.
I believe #Mike was referring to: Install Slave as a Windows service (cmd version), which does not need JNLP/javaws.
OR, under Jenkins 2 (we are using 2.121.1 plus WMI Windows Agents (aka windows-slaves:1.4), there is now an option to "Let Jenkins control this Windows slave as a Windows service", essentially in the same fashion as a Linux node.
You must have an Admin account (and password). We had to follow the guidance to edit registry when "Windows agents fail to start via DCOM" (Guidance is for Server 2012) to remotely manage the service. So far, seems to be working OK, YMMV. So far, seems to be working OK, no "subtle probelms"; YMMV.
For entries:
HKEY_LOCAL_MACHINE\SOFTWARE\Classes\Wow6432Node\CLSID{72C24DD5-D70A-438B-8A42-98424B88AFB8}
HKEY_CLASSES_ROOT\CLSID{76A64158-CB41-11D1-8B02-00600806D9B6}
(There are several matches for the keys; just change the two locations indicated)
Launch 'regedit' (as Administrator)
Find (Ctrl+F) the following registry key: "{72C24DD5-D70A-438B-8A42-98424B88AFB8}" in
HKEY_LOCAL_MACHINE\SOFTWARE\Classes\Wow6432Node\CLSID\
Right click and select 'Permissions'
Change owner to administrators group (Advanced...).
Change permissions for administrators group. Grant Full Control.
Change owner back to TrustedInstaller (user is "NT Service\TrustedInstaller" on local machine)
Repeat the steps 1-6 for HKEY_CLASSES_ROOT\CLSID{76A64158-CB41-11D1-8B02-00600806D9B6}
Restart Remote Registry Service (Administrative Tools / Services)
For me the easiest and best working solution to launch Windows Jenkins Agents without JNLP was to run in the command prompt java -jar agent.jar ... command. The command parameters should be grabbed from the url of the agent:
http://your_jenkins_url/computer/your_agent_name_or_ip/
I downloaded the agent.jar on my PC, moved it to a desired directory, launched the Command Prompt from the dir where the agent.jar was moved and executed the proposed command with the appropriate parameters.
In my case:
java -jar agent.jar -jnlpUrl http://your_jenkins_url/computer/your_agent_name_or_ip/slave-agent.jnlp -secret 76986574e97c2b635c7076740dc93326eaaf5a3ad30573144915489a1ccfee44 -workDir "D:\jenkins"
This was one of the proposed approaches in the docs here - https://www.jenkins.io/doc/administration/requirements/upgrade-java-guidelines/
Java Web Start Java Web Start has been removed in Java 11. When a
Jenkins controller is running on Java 11, the Java Web Start button
will no longer appear in the Web UI. Agents for a Java 11 Jenkins
server can’t be launched from a *.jnlp file downloaded to a web
browser.
There are no plans to replace this functionality. Connect agents to
Jenkins on Java 11 with plugins like SSH Build Agents Plugin, with
operating system command line calls to java -jar agent.jar, or by
using containers.
can you all please give me idea on how to integrate ZAP tool with JMeter for security concerns and I need it to involve with daily CI builds??
And if this is wrong stack to ask this question, please direct me to one.
Thanks in advance.
EDIT: I recorded all the JMeter scripts. When I run JMeter, I need ZAP also to apply its security testings for all JMeter API calls. I need JMeter and ZAP to run in sequence for every API.
JMeter can be configure as a proxy to save all ZAP requests and then you can re send same OWASP test on same or environment after fix or change.
you can also configure ZAP to connect through another proxy - this is often necessary in a corporate environment.
You basically need to configure JMeter to send all requests through ZAP Intercepting Proxy. By default ZAP is running on the same host using port 8080 so you either need to run JMeter like:
jmeter -H localhost -P 8000 -n -t test.jmx -l result.jtl
Alternatively you can define the same settings via Java Networking Properties like:
jmeter -Dhttp.proxyHost=localhost -Dhttp.proxyPort=8080 -Dhttps.proxyHost=localhost -Dhttps.proxyPort=8080 -n -t test.jmx -l result.jtl
And finally you can add the next lines to system.properties file (lives in "bin" folder of your JMeter installation)
http.proxyHost=localhost
http.proxyPort=8080
https.proxyHost=localhost
https.proxyPort=8080
JMeter restart will be required to pick the properties up.
More information:
Using JMeter behind a proxy
Apache JMeter Properties Customization Guide
I'm building a Git client in Java and would like to write automated tests that verify if Git commands running on an HTTP(s) server are running correctly. I've seen any number of local small Java HTTP servers, but it's unclear to me what would be involved in making them respond correctly to Git commands. I'm trying to avoid installing a local HTTP server because I'd like these automated tests to run anywhere. Can anyone point me in the right direction?
FYI, for ssh remote testing, I'm having fantastic success with the sshd mina system, and using its built-in GitPackCommandFactory. Something similar to HTTP(s) would be wonderful.
The smart HTTP protocol of git is implemented in git-http-backend. This is a CGI binary that can be used in combination with some web-servers. Unfortunately it ignores the content-length of the request and relies on the server to close its input, which makes it incompatible with some CGI servers.
If you have python available, an easy way to run it would be using the CGIHTTPServer. This one is affected by the mentioned incompatibility. With a rather ugly workaround you can nevertheless make it work:
$ python -c 'import CGIHTTPServer; CGIHTTPServer.CGIHTTPRequestHandler.have_fork = False; CGIHTTPServer.test()' 8000
Or with python 3:
$ python3 -c 'import http.server; http.server.CGIHTTPRequestHandler.have_fork = False; http.server.test(HandlerClass=http.server.CGIHTTPRequestHandler, port=8000)'
These set the internal have_fork of CGIHTTPRequestHandler to False which causes the implementation to use subprocesses and pipes instead of fork to run the CGI binary. With this method, the incoming requests are buffered and written to a pipe, which is then closed. This fits the expected model of git-http-backend and therefore makes it work.
If you are on a system without fork in the first place (i.e. Windows) you don't need the workaround and can directly use:
$ python -m CGIHTTPServer 8000
$ python3 -m http.server --cgi 8000
Both setups require that git-http-backend is made available under a cgi-bin directory. The easiest setup is to create the cgi-bin directory inside a bare git repository, symlink or copy the git-http-backend binary into it and start the server from the bare repository:
$ git clone --bare <repo> bare.git
$ cd bare.git
$ touch git-daemon-export-ok
$ mkdir cgi-bin
$ ln -s /usr/lib/git-core/git-http-backend cgi-bin/git
$ python -c 'import CGIHTTPServer; CGIHTTPServer.CGIHTTPRequestHandler.have_fork = False; CGIHTTPServer.test()' 8000
With that setup, the repository is served at http://localhost:8000/cgi-bin/git because git-http-backend was symlinked to cgi-bin/git. So the following should then work:
$ git clone http://localhost:8000/cgi-bin/git clonedRepoName
To allow pushing into the bare repository use:
$ git config http.receivepack true
More advanced setups, with more than one repository served, can be made using the appropriate environment variables. Since the environment is inherited by the server and passed to the CGI binary, simply exporting the desired environment variables should work. See the git-http-backend documentation for all possible configuration and environment variables.
Based on Gustave's suggestion, I've discovered SimpleHTTPServer within JGit. Works like a charm. Thank you!
I want to deploy a a project to ec2. It was modify in Intellij which used MAVEN. It work perfect in Intellij. Below is the configuration of Intellij.
I have tried command line like java, javac. These kind of comment lines are all failed. I am wondering there must some way to convert Intellij configuration to command lines. I was relying too much on IDE. XD. Now I am regreted.
EC2 is the amazon cloud service for instances. If you aren't very experienced with it - the easiest you can do is to hire a machine with your fav operating system - for example Windows and use remote desktop to log in it and just do your normal IDE setup and run your app. You will have your app running on your instance and it will be working, even it is done the hard way. I will get downvotes for that, but it will work as a Swiss watch.
It depends from the kind of EC2 instance you're using.
For me would be simpler if it was a Linux instance.
I'll connect via ssh and git clone my project there from my bitbucket/github repository.
Then I'll use Maven, and there are many different options to run your project with Maven.
As suggested in the comments you could use the exec plugin.
But even in this way you'll have a list of problems to overcame.
For example your linux instance does not came with an preinstalled Java Runtime.
You have to install at least Java and Maven (choose ubuntu distro, so you can do easily with a package manager like apt-get).
Or, for example, another not easy task is connect via ssh to a Linux instance.
AWS generates a key.pem file that you have to use in order to successfully connect to your instance.
ssh -i .ssh/your-key-file.pem username#your-ec2-instance-address
And again, AWS does not tell you the name of the default user you must use to connect to your EC2 instance (if you choose ubuntu distro the username is ubuntu).
And again, when you save your-key-file.pem in your computer it must have the right permission
-rw-------# 1 freedev staff 1692 Apr 21 09:46 /Users/freedev/.ssh/your-key-file.pem
or your ssh client wont read it.
...looking back it was really a long way make a deploy on a EC2 instance.
I have solved it by myself.
Use mvn package to generated jar file for maven project.
nohup java xx.jar debug.xml & use this to run jar in shell. By using this command line, when exit (abort remote connection), the process / command will not get killed.
Command line jcmd is used to check running process pid.
Nexus Repository Manager OSS can be used as caching proxy for jar artifacts, and as new feature for docker images.
I'd like to set up it for company usage within LAN. And want to use docker to install it (I could find docker image for nexus https://github.com/sonatype/docker-nexus3).
How can I point Nexus to use docker images on the server where it is installed, as Nexus OSS is actually Java application.
Note that we should be cautious about disk usages, possibly there's no way yet to clean-up registry.
To accomplish something like this, you'll likely want to setup https for Nexus Repository. This Docker image is a good starting point for that: https://hub.docker.com/r/bradbeck/nexus-https/
You'll also want to expose ports for any proxy, hosted or group repositories you plan on setting up. Follow instructions here: https://books.sonatype.com/nexus-book/3.2/reference/docker.html#docker-proxy for setting up a proxy.
Once you've setup the repositories you want, you will also need to expose these ports via Docker. You'll want to run a command similar to the following:
docker run -d -p 8081:8081 -p 8443:8443 -p 8444:8444 -v ~/nexus-data:/nexus-data -v ~/nexus-ssl:/opt/sonatype/nexus/etc/ssl --name nexus bradbeck/nexus-https
You may also need to modify the Dockerfile itself to expose the ports you want to use (I've used 8444 in this case). You'll do this modification here: https://github.com/bradbeck/nexus-https/blob/master/Dockerfile#L56
As well, a community member sent us a PR a while ago that setup a default Docker registry: https://github.com/sonatype/docker-nexus3/pull/48/files . We did not merge it as the changes are a bit broad (not everyone needs one setup by default), but perhaps it will help you on your quest :)