We Keep Coding

How to write files in different directories based on different spring-profiles?

I generate a Zip File in my code and now i want to weite it in a directory
/export
in Production then I want to write it in
/foo/bar
I already got 2 Profiles (it & Production):
application-it.yaml
application-production.yaml
But how do I manage two save locations in my code now or what propertie should I use for that?

In Spring Boot you can do it like this.
'application-it.yml' content:
file.storage: "/export"
'application-production.yml' content:
file.storage: "/foo/bar"
I don't know your real implementation. But imagine that you use service to process the file. Then you can include #Value to your service to include specific variable.
My 'SimpleService.java' content:
package test.service;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
import org.springframework.beans.factory.annotation.Value;
import org.springframework.stereotype.Service;
import java.io.File;
import java.io.FileOutputStream;
import java.io.IOException;
import java.util.zip.ZipEntry;
import java.util.zip.ZipOutputStream;
#Service
public class SimpleService {
private static final Logger LOG = LoggerFactory.getLogger(SimpleService.class);
#Value("${file.storage}")
private String fileStorage;
public void saveFile() throws IOException {
LOG.info("Folder to store files: {}", fileStorage);
File file = new File(fileStorage + File.separatorChar + "test.zip");
ZipOutputStream outputStream = new ZipOutputStream(new FileOutputStream(file));
ZipEntry entry = new ZipEntry("entry.txt");
outputStream.putNextEntry(entry);
outputStream.write("test".getBytes());
outputStream.closeEntry();
outputStream.close();
}
}
And finally, when you start your app (in my case with Maven) specify active app profile:
-Dspring.profiles.active=it
or
-Dspring.profiles.active=production

Getting java.nio.file.AccessDeniedException , when trying to create zip file using java NIO 2 on MAC

import java.io.IOException;
import java.net.URI;
import java.nio.file.FileSystem;
import java.nio.file.FileSystems;
import java.util.HashMap;
import java.util.Map;
public class ZipFileSystem {
public static void main(String[] args) throws IOException {
URI uri = URI.create("jar:file:///sample.zip");
Map<String,String> options = new HashMap<>();
options.put("create","true");
FileSystem fileSystem = FileSystems.newFileSystem(uri, options);
}
}
I have this java simple code , when i try to run this on MAC, getting below exception, am I missing anything?
Exception in thread "main" java.nio.file.AccessDeniedException: /sample.zip
at sun.nio.fs.UnixException.translateToIOException(UnixException.java:84)
at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:102)
at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:107)
at sun.nio.fs.UnixFileSystemProvider.newByteChannel(UnixFileSystemProvider.java:214)
at java.nio.file.spi.FileSystemProvider.newOutputStream(FileSystemProvider.java:434)
at java.nio.file.Files.newOutputStream(Files.java:216)
at com.sun.nio.zipfs.ZipFileSystem.<init>(ZipFileSystem.java:116)
at com.sun.nio.zipfs.ZipFileSystemProvider.newFileSystem(ZipFileSystemProvider.java:117)
at java.nio.file.FileSystems.newFileSystem(FileSystems.java:326)
at java.nio.file.FileSystems.newFileSystem(FileSystems.java:276)
at ZipFileSystem.main(ZipFileSystem.java:14)

Typo in word "hdfs" gives me: "java.io.IOException: No FileSystem for scheme: hdfs". Using FileSystem lib over hadoop 2.7.7

While using FileSystem.get(URI.create("hdfs://localhost:9000/"), configuration) I'm getting the error "Typo in word hdfs", when I tried to run the code gives me the IOException:
java.io.IOException: No FileSystem for scheme: hdfs
at org.apache.hadoop.fs.FileSystem.getFileSystemClass(FileSystem.java:2658)
at org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:2665)
at org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:93)
at org.apache.hadoop.fs.FileSystem$Cache.getInternal(FileSystem.java:2701)
at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:2683)
at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:372)
at com.oracle.hadoop.client.Test.main(Test.java:53)
I already tried to use in different ways to use the call to hdfs, I'm using the libraries for hadoop 2.7.7
Here is my current code:
import org.apache.hadoop.conf.Configuration;
import org.apache.hadoop.fs.*;
import org.apache.hadoop.fs.Path;
import org.apache.hadoop.io.IOUtils;
import org.apache.log4j.BasicConfigurator;
import java.io.IOException;
import java.io.InputStream;
import java.net.URI;
public class Test {
public static void main(String []args) {
Configuration conf = new Configuration();
InputStream in = null;
try {
FileSystem fs = FileSystem.get(URI.create("hdfs://localhost:9000/"), conf);
in = fs.open(new Path(uri));
IOUtils.copyBytes(in, System.out, 4096, false);
} catch (IOException e) {
e.printStackTrace();
} finally {
IOUtils.closeStream(in);
}
}

Actually, I just added this maven dependency: http://mvnrepository.com/artifact/org.apache.hadoop/hadoop-hdfs/2.7.7
to maven pom.xml and problem solved.

How to send an image with JDA

im developing my Java bot for discord. And I want to send an image. I tried using TextChannel.sendFile(File, Message), but it`s not that result that I want to get. I want this file to be displayed like a normal image.
The imports:
import java.io.File;
import java.io.IOException;
import java.net.HttpURLConnection;
import java.net.URL;
import java.util.ArrayList;
import java.util.List;
import java.util.concurrent.ThreadLocalRandom;
import javax.xml.namespace.QName;
import javax.xml.stream.FactoryConfigurationError;
import javax.xml.stream.XMLEventReader;
import javax.xml.stream.XMLInputFactory;
import javax.xml.stream.XMLStreamException;
import javax.xml.stream.events.Attribute;
import javax.xml.stream.events.StartElement;
import javax.xml.stream.events.XMLEvent;
import org.apache.commons.io.FileUtils;
import net.dv8tion.jda.core.MessageBuilder;
import net.dv8tion.jda.core.entities.Message;
import net.dv8tion.jda.core.entities.TextChannel;
import net.dv8tion.jda.core.events.message.MessageReceivedEvent;
And the other code:
URL url = new URL(s.toString());
HttpURLConnection conn = (HttpURLConnection) url.openConnection();
XMLEventReader reader = XMLInputFactory.newInstance().createXMLEventReader(conn.getInputStream());
final List<String> files = new ArrayList<>();
while (reader.hasNext()) {
XMLEvent e = reader.nextEvent();
if (e.isStartElement()) {
StartElement se = e.asStartElement();
if (se.getName().getLocalPart().equals("post")) {
Attribute purl = se.getAttributeByName(new QName("file_url"));
files.add(purl.getValue());
}
}
}
int rid = ThreadLocalRandom.current().nextInt(files.size() - 1);
String p = files.get(rid);
files.clear();
URL u = new URL(p);
final String[] dots = p.split("\\.");
final String format = dots[dots.length - 1];
File f = new File("its not a porn." + format);
FileUtils.copyURLToFile(url, f);
Message m = new MessageBuilder().append("okay :)").build();
c.sendFile(f, m).queue();
}
I tried to find a solution somewhere but i haven found any info that could help.

At JDA 4.2.0_168
the message on sendFile() is the name of the file that you are sending to the discord servers, so it needs an extension
example:
File f = new File("image.png");
TextChannel.sendFile(f, "image.png").queue();
if you want comments in the message
File f = new File("image.png");
//the name doesn't need to be the same, just the same extension
TextChannel.sendFile(f, "another_name.png").append("okay :)").queue();
Result of last code

Reading through the docs, you need to create a MessageEmbed and add it to the message using m.setEmbed(..)

Java - A master URL must be set in your configuration

I am trying to run some algorithm in apache Spark. I am getting
Java - A master URL must be set in your configuration error even if I set the configuration.
SparkSession spark = SparkSession.builder().appName("Sp_LogistcRegression").config("spark.master", "local").getOrCreate();
This is the code I work with
import java.io.FileOutputStream;
import java.io.IOException;
import java.io.ObjectOutputStream;
import org.apache.spark.SparkConf;
import org.apache.spark.api.java.JavaSparkContext;
import org.apache.spark.ml.classification.LogisticRegression;
import org.apache.spark.ml.classification.LogisticRegressionModel;
import org.apache.spark.sql.Dataset;
import org.apache.spark.sql.Row;
import org.apache.spark.sql.SparkSession;
import org.apache.spark.mllib.util.MLUtils;
public class Sp_LogistcRegression {
public void trainLogisticregression(String path, String model_path) throws IOException {
//SparkConf conf = new SparkConf().setAppName("Linear Regression Example");
// JavaSparkContext sc = new JavaSparkContext(conf);
SparkSession spark = SparkSession.builder().appName("Sp_LogistcRegression").config("spark.master", "local").getOrCreate();
Dataset<Row> training = spark.read().option("header","true").csv(path);
System.out.print(training.count());
LogisticRegression lr = new LogisticRegression().setMaxIter(10).setRegParam(0.3);
// Fit the model
LogisticRegressionModel lrModel = lr.fit(training);
lrModel.save(model_path);
spark.close();
}
}
This is my test case:
import java.io.File;
import org.junit.Test;
public class Sp_LogistcRegressionTest {
Sp_LogistcRegression spl =new Sp_LogistcRegression ();
#Test
public void test() {
String filename = "datas/seg-large.csv";
ClassLoader classLoader = getClass().getClassLoader();
File file1 = new File(classLoader.getResource(filename).getFile());
spl. trainLogisticregression( file1.getAbsolutePath(), "/tmp");
}
}
Why I am getting this error? I checked the solutions here
Spark - Error "A master URL must be set in your configuration" when submitting an app
It does n´t work.
Any clues ?

your
SparkSession spark = SparkSession.builder().appName("Sp_LogistcRegression").config("spark.master", "local").getOrCreate();
should be
SparkSession spark = SparkSession.builder().appName("Sp_LogistcRegression").master("local").getOrCreate();
Or
when you run, you need to
spark-submit --class mainClass --master local yourJarFile