I'm trying to upload an image to a existing bucket in my Google Cloud Storage.
The image file gets uploaded successfully when I go and check, but the returned download url is null
CODE
private String uploadImage(File filePath, String blobName, File uploadCreds) throws FileNotFoundException, IOException{
Storage storage = StorageOptions.newBuilder().setProjectId("myProjectId")
.setCredentials(ServiceAccountCredentials.fromStream(new FileInputStream(uploadCreds)))
.build()
.getService();
String bucketName = "myBucketName";
Bucket bucket = storage.get(bucketName);
BlobId blobId = BlobId.of(bucket.getName(), blobName);
InputStream inputStream = new FileInputStream(filePath);
BlobInfo blobInfo = BlobInfo.newBuilder(blobId).setContentType("image/jpeg").build();
try (WriteChannel writer = storage.writer(blobInfo)) {
byte[] buffer = new byte[1024];
int limit;
try {
while ((limit = inputStream.read(buffer)) >= 0) {
writer.write(ByteBuffer.wrap(buffer, 0, limit));
}
} catch (Exception ex) {
ex.printStackTrace();
}finally {
writer.close();
}
System.out.println("Image URL : " + blobInfo.getMediaLink());
System.out.println("Blob URL : " + blobInfo.getSelfLink());
return blobInfo.getMediaLink();
}
}
filePath is the Image File
blobName is a random Image Name
uploadCreds is my credintials.json file
Why is the blobInfo.getMediaLink() and blobInfo.getSelfLink() returning null? What am i doing wrong?
Here is my code that works perfectly
#RestController
#RequestMapping("/api")
public class CloudStorageHelper {
Credentials credentials = GoogleCredentials.fromStream(new FileInputStream("C:\\Users\\sachinthah\\Downloads\\MCQ project -1f959c1fc3a4.json"));
Storage storage = StorageOptions.newBuilder().setCredentials(credentials).build().getService();
public CloudStorageHelper() throws IOException {
}
#SuppressWarnings("deprecation")
#RequestMapping(method = RequestMethod.POST, value = "/imageUpload112")
public String uploadFile(#RequestParam("fileseee")MultipartFile fileStream)
throws IOException, ServletException {
BlobstoreService blobstoreService = BlobstoreServiceFactory.getBlobstoreService();
String bucketName = "mcqimages";
checkFileExtension(fileStream.getName());
DateTimeFormatter dtf = DateTimeFormat.forPattern("-YYYY-MM-dd-HHmmssSSS");
DateTime dt = DateTime.now(DateTimeZone.UTC);
String fileName = fileStream.getOriginalFilename();
BlobInfo blobInfo = BlobInfo.newBuilder(bucketName, fileName)
.setAcl(new ArrayList<>(Arrays.asList(Acl.of(User.ofAllUsers(), Role.READER))))
.build(),
fileStream.getInputStream());
System.out.println(blobInfo.getMediaLink());
// sachintha added a comma after the link to identify the link that get generated
return blobInfo.getMediaLink() + ",";
}
private void checkFileExtension(String fileName) throws ServletException {
if (fileName != null && !fileName.isEmpty() && fileName.contains(".")) {
String[] allowedExt = {".jpg", ".jpeg", ".png", ".gif"};
for (String ext : allowedExt) {
if (fileName.endsWith(ext)) {
return;
}
}
throw new ServletException("file must be an image");
}
}
The Answer was quite simple, i just got rid of the manual upload method and used the inbuilt create blob.
private String uploadImage(File filePath, String blobName, File uploadCreds) throws FileNotFoundException, IOException{
Storage storage = StorageOptions.newBuilder().setProjectId("porjectId")
.setCredentials(ServiceAccountCredentials.fromStream(new FileInputStream(uploadCreds)))
.build()
.getService();
String bucketName = "bucketName";
Bucket bucket = storage.get(bucketName);
BlobId blobId = BlobId.of(bucket.getName(), blobName);
InputStream inputStream = new FileInputStream(filePath);
BlobInfo blobInfo = BlobInfo.newBuilder(blobId).setContentType("image/jpeg").build();
Blob blob = storage.create(blobInfo, inputStream);
System.out.println("Image URL : " + blob.getMediaLink());
return blob.getMediaLink();
}
In case you want to store it in a special folder get the blobinfo.name() and append the "/" to it. for e.g if temp.jpg need to be stored with date folders. get the date from date object and format it with date formatter and prepend it
blobinfo.name() = date+"/"+blobinfo.name();
will classify all images date wise..
Related
I am going to convert MultipartFile to File and upload it to S3 bucket.
However, when running the test, an error occurs in the process of converting MultipartFile to File.
ERROR : java.nio.file.AccessDeniedException: D:\workspace_intellij_forKiri\Kiri\server\kiri\temp\8b28a2f2-7276-4036
multipartFile.transferTo(file);
Please advise if there is anything I am missing.
The spring boot version is 2.7.7 version.
Test code
#WithAccount("creamyyyy")
#DisplayName("image save test")
#Test
public void createImageTest() throws Exception {
//given
String filename = "files";
String contentType = "png";
MockMultipartFile image1 = new MockMultipartFile(
filename,
filename + "." + contentType,
"image/png",
filename.getBytes());
//when
//then
this.mockMvc.perform( //== ERROR!!!
MockMvcRequestBuilders
.multipart("/api/posts/image")
.file(image1)
.contentType(MediaType.MULTIPART_FORM_DATA)
.characterEncoding("UTF-8")
)
.andDo(print())
.andExpect(status().isOk());
}
ImageService Code
// FileSave
public List<ImageResDto> addFile(List<MultipartFile> multipartFiles) throws IOException {
List<ImageResDto> imageResDtoList = new ArrayList<>();
/**
* <ImageResDto>
* private Long image_id;
* private String imgUrl;
*/
String absolutePath = new File("").getAbsolutePath() + File.separator + "temp";
for (MultipartFile multipartFile : multipartFiles) {
String contentType = multipartFile.getContentType();
if(ObjectUtils.isEmpty(contentType)) {
throw new RuntimeException("FILE TYPE NOT FOUND");
} else if(!verifyContentType(contentType)){
throw new RuntimeException("FILE TYPE NOT FOUND");
}
}
for (MultipartFile multipartFile : multipartFiles) {
String filename = UUID.randomUUID() + multipartFile.getOriginalFilename();
// save in local
String fullFilePath = absolutePath + File.separator + filename;
System.out.println("fullFilePath = " + fullFilePath);
File file = new File(fullFilePath);
if(!file.exists()) { file.mkdirs(); }
multipartFile.transferTo(file); // ERROR ... OTL
file.createNewFile();
// S3 upload
amazonS3.putObject(
new PutObjectRequest(bucket, filename, file)
.withCannedAcl(CannedAccessControlList.PublicRead)
);
String imgUrl = amazonS3.getUrl(bucket, filename).toString();
Image newImage = Image.builder()
.filename(filename)
.filepath(filename)
.imgUrl(imgUrl)
.build();
imageRepository.save(newImage);
ImageResDto imageResDto = ImageResDto.of(newImage);
imageResDtoList.add(imageResDto);
file.delete(); // local file delete
}
return imageResDtoList;
}
ImageController Code
#PostMapping(value = "/api/posts/image", consumes = {MediaType.MULTIPART_FORM_DATA_VALUE, MediaType.APPLICATION_JSON_VALUE})
public ResponseEntity createImage(#RequestPart(value = "files") List<MultipartFile> multipartFiles) throws IOException {
System.out.println("ImageController Runnnn");
// get member
PrincipalDetails principalDetails = (PrincipalDetails) SecurityContextHolder.getContext().getAuthentication().getPrincipal();
Member member = principalDetails.getMember();
List<ImageResDto> imageResDtoList = imageService.addFile(multipartFiles);
return new ResponseEntity(imageResDtoList, HttpStatus.CREATED);
}
I tried to specify a separate route using Path, but I failed.
// Error ..java.nio.file.AccessDeniedException => Path
// multipartFile -> File
Path path = Paths.get(fullFilePath).toAbsolutePath();
multipartFile.transferTo(path.toFile());
Files.createFile(path);
What is incomprehensible is that when tested using PostMan, the file is normally uploaded to the S3 bucket.
Please tell me if I applied anything wrong.
I am trying to upload a huge video file to the s3 bucket.
I am getting the data from the client-side in base64 format which I am sending to the S3client to upload to my bucket as shown below:-
public class UploadFileService {
private static final String BUCKET_NAME = "data";
private static final Regions REGION = Regions.US_EAST_2;
LoggerUtils loggerUtils = new LoggerUtils ();
public String uploadFile(String fileData, String fileName, String contentType, String extension){
try {
loggerUtils.log ("File Data" , fileData);
byte[] bI = org.apache.commons.codec.binary.Base64.decodeBase64 ((fileData.substring (fileData.indexOf (",") + 1)).getBytes ());
InputStream fis = new ByteArrayInputStream (bI);
AmazonS3 s3 = new AmazonS3Client ();
Region usWest02 = Region.getRegion (REGION);
s3.setRegion (usWest02);
ObjectMetadata metadata = new ObjectMetadata ();
metadata.setContentLength (bI.length);
metadata.setContentType ("video/mp4");
//metadata.setContentType (contentType + "/" + extension.substring (1));
metadata.setCacheControl ("public, max-age=0");
s3.putObject (BUCKET_NAME, fileName, fis, metadata);
s3.setObjectAcl (BUCKET_NAME, fileName, CannedAccessControlList.PublicRead);
URL s3Url = s3.getUrl(BUCKET_NAME, fileName);
return s3Url.toExternalForm();
}
catch (Exception exception){
loggerUtils.log (exception.toString ());
throw exception;
}
}
public static void main(String[] args) {
String fileName = "abc1.mp4";
String fileData = "hkbk";
new UploadFileService ().uploadFile (fileData, fileName, null, null);
}
}
But if the fileData is usage (base64 of a 2MB video) then I am getting the below error:-
Error:(46, 27) java: constant string too long
I have written a piece of code, to upload a file on Amazon S3. It is working fine in my local system. I am able to upload a file and as a response I am getting the file url. But when I try to upload those files to AWS server, they are not getting uploaded, I am getting a 200 response as well, and its showing a rectangle box on postman.
Can anyone help me to solve this issue? Any help will be appreciated. Thanks!
#Service
public class AmazonClient {
private AmazonS3 amazonS3;
#Value("${amazonProperties.accessKey}")
public String accessKey;
#Value("${amazonProperties.secretKey}")
public String secretKey;
#Value("${amazonProperties.bucketName}")
public String bucketName;
#Value("${amazonProperties.endpointUrl}")
public String endpointUrl;
#Value("${amazonProperties.region}")
public String region;
#PostConstruct
private void initializeAmazon()
{
AWSCredentials credentials = new BasicAWSCredentials(accessKey, secretKey);
this.amazonS3 = AmazonS3ClientBuilder
.standard()
.withCredentials(new AWSStaticCredentialsProvider(credentials))
.withRegion(region)
.build();
}
private File convertMultiPartToFile(MultipartFile file) throws IOException
{
File convFile = new File(file.getOriginalFilename());
FileOutputStream fos = new FileOutputStream(convFile);
fos.write(file.getBytes());
fos.close();
return convFile;
}
private String generateFileName(MultipartFile multiPart)
{
return new Date().getTime() + "-" + multiPart.getOriginalFilename().replace(" ", "_");
}
private void uploadFileTos3bucket(String fileName, File file)
{
amazonS3.putObject(new PutObjectRequest(bucketName, fileName, file)
.withCannedAcl(CannedAccessControlList.PublicRead));
}
public String uploadFile(MultipartFile multipartFile) {
String fileUrl = "";
try
{
File file = convertMultiPartToFile(multipartFile);
String fileName = generateFileName(multipartFile);
fileUrl = endpointUrl + "/" + bucketName + "/" + fileName;
uploadFileTos3bucket(fileName, file);
//file.delete();
}
catch (Exception e)
{
e.printStackTrace();
}
return fileUrl;
}
public String deleteFileFromS3Bucket(String fileUrl)
{
String fileName = fileUrl.substring(fileUrl.lastIndexOf("/") + 1);
amazonS3.deleteObject(new DeleteObjectRequest(bucketName + "/", fileName));
return "Successfully deleted";
}
}
Please ensure that the AWS IAM credentials provided are working and the bucket location, as well as the name, are provided correctly. If it is then:-
i. Make sure that the bucket is accessible programmatically.
ii. Make sure that the IAM role has privileges enough to interact with S3 bucket eg(Full S3 access policy).
These might be probable resolutions.
I want to upload images to google cloud storage and here is my code from spring boot. But the problem is this isnt working at all giving me error like this:
2018-10-22 15:22:55.628 ERROR 6172 --- [nio-8080-exec-6] o.a.c.c.C.[.[.[/].[dispatcherServlet] : Servlet.service() for servlet [dispatcherServlet] in context with path [] threw exception [Request processing failed; nested exception is java.lang.IllegalArgumentException: Invoked method public abstract java.io.InputStream org.apache.commons.fileupload.FileItemStream.openStream() throws java.io.IOException is no accessor method!] with root cause
please help me. following is the code i wrote
private static Storage storage = null;
// [START init]
static {
storage = StorageOptions.getDefaultInstance().getService();
}
#SuppressWarnings("deprecation")
#RequestMapping(method = RequestMethod.POST, value = "/imageUpload")
public String uploadFile(FileItemStream fileStream)
throws IOException, ServletException {
String bucketName = "mcqimages";
checkFileExtension(fileStream.getName());
DateTimeFormatter dtf = DateTimeFormat.forPattern("-YYYY-MM-dd-HHmmssSSS");
DateTime dt = DateTime.now(DateTimeZone.UTC);
String dtString = dt.toString(dtf);
final String fileName = fileStream.getName() + dtString;
BlobInfo blobInfo =
storage.create(
BlobInfo
.newBuilder(bucketName, fileName)
.setAcl(new ArrayList<>(Arrays.asList(Acl.of(User.ofAllUsers(), Role.READER))))
.build(),
fileStream.openStream());
return blobInfo.getMediaLink();
}
private void checkFileExtension(String fileName) throws ServletException {
if (fileName != null && !fileName.isEmpty() && fileName.contains(".")) {
String[] allowedExt = {".jpg", ".jpeg", ".png", ".gif"};
for (String ext : allowedExt) {
if (fileName.endsWith(ext)) {
return;
}
}
throw new ServletException("file must be an image");
}
}
finally i came up with this code :). worked very well. need the Credentials to upload files into GCP storage. you can generate the credentials from JSON format too.
https://cloud.google.com/docs/authentication/production
Credentials credentials = GoogleCredentials.fromStream(new FileInputStream("C:\\Users\\sachinthah\\Downloads\\MCQ project -1f959c1fc3a4.json"));
Storage storage = StorageOptions.newBuilder().setCredentials(credentials).build().getService();
public CloudStorageHelper() throws IOException {
}
#SuppressWarnings("deprecation")
#RequestMapping(method = RequestMethod.POST, value = "/imageUpload112")
public String uploadFile(#RequestParam("fileseee")MultipartFile fileStream)
throws IOException, ServletException {
String bucketName = "mcqimages";
checkFileExtension(fileStream.getName());
DateTimeFormatter dtf = DateTimeFormat.forPattern("-YYYY-MM-dd-HHmmssSSS");
DateTime dt = DateTime.now(DateTimeZone.UTC);
String dtString = dt.toString(dtf);
final String fileName = fileStream.getName() + dtString;
File file = convertMultiPartToFile( fileStream );
BlobInfo blobInfo =
storage.create(
BlobInfo
.newBuilder(bucketName, fileName)
.setAcl(new ArrayList<>(Arrays.asList(Acl.of(User.ofAllUsers(), Role.READER))))
.build()
// file.openStream()
);
System.out.println(blobInfo.getMediaLink());
return blobInfo.getMediaLink();
}
private File convertMultiPartToFile(MultipartFile file ) throws IOException
{
File convFile = new File( file.getOriginalFilename() );
FileOutputStream fos = new FileOutputStream( convFile );
fos.write( file.getBytes() );
fos.close();
return convFile;
}
private void checkFileExtension(String fileName) throws ServletException {
if (fileName != null && !fileName.isEmpty() && fileName.contains(".")) {
String[] allowedExt = {".jpg", ".jpeg", ".png", ".gif"};
for (String ext : allowedExt) {
if (fileName.endsWith(ext)) {
return;
}
}
throw new ServletException("file must be an image");
}
}
I would try to upload the file instead:
public String uploadFile(#RequestParam("file") MultipartFile file) {
if (file.isEmpty()) {
//Set error message
}
else {
try {
String extension = FilenameUtils.getExtension(file.getOriginalFilename()); //Commons IO
// Get the file
byte[] bytes = file.getBytes();
....
}
A good example of uploading files is here: https://www.baeldung.com/spring-file-upload
Credentials credentials = GoogleCredentials.fromStream(new ClassPathResource("key.json").getInputStream());
Storage storage=StorageOptions.newBuilder().setCredentials(credentials).build().getService();
BlobId blobId = BlobId.of(bucketName, yourobjectName);
BlobInfo blobInfo = BlobInfo.newBuilder(blobId).build();
storage.create(blobInfo,Files.readAllBytes(Paths.get(filePath)));
I need to export/download all files of the other domain users. I used the client login with administer account to see the all files of domain users. however,only document can be export/download,others are fail.
so what is the download url format of the others(For File,pdf,presentation and spreadsheet)??
my document download url is
https://docs.google.com/feeds/download/documents/Export?xoauth_requestor=admin#domain.com&docId=<id>&exportFormat=doc
my program is as following:
public class AuthExample {
private static DocsService docService = new DocsService("Auth Example");
public static void main(String[] args)
throws Exception
{
String adminUser = admin;
String adminPassword = adminpasswd;
String impersonatedUser = "user#domain.com";
docService.setUserCredentials(adminUser, adminPassword);
URL url = new URL( "https://docs.google.com/feeds/" + impersonatedUser + "/private/full");
DocumentListFeed feed = docService.getFeed(url, DocumentListFeed.class);
for (DocumentListEntry entry : feed.getEntries()) {
String title = entry.getTitle().getPlainText();
System.out.println( title );
String type = entry.getType();
if ( type.equals("document") )
{
String encodedAdminUser = URLEncoder.encode(adminUser);
String resourceId = entry.getResourceId();
String resourceIdNoPrefix = resourceId.substring( resourceId.indexOf(':')+1 );
String downloadUrl =
"https://docs.google.com/feeds/download/documents/Export" +
"?xoauth_requestor=" + encodedAdminUser +
"&docId=" + resourceIdNoPrefix +
"&exportFormat=doc";
downloadFile( downloadUrl, title + ".doc" );
}
}
}
// Method pasted directly from Google documentation
public static void downloadFile(String exportUrl, String filepath)
throws IOException, MalformedURLException, ServiceException
{
System.out.println("Exporting document from: " + exportUrl);
MediaContent mc = new MediaContent();
mc.setUri(exportUrl);
MediaSource ms = docService.getMedia(mc);
InputStream inStream = null;
FileOutputStream outStream = null;
try {
inStream = ms.getInputStream();
outStream = new FileOutputStream(filepath);
int c;
while ((c = inStream.read()) != -1) {
outStream.write(c);
}
} finally {
if (inStream != null) {
inStream.close();
}
if (outStream != null) {
outStream.flush();
outStream.close();
}
}
}
}
Don't build the download link manually, instead use the entry's content link as explained in the docs:
https://developers.google.com/google-apps/documents-list/#downloading_documents_and_files