We Keep Coding

Trying to add substrings from newLines in a large file to a list

I downloaded my extended listening history from Spotify and I am trying to make a program to turn the data into a list of artists without doubles I can easily make sense of. The file is rather huge because it has data on every stream I have done since 2016 (307790 lines of text in total). This is what 2 lines of the file looks like:
{"ts":"2016-10-30T18:12:51Z","username":"edgymemes69endmylifepls","platform":"Android OS 6.0.1 API 23 (HTC, 2PQ93)","ms_played":0,"conn_country":"US","ip_addr_decrypted":"68.199.250.233","user_agent_decrypted":"unknown","master_metadata_track_name":"Devil's Daughter (Holy War)","master_metadata_album_artist_name":"Ozzy Osbourne","master_metadata_album_album_name":"No Rest for the Wicked (Expanded Edition)","spotify_track_uri":"spotify:track:0pieqCWDpThDCd7gSkzx9w","episode_name":null,"episode_show_name":null,"spotify_episode_uri":null,"reason_start":"fwdbtn","reason_end":"fwdbtn","shuffle":true,"skipped":null,"offline":false,"offline_timestamp":0,"incognito_mode":false},
{"ts":"2021-03-26T18:15:15Z","username":"edgymemes69endmylifepls","platform":"Android OS 11 API 30 (samsung, SM-F700U1)","ms_played":254120,"conn_country":"US","ip_addr_decrypted":"67.82.66.3","user_agent_decrypted":"unknown","master_metadata_track_name":"Opportunist","master_metadata_album_artist_name":"Sworn In","master_metadata_album_album_name":"Start/End","spotify_track_uri":"spotify:track:3tA4jL0JFwFZRK9Q1WcfSZ","episode_name":null,"episode_show_name":null,"spotify_episode_uri":null,"reason_start":"fwdbtn","reason_end":"trackdone","shuffle":true,"skipped":null,"offline":false,"offline_timestamp":1616782259928,"incognito_mode":false},
It is formatted in the actual text file so that each stream is on its own line. NetBeans is telling me the exception is happening at line 19 and it only fails when I am looking for a substring bounded by the indexOf function. My code is below. I have no idea why this isn't working, any ideas?
import java.util.*;
public class MainClass {
public static void main(String args[]){
File dat = new File("SpotifyListeningData.txt");
List<String> list = new ArrayList<String>();
Scanner swag = null;
try {
swag = new Scanner(dat);
}
catch(Exception e) {
System.out.println("pranked");
}
while (swag.hasNextLine())
if (swag.nextLine().length() > 1)
if (list.contains(swag.nextLine().substring(swag.nextLine().indexOf("artist_name"), swag.nextLine().indexOf("master_metadata_album_album"))))
System.out.print("");
else
try {list.add(swag.nextLine().substring(swag.nextLine().indexOf("artist_name"), swag.nextLine().indexOf("master_metadata_album_album")));}
catch(Exception e) {}
System.out.println(list);
}
}

Find a JSON parser you like.
Create a class that with the fields you care about marked up to the parsers specs.
Read the file into a collection of objects. Most parsers will stream the contents so you're not string a massive string.
You can then load the data into objects and store that as you see fit. For your purposes, a TreeSet is probably what you want.

Your code will throw a lot of exceptions only because you don't use braces. Please do use braces in each blocks, whether it is if, else, loops, whatever. It's a good practice and prevent unnecessary bugs.
However, everytime scanner.nextLine() is called, it reads the next line from the file, so you need to avoid using that in this way.
The best way to deal with this is to write a class containing the fields same as the json in each line of the file. And map the json to the class and get desired field value from that.
Your way is too much risky and dependent on structure of the data, even on whitespaces. However, I fixed some lines in your code and this will work for your purpose, although I actually don't prefer operating string in this way.
while (swag.hasNextLine()) {
String swagNextLine = swag.nextLine();
if (swagNextLine.length() > 1) {
String toBeAdded = swagNextLine.substring(swagNextLine.indexOf("artist_name") + "artist_name".length() + 2
, swagNextLine.indexOf("master_metadata_album_album") - 2);
if (list.contains(toBeAdded)) {
System.out.print("Match");
} else {
try {
list.add(toBeAdded);
} catch (Exception e) {
System.out.println("Add to list failed");
}
}
System.out.println(list);
}
}

Cannot add new object to a set, values come from a file

I am trying to create a public instance method that takes no arguments and returns no values. It is required to get an input from a user to select a file, this part I have no issues with. The method needs to make use of the BufferReader and Scanner Objects. So that it can read the file selected. For each line that is read, a new object should be created and its instance variables set using the values found in the file.
That object that is created should then be added to a list. This is where I am having issues, it won't let me add the new object to the list. Below is my code:
public void readInEntrants()
{
String pathname = OUFileChooser.getFilename();
File aFile = new File(pathname);
Scanner bufferedScanner = null;
Set<Entrant> entrantSet = new HashSet<>();
try
{
String currentEntrantLine;
Scanner lineScanner;
bufferedScanner = new Scanner(new BufferedReader(new FileReader(aFile)));
while (bufferedScanner.hasNextLine())
{
currentEntrantLine = bufferedScanner.nextLine();
lineScanner = new Scanner(currentEntrantLine);
lineScanner.useDelimiter(" ");
currentEntrantLine = lineScanner.next();
entrantSet.add(new Entrant(currentEntrantLine)); // <----- Here is where I am having trouble. It won't let me add the new object to the class Entrant
}
}
catch (Exception anException)
{
System.out.println("Error: " + anException);
}
finally
{
try
{
bufferedScanner.close();
}
catch (Exception anException)
{
System.out.println("Error: " + anException);
}
}
return entrantSet;
}
I'm not sure what to do. Could anyone see what I am doing wrong?
Sorry for got to add that it is a compilation issue, it will not compile properly.

Use an IDE ,I bet you dont (otherwise it would mark compilation error immediatly with red -> you use return in void method ) and in this case you would see other errors.
(off: this would go to comment section however under 50reputation I am not allowed to do that. Stackoverflow should change this imo. )

First of all:
You marked function readInEntrants as public void so you can't use return inside.
You could either remove return entrantSet; instruction or change function definition to public Set<Entrant> readInEntrants.
Concerning problem you have:
Basing on comment you left on beatrice answer I think you have only parameterless constructor for 'Entrant' class, while you try to create it passing string as parameter.
new Entrant(currentEntrantLine)
What you need to do is define Entrant class constructor that accept String as it's argument. For example:
public Entrant(String dataToParse)
{
// here you parse data from string to entrant fields
}
On the side:
You use bufferedReader to read entire file line at once and that's ok, but then you define Scanner lineScanner to iterate through line elements and then use it only once.
This way for file... let's say:
One Two Three
Four Five Six
Your while loop would work like this:
Store "One Two Three" inside currentEntrantLine.
Create scanner that'll work on "One Two Three", and set it to use space as delimiter.
Use .next to "Finds and returns the next complete token" (see documentation) and then store value inside currentEntrantLine. This way contents of currentEntrantLine is "One". Not entire line.
In next iteration you would have scanner working on "Four Five Six" and "Four" as currentEntranceLine content.

It seems the constructor of entrant class does not have any argument. Pass String as an argument type in the constructor to set the String field inside the Entrant class .

Java Throwing OutOfMemory Even Though (I Think) I've Set References To Null

I have an application that takes a number of different images, makes new images from them and then saves them for use to make a video. The images are all PNG's and the videos are several minutes long so the program requires a lot of memory (one image every 33.33 MS of video play time). When I process a single video, it all works fine. I can even process several videos and it all works fine. But, eventually, I get an outofmemory error if I try to process 1 + n videos.
What is confusing me is how that error happens. Here is the part of the program where the error happens:
ComposeVideoController cvc = new ComposeVideoController();
boolean made = cvc.setXmlUrl(sourcePath, saveDir, fileId);
cvc = null;
To be more precise, the error happens in one of the frame construction classes which is referenced by the ComposeVideoController. ComposeVideoController is scoped to a single void method that runs recursively (if there are more videos to be made). I have gone through all the objects referenced by ComposeVideoController, which is the entry point to the library that builds the videos, and made sure they are all set to null too.
How can I get outofmemory errors in ComposeVideoController when any individual video does not cause an outofmemory error and ComposeVideoController is out of scope (and set null) after any given video is made?
The full recursion is shown below. I have one method that checks to see if there are new messages in queue (messages are sent by Socket) and if there are, it calls the method that processes the video. If not, the recursion ends:
private void processQueue() {
if(makingVideo)
return;
MakeVideoObject mvo = queue.remove(0);
makingVideo = true;
String[] convertArr = mvo.getConvertArrayCommand();
String sourcePath = convertArr[1];
String fileId = convertArr[2] + ".mp4";
String saveDir = convertArr[3] + System.getProperty("file.separator");
try {
ComposeVideoController cvc = new ComposeVideoController();
boolean made = cvc.setXmlUrl(sourcePath, saveDir, fileId);
cvc = null;
if(made) {
cleanDir(mvo);
}
}
catch(Exception e) {
e.printStackTrace();
}
}
/**
* Moves all the assets off to a storage directory where we can be
* able to recover the video assets if something goes wrong during
* video creation.
*
* #param mvo
*/
private void cleanDir(MakeVideoObject mvo) {
String[] convertArr = mvo.getConvertArrayCommand();
String sourceDir = convertArr[1];
String saveDir = convertArr[3] + System.getProperty("file.separator");
String fileId = convertArr[2];
sourceDir = sourceDir.substring(0, sourceDir.lastIndexOf(System.getProperty("file.separator")));
try {
File f = new File(sourceDir);
File[] files = f.listFiles();
for(File file : files) {
if(file.getName().indexOf(fileId) != -1) {
file.renameTo(new File(saveDir + file.getName()));
}
}
makingVideo = false;
mvo = null;
if(queue.size() > 0) {
processQueue();
}
}
catch(Exception e) {
e.printStackTrace();
}
}
[Edited to show more of the program]

thats pretty much what happens if you execute nontrivial code recursively (either this or a classic stack overflow, whichever occurs first) - recursion is VERY resource-intensive, one should avoid it at all costs. Simply exchanging your recursion with an iterative algorithm will make your error go away, most likely

I'm posting an answer since I've figured it out finally and in the off-case this helps anyone else trying to diagnose a memory leak in the future.
During my profiling, I could see that there were objects held in memory, but it made no sense to me as they had been set to null. After going through one of the objects again, I noticed that I had declared it static. Because it was static, it also had static members, one of which was a ConcurrentHashMap... So, those Maps were having things added to them and since the object was static, the object and its members would never be dereferenced. Another lesson for me as to why I almost never declare objects static.

Use -Xmx to increase your memory space. But also, I'd advise explicitly calling System.gc() right about where you're getting the OutOfMemoryException... If you are nulling out your other objects, this will help a lot

how do i use a hashmap keys to an array of strings?

im currently working on a multiple class assignment where i have to add a course based on whether the prerequisites exist within the program.
im storing my courses within the program class using a hashmap. (thought i would come in handy) however, im having a bit of trouble ensuring that these preReqs exist.
here is some code ive currently got going
public boolean checkForCourseFeasiblity(AbstractCourse c) throws ProgramException
{
AbstractCourse[] tempArray = new AbstractCourse[0];
tempArray= courses.keySet().toArray(tempArray);
String[] preReqsArray = new String[1];
preReqsArray = c.getPreReqs();
//gets all course values and stores them in tempArray
for(int i = 0; i < preReqsArray.length; i++)
{
if(courses.containsKey(preReqsArray[i]))
{
continue;
}
else if (!courses.containsKey(preReqsArray[i]))
{
throw new ProgramException("preReqs do not exist"); //?
}
}
return true;
}
ok so basically, tempArray is storing all the keySets inside the courses hashmap and i need to compare all of them with the preReqs (which is an array of Strings). if the preReqs exist within the keyset then add the course, if they dont do not add the course. return true if the course adds otherwise through me an exception. keep in mind my keysets are Strings e.g. a keyset value could be "Programming1" and the required prerquisite for a course could be "programming1". if this is the case add then add the course as the prereq course exists in the keyset.
i believe my error to be when i initialize mypreReqsArray with c.getPreReqs (note: getPreReqs is a getter with a return type String[]).
it would be really great if someone could aid me with my dilemma. ive tried to provide as much as possible, i feel like ive been going around in circles for the past 3 hours :(
-Thank you.

Try something like this, you don't need tempArray. The "for each" loop looks lots nicer too. If you want to throw an Exception I would put that logic in the place that calls this method.
public boolean checkForCourseFeasiblity(AbstractCourse c)
{
for(String each : c.getPreReqs())
{
if(! courses.containsKey(each))
{
return false;
}
}
return true;
}

Duplicate Check in Java

I am writing a small integration piece to to retrive the testcases from TestCase Management tool in java, in which i have the following scenarios:-
1) I have testcase which is “failed”, that time I am checking whether there is any defect which is already exists in the Defect management tool for the failed testcase using the testcase name , because the testcase name and the defect name are same in our case.
If not I am logging the new defect. This is fine.
2) In another case, I have testcase which is “Passed” at the first time, for that also I am checking the whether there are any duplicate defect is present in the Defect management tool , eventhough I am not going to log any defect.
This I am doing because, I don’t know whether the testcase is “Pass” or “Fail” in the first attempt or not. Hence I am doing this mandatory check , to see whether the duplicate defect exists or not for both “pass” and “fail” testcase.
I know that it is wrong to check the duplicate existence of the defect for the “pass” testcase. But there is no option I have. Is there any way we can ignore calling duplicate existence of the defect method if the testcase is “passed”?
I want your guys opinion on this.
This is the code which i have:-
private int NOT_TESTED = 0;
private int PASSED_1 = 0;
private int PASSED_2 = 0;
private int FAILED =0;
private String testStatus = "pass"; // will be fetched dynamically
private void execute(){
if(testStatus.equalsIgnoreCase("fail")){
//FAILED--;
FAILED = FAILED + 1;
System.out.println("the failed value is:"+FAILED);
}else if(testStatus.equalsIgnoreCase("pass")){// first attempt
PASSED_1 = PASSED_1 + 1;
System.out.println("the Passed_1 value is:"+PASSED_1);
if(PASSED_1 == 1){
System.out.println("Passed in the first attempt, hence no need to create a defect");
}
}else if(testStatus.equalsIgnoreCase("pass") && FAILED ==1){// second attempt
PASSED_2 = PASSED_2+1;
System.out.println("the Passed_2 value is:"+PASSED_2);
if(PASSED_2 ==1){
System.out.println("Passed in the second attempt, create a defect");
// logic to create a new defect
}
}else {
System.out.println("The test not tested");
}
}
This code is not working as it always go to the first pass attempt state, hence please do provide a solution to find if the testcase is passed in the second attempt (FAIL->PASS) so that we can take appropriate action on this.
Thanks in advance.

if the condition (testStatus.equalsIgnoreCase("pass") && FAILED ==1) is true, it means that the condition before it is also true:
if(testStatus.equalsIgnoreCase("pass"))
Since you used if-else, it will go into the first condition and then skip the rest.
Switching between those two will give you the desired result I think.