We Keep Coding

How to define Spark RDD transformation with non-Lambda Function

I recently started using with Spark and Java. I am currently experimenting with RDD transformations and actions. For the moment I am reading data out of a csv that contains some DateTime fields and then I apply a filter to keep only those rows that are younger than 2 days and finally I check if the resulting RDD is empty. I wrote a simple snippet that does what I want on a minimal level.
Function<List<String>, Boolean> filterPredicate = row -> new DateTime(row.get(1).isAfter(dtThreshold);
sc.textFile(inputFilePath)
.map(text -> Arrays.asList(text.split(",")))
.filter(filterPredicate)
.isEmpty();
On this simple case I have assumed that the DateTime objects always live on the first column. I now want to expand that to use multiple column indexes.
But to do that I need to be able to define a predicate function with more than one lines. That is the reason why I have separated the predicate function definition from the transformation code.
How I am supposed to define such a function?

Use the curly brace notation...
Function<List<String>, Boolean> filterPredicate = row -> {
boolean isDateAfter = new DateTime(row.get(1)).isAfter(dtThreshold);
boolean hasName = row.get(2) != "";
return isDateAfter && hasName;
}

Generic Class for JTable

i got a task which iam not sure of how to solve.
I have to fill a JTable with rows i get from a .txt document. The problem is that there are multiple .txt documents which have more or less rows and columns for the JTable.
example:
inside the cars.txt:
id;hp;price;quantity
1;100;7000;5
4;120;20000;2
7;300;80000;3
inside the bikes.txt
id;price;quantity;color;year
3;80;20;red;2010
5;200;40;green;2011
12;150;10;blue;2007
So, when a .txt is chosen a JDialog will pop up with a JTable inside, where the data will be shown.
I thought that i could maybe create a "class Anything" where i have a instance variable String[][] which i can define the sizes by reading the .txt and after saving the data in one array i can count how many rows and how many columns it has,
with the cars.txt example it would be: String[4][3]
Is that a good way to work with or is there a better way to do it?
Thanks for the help :D

Your question is a bit vague on what you want to do specifically.
Do you want to simply fill the table with all data given? Or do you only want certain columns used? When you choose the text files are you aware of which column names they have (can you hardcode this or not).
A good start would be...
EDITED here's the solution.....
DefaultTableModel dtm = (DefaultTableModel)yourJTable.getModel();
// This divides your txt file to a string array divided by rows
string[] RowSplit = yourTxtFileThatYouRead.split("\n");
//this assumes that your txt file contains the column headers
dtm.setColumnHeaders(RowSplit[0].split(";"));
//Start the iteration at 1 to skip the column headers
for (int i = 1; i < RowSplit.length; ++i) {
dtm.addRow(RowSplit[i].split(//some delimeter //));
dtm.fireTableDataChanged();
The first part sets the column headers and enables for variation within your table column size.
The second part sequentially adds rows.
edited for formatting
edited for better answer

As shown in How to Use Tables: Creating a Table Model, you can extend AbstractTableModel to manage models of arbitrary dimensions. Let your model manage a List<List<String>>. Parse the first line of each file into a List<String> that is accessed by your implementations of getColumnCount() and getColumnName(). Parse subsequent lines into one List<String> per row; access the List of such rows in your implementation of getValueAt(). A related example that manages a Map<String, String> is shown here. Although more complex, you can use Class Literals as Runtime-Type Tokens for non-string data; return the token in your implementation of getColumnClass() to get the default render and editor for supported types. Finally, consider one of these file based JDBC drivers for flat files.

Sanitize cell values with SuperCSV

What is the best way to sanitize fields from a csv in supercsv? For example the First_Name column: trim the field, capitalize the first letter, remove various characters (quotes, commas, asterisks etc). Is it to write a custom CellProcessor like FmtName()? Maybe another one for FmtEmail() that lowercases everything, removes certain invalid characters?

I think the question you're trying to ask is:
"Is it better to write a custom cell processor that does all the
conversions for a column, or to chain multiple reusable processors
together?"
For example, with your first name example you could either:
a) write a custom cell processor which trimmed, capitalised and replaced all in the one processor:
new ParseFirstName()
b) chain together reusable processors (including the existing Super CSV processors and a new Capitalize custom cell processor that calls StringUtils.capitalize())
new Trim(new Capitalize(new StrReplace("[\",\\*]", "")))
I think it's really up to personal preference. Defining cell processors as done in b) can be quite verbose, but it means you can see all of the conversions/validation for all columns in the one place.
On the other hand, defining a custom cell processor for each column makes your cell processor setup very clean, but you may end up with duplicated code (e.g. if you wanted to capitalize multiple columns) and you can't see all the conversions at once. You'll also have more classes (more code).

Format a Jtable Cell, if illegal format, then do nothing

Say I have following columns in my Jtable:
{"Item", "Price"}
I want format the price column to "0.00" format.
But if user enter an illegal string, then keep the original value of the cell.
ex, when the price is "1.23", but when user want to enter "abc", then the cell should still be "1.23"
How could I do this?
Thanks

Let's split your big problemn into smaller ones.
First, you want to have items of your Price column to be displayed according to a specified format, which you can achieve using a TableCellRenderer (which, when rendering numbers, will use a NumberFormat to display the correct number of decimals). As your table has only two columns of different types, the simplest solution is to use JTable#setDefaultRenderer(Class<?> columnClass, TableCellRenderer renderer). This way, your numbers will always be displayed using the correct number of decimals.
Second, for your edition issue, the same solution can be rather elegant : call JTable#setDefaultEditor(java.lang.Class, javax.swing.table.TableCellEditor) and set as editor a component one that will, before commiting change to your table model, ensure the new value is a valid number (valid of course according to your rules).

A JTable supports this by default. The table will choose the renderer/editor based on the data stored in each column. So all you need to do is override the getColumnClass(...) method to return Double.class. Then when you edit a cell the editor will make sure the value entered is a valid number.
If you want more control over how the number is formatted then you can use Table Format Renderer.

JFormattedTextField works the way you describe so just create a new CellEditor that extends JFormattedTextField.

Put the formatting command into a try/catch block. If the operation fails with an NumberFormatException, cancel the edit.
Using your original code, make a few minor changes:
public void setValue(Object value) {
NumberFormat formatter = new DecimalFormat("0.00");
String s = null;
try {
s = formatter.format(Integer.parseInt((String)value));
if (null != s) setText(s);
} catch (NumberFormatException ex) {
// what could I do here?
// You could display a warning JOptionPane and/or output to System.out
}
}

Reading string value from Excel with HSSF but it's double

I'm using HSSF-POI for reading excel data. The problem is I have values in a cell that look like a number but really are strings. If I look at the format cell in Excel, it says the type is "text". Still the HSSF Cell thinks it's numeric. How can I get the value as a string?
If I try to use cell.getRichStringValue, I get exception; if cell.toString, it's not the exact same value as in Excel sheet.
Edit: until this gets resolved, I'll use
new BigDecimal(cell.getNumericCellValue()).toString()

The class you're looking for in POI is DataFormatter
When Excel writes the file, some cells are stored as literal Strings, while others are stored as numbers. For the latter, a floating point value representing the cell is stored in the file, so when you ask POI for the value of the cell that's what it actually has.
Sometimes though, especially when doing Text Extraction (but not always), you want to make the cell value look like it does in Excel. It isn't always possible to get that exactly in a String (non full space padding for example), but the DataFormatter class will get you close.
If you're after a String of the cell, looking much as you had it looking in Excel, just do:
// Create a formatter, do this once
DataFormatter formatter = new DataFormatter(Locale.US);
.....
for(Cell cell : row) {
CellReference ref = new CellReference(cell);
// eg "The value of B12 is 12.4%"
System.out.println("The value of " + ref.formatAsString() + " is " + formatter.formatCellValue(cell));
}
The formatter will return String cells as-is, and for Numeric cells will apply the formatting rules on the style to the number of the cell

If the documents you are parsing are always in a specific layout, you can change the cell type to "string" on the fly and then retrieve the value. For example, if column 2 should always be string data, set its cell type to string and then read it with the string-type get methods.
cell.setCellType(Cell.CELL_TYPE_STRING);
In my testing, changing the cell type did not modify the contents of the cell, but did allow it to be retrieved with either of the following approaches:
cell.getStringCellValue();
cell.getRichStringCellValue().getString();
Without an example of a value that is not converting properly, it is difficult to know if this will behave any differently than the cell.toString() approach you described in the description.

You mean HSSF-POI says
cell.getCellType() == Cell.CELL_TYPE_NUMERIC
NOT
Cell.CELL_TYPE_STRING as it should be?
I would think it's a bug in POI, but every cell contains a Variant, and Variant has a type. It's kind of hard to make a bug there, so instead I think Excel uses some extra data or heuristic to report the field as text. Usual MS way, alas.
P.S. You cannot use any getString() on a Variant containing numeric, as the binary representation of the Variant data depends on it's type, and trying to get a string from what is actually a number would result in garbage -- hence the exception.

This below code works fine to read any celltype but that cell should contain numeric value
new BigDecimal(cell.getNumericCellValue()));
e.g.
ase.setGss(new BigDecimal(hssfRow.getCell(3).getNumericCellValue()));
where variable gss is of BigDecimal type.

Excel will convert anything that looks like a number or date or time from a string. See MS Knowledge base article, which basically suggests to enter the number with an extra character that makes it a string.

You are probably dealing with an Excel problem. When you create the spreadsheet, the default cell type is Generic. With this type, Excel guesses the type based on the input and this type is saved with each cell.
When you later change the cell format to Text, you are just changing the default. Excel doesn't change every cell's type automatically. I haven't found a way to do this automatically.
To confirm this, you can go to Excel and retype one of the numbers and see if it's text in HSSF.
You can also look at the real cell type by using this function,
#Cell("type", A1)
A1 is the cell for the number. It shows "l" for text, "v" for numbers.

The problem with Excel is that the default format is generic. With this format Excel stores numbers entered in the cell as numeric. You have to change the format to text before entering the values. Reentering the values after changing the format will also work.
That will lead to little green triangles in the left upper corner of the cells if the content looks like a number to Excel. If this is the case the value is really stored as text.
With new BigDecimal(cell.getNumericCellValue()).toString() you will still have a lot of problems. For example if you have identifying numbers (e.g. part numbers or classification numbers) you probably have cases that have leading zeros which will be a problem with the getNumericCellValue() approach.
I try to thoroughly explain how to correctly create the Excel to the party creating the files I have to handle with POI. If the files are uploaded by end users I even have created a validation program to check for expected cell types if I know the columns in advance. As a by-product you can also check various other things of the supplied files (e.g. are the right columns provided or mandatory values).

"The problem is I have values in a cell that look like a number" => look like number when viewed in Excel?
"but really are strings" => what does that mean? How do you KNOW that they really are strings?
"If I look at the format cell" => what's "the format cell"???
'... in Excel, it says the type is "text"' => Please explain.
"Still the HSSF Cell thinks it's numeric." => do you mean that the_cell.getCellType() returns Cell.CELL_TYPE_NUMERIC?
"How can I get the value as a string?" => if it's NUMERIC, get the numeric value using the_cell.getNumericCellValue(), then format it as a string any way you want to.
"If I try to use cell.getRichStringValue, I get exception;" => so it's not a string.
"if cell.toString, it's not the exact same value as in Excel sheet." => so cell.toString() doesn't format it the way that Excel formats it.
Whatever heuristic Excel uses to determine type is irrelevant to you. It's the RESULT of that decision as stored in the file and revealed by getCellType() that matters.