We Keep Coding

Debug MapReduce code to find the cause of NULLPointerException error

I am trying to run a MapReduce code to calculate average satisfaction level per department. Below is the sample data. The file has headers and in order to remove the headers I placed these in another txt file and used it to check if the row values are not equal to headers.
Sample Data
satisfaction_level,last_evaluation,number_project,average_montly_hours,time_spend_company,Work_accident,left,promotion_last_5years,sales,salary
0.38,0.53,2,157,3,0,1,0,sales,low
0.8,0.86,5,262,6,0,1,0,sales,medium
0.11,0.88,7,272,4,0,1,0,sales,medium
0.72,0.87,5,223,5,0,1,0,sales,low
0.37,0.52,2,159,3,0,1,0,sales,low
0.41,0.5,2,153,3,0,1,0,sales,low
0.1,0.77,6,247,4,0,1,0,sales,low
0.92,0.85,5,259,5,0,1,0,sales,low
0.89,1,5,224,5,0,1,0,sales,low
Below is my mapper code.
public class SatisfactionLevelMapper extends Mapper<LongWritable, Text, Text, FloatWritable>
{
String header;
#Override
protected void setup(Mapper.Context context) throws IOException, InterruptedException
/* Headers are placed in a separate header.txt file for which the location is specified in the driver */
{
BufferedReader bufferedReader = new BufferedReader(new FileReader("header.txt"));
header = bufferedReader.readLine();
}
public void map(LongWritable key, Text text, Context context) throws IOException, InterruptedException
{
String row = text.toString();
String [] values = row.trim().split(","); //dataset is a CSV file with 10 columns
float satisfaction = 0.0f;
String dept = null;
try
{
if(values.length == 9 && header != row)
{
satisfaction = Float.parseFloat(values[0]); }
}
catch (Exception e)
{
e.printStackTrace();
}
context.write(new Text(dept), new FloatWritable(satisfaction));
}
}
I am getting a NullPointerException Error as below.
Error: java.lang.NullPointerException
at org.apache.hadoop.io.Text.encode(Text.java:450)
at org.apache.hadoop.io.Text.set(Text.java:198)
at org.apache.hadoop.io.Text.<init>(Text.java:88)
at com.df.hra.SatisfactionLevelMapper.map SatisfactionLevelMapper.java:62)
at com.df.hra.SatisfactionLevelMapper.map(SatisfactionLevelMapper.java:1)
at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:145)
at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:784)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:341)
at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:168)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1642)
at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:163)
Is there a way to figure out which part of my code has the issue?
I am new to Java and trying to figure out a way to debug MapReduce codes. Any help would be appreciated. Thanks!

You cannot initialize a Text value with null, as you're doing here (with dept). Consider using an empty String ("") instead.

Vaadin FileDropTarget terrible performance

I'm trying to use Vaadin 8 FileDropTarget with a Tomcat server. The example code is little bit long, but maybe it will fit here:
import java.io.FileNotFoundException;
import java.io.FileOutputStream;
import java.io.OutputStream;
import com.vaadin.server.StreamVariable;
import com.vaadin.ui.Alignment;
import com.vaadin.ui.Label;
import com.vaadin.ui.Notification;
import com.vaadin.ui.ProgressBar;
import com.vaadin.ui.VerticalLayout;
import com.vaadin.ui.dnd.FileDropTarget;
import hu.dynaxpress.base.mvc.DynaAbstractController;
#SuppressWarnings({ "serial", "rawtypes" })
public class FileStreamController extends DynaAbstractController {
private VerticalLayout layout;
private ProgressBar progress;
#Override
protected void onInit() {
super.onInit();
System.err.println("FileStreamController.onInit");
}
#Override
protected void onEnter() {
System.err.println("FileStreamController.onEnter");
// https://vaadin.com/docs/v8/framework/components/components-upload.html
layout = new VerticalLayout();
final Label infoLabel = new Label("DROP A FILE ON THIS TEXT");
infoLabel.setWidth(240.0f, Unit.PIXELS);
final VerticalLayout dropPane = new VerticalLayout(infoLabel);
dropPane.setComponentAlignment(infoLabel, Alignment.MIDDLE_CENTER);
dropPane.addStyleName("drop-area");
dropPane.setSizeUndefined();
dropPane.setWidth("100%");
progress = new ProgressBar();
progress.setIndeterminate(false);
progress.setVisible(false);
progress.setWidth("100%");
dropPane.addComponent(progress);
layout.addComponent(dropPane);
new FileDropTarget<>(dropPane, fileDropEvent -> {
final long fileSizeLimit = 20 * 1024 * 1024 * 1024; // 20GB
fileDropEvent.getFiles().forEach(html5File -> {
// html5File.getFileSize() always returns zero, but why?
if (false && ( html5File.getFileSize() > fileSizeLimit) ) {
Notification.show(
"File rejected. Max size=" +fileSizeLimit+ ", actual="+html5File.getFileSize(),
Notification.Type.WARNING_MESSAGE);
} else {
Label lbl = new Label(html5File.getFileName() + " size=" + html5File.getFileSize() + " " + html5File.getType());
lbl.setWidth("100%");
layout.addComponent(lbl);
final StreamVariable streamVariable = new StreamVariable() {
#Override
public OutputStream getOutputStream() {
try {
return new FileOutputStream("F:\\"+html5File.getFileName());
} catch (FileNotFoundException e) {
e.printStackTrace();
throw new Error("F:\\"+html5File.getFileName());
}
}
#Override
public boolean listenProgress() { return false; }
#Override
public void onProgress(final StreamingProgressEvent event) {
progress.setValue(
event.getBytesReceived() / event.getContentLength()
);
}
#Override
public void streamingStarted(
final StreamingStartEvent event) {
progress.setVisible(true);
progress.setCaption(event.getFileName());
}
#Override
public void streamingFinished(
final StreamingEndEvent event) {
progress.setVisible(false);
}
#Override
public void streamingFailed(final StreamingErrorEvent event) {
progress.setVisible(false);
}
#Override
public boolean isInterrupted() { return false; }
};
html5File.setStreamVariable(streamVariable);
progress.setVisible(true);
}
});
});
setCompositionRoot(layout);
}
}
This code works, but it has terrible performance. In this MWE, the program writes all dropped files to the F:\ drive. It is a flash drive (I wanted to simulate streaming large files and see how much memory is consumed over time). But I found that this program writes out a 40MB file in 20 seconds. That is 2MB/sec speed. The server runs on localhost, the source file and the target F: drive are all on my local machine. If I simply copy the same file to the drive then it takes less than 2 seconds.
I have tried with another (very slow) flash drive too, and had similar results. When tomcat is streaming the file, then it is about 10-20 times slower.
Is there a way to make this faster? What am I doing wrong? I do not see any way to speed this up, because the streaming happens inside the component.
The second important (no related) question is that why can't I get the file size? See the comment in the code: "html5File.getFileSize() always returns zero, but why?" I have checked the POST headers from firefox debugger, and the browser sends the size of the file in an JSON/RPC call, before it sends the actual file data. So the server should know the file size before the first byte arrives.

Why cant I open 32bit Outlook on a 64bit System using a JAR when it worked in Eclipse...?

The Java app I am developing is an internal tool for others to use to help our daily activities. When the tool is used (on a JButton press which calls the code below) I want it to open a new Outlook EMail for the user to see/edit.
At first I was developing this app in a 64 bit Eclipse and could not get SWT to open Outlook despite all my research. After I had some issues running 64bit versus 32bit SWT, I had the idea to check Outlook, and sure enough the company using is 32 bit. I loaded up a 32 bit eclipse, imported my project, switch SWT to 32 bit and it worked exactly as intended.
I noticed the process that was running was javaw.exe*32 but the 64bit Eclipse was using the process javaw.exe. I exported the JAR from the 32bit Eclipse and gave it a shot and no EMail showed up. I checked the JRE's installed and saw both 32bit and 64bit but my company had a policy that forced only the 64bit JRE in the Java Control Panel. I worked with some others and got both installed and enabled as seen here. Still the JAR fails to open the EMail. I even tried disabling the 64bit and it still does not work.
Is there anything that can be done to remedy this situation? Please let me know if I can elaborate better or provide more info!
package EDM_PKG;
import org.eclipse.swt.SWT;
import org.eclipse.swt.layout.FillLayout;
import org.eclipse.swt.ole.win32.OLE;
import org.eclipse.swt.ole.win32.OleAutomation;
import org.eclipse.swt.ole.win32.OleClientSite;
import org.eclipse.swt.ole.win32.OleFrame;
import org.eclipse.swt.ole.win32.Variant;
import org.eclipse.swt.widgets.Display;
import org.eclipse.swt.widgets.Shell;
public class CreateEmail {
private static Display display = new Display();
public static void sendEMail(String env) {
//String currUser = System.getProperty("user.name");
String msg = getMessage(env);
String sub = getSubject(env);
Shell shell = new Shell(display);
OleFrame frame = new OleFrame(shell, SWT.NONE);
// This should start outlook if it is not running yet
//OleClientSite site = new OleClientSite(frame, SWT.NONE,"OVCtl.OVCtl");
//site.doVerb(OLE.OLEIVERB_INPLACEACTIVATE);
// Now get the outlook application
OleClientSite site2 = new OleClientSite(frame, SWT.NONE, "Outlook.Application");
OleAutomation outlook = new OleAutomation(site2);
OleAutomation mail = invoke(outlook, "CreateItem", 0 /* Mail item */).getAutomation();
setProperty(mail, "BodyFormat", 2 /* HTML */);
setProperty(mail, "Subject", sub);
setProperty(mail, "To", "example#gmail.com");
setProperty(mail, "HtmlBody", msg);
invoke(mail, "Display" /* or "Send" */);
display = null;
}
private static String getMessage(String env) {
String msg = "";
if (env.equalsIgnoreCase("quas")) {
msg = "<html><body>The <b>USER</b> excel has been migrated to <b>QUAS.</b><br><br> This email was generated by Excel Datatable Migrator.</body></html>";
} else if (env.equalsIgnoreCase("prod")) {
msg = "<html><body>The <b>QUAS</b> excel has been migrated to <b>PROD.</b><br><br>This email was generated by Excel Datatable Migrator.</body></html>";
} else {
msg = "Somthing happened with the automated message of EDM. Please contact the user with the eCode: "+System.getProperty("user.name")+".</body></html>";
}
return msg;
}
private static String getSubject(String env) {
String sub = "";
if (env.equalsIgnoreCase("quas")) {
sub = "EDM has been used to move USER to QUAS...";
} else if (env.equalsIgnoreCase("prod")) {
sub = "EDM has been used to move QUAS to PROD...";
} else {
sub = "Somthing didnt quite work right...";
}
return sub;
}
private static OleAutomation getProperty(OleAutomation auto, String name) {
Variant varResult = auto.getProperty(property(auto, name));
if (varResult != null && varResult.getType() != OLE.VT_EMPTY) {
OleAutomation result = varResult.getAutomation();
varResult.dispose();
return result;
}
return null;
}
private static Variant invoke(OleAutomation auto, String command,
String value) {
return auto.invoke(property(auto, command),
new Variant[] { new Variant(value) });
}
private static Variant invoke(OleAutomation auto, String command) {
return auto.invoke(property(auto, command));
}
private static Variant invoke(OleAutomation auto, String command, int value) {
return auto.invoke(property(auto, command),
new Variant[] { new Variant(value) });
}
private static boolean setProperty(OleAutomation auto, String name,
String value) {
return auto.setProperty(property(auto, name), new Variant(value));
}
private static boolean setProperty(OleAutomation auto, String name,
int value) {
return auto.setProperty(property(auto, name), new Variant(value));
}
private static int property(OleAutomation auto, String name) {
return auto.getIDsOfNames(new String[] { name })[0];
}
}

There's no need to use the heavyweight and difficult to use Ole classes.
If all you want is to send an email, just call this:
Program.launch("mailto:bla#blubb.com&subject=Subject&body=Me‌​ssage here");
This will work on all architectures and operating systems.

Detect file type based on content

Tried the following:
import java.io.IOException;
import java.nio.file.Path;
import java.nio.file.Paths;
import java.nio.file.spi.FileTypeDetector;
import org.apache.tika.Tika;
import org.apache.tika.mime.MimeTypes;
/**
*
* #author kiriti.k
*/
public class TikaFileTypeDetector {
private final Tika tika = new Tika();
public TikaFileTypeDetector() {
super();
}
public String probeContentType(Path path) throws IOException {
// Try to detect based on the file name only for efficiency
String fileNameDetect = tika.detect(path.toString());
if (!fileNameDetect.equals(MimeTypes.OCTET_STREAM)) {
return fileNameDetect;
}
// Then check the file content if necessary
String fileContentDetect = tika.detect(path.toFile());
if (!fileContentDetect.equals(MimeTypes.OCTET_STREAM)) {
return fileContentDetect;
}
// Specification says to return null if we could not
// conclusively determine the file type
return null;
}
public static void main(String[] args) throws IOException {
Tika tika = new Tika();
// expects file path as the program argument
if (args.length != 1) {
printUsage();
return;
}
Path path = Paths.get(args[0]);
TikaFileTypeDetector detector = new TikaFileTypeDetector();
// Analyse the file - first based on file name for efficiency.
// If cannot determine based on name and then analyse content
String contentType = detector.probeContentType(path);
System.out.println("File is of type - " + contentType);
}
public static void printUsage() {
System.out.print("Usage: java -classpath ... "
+ TikaFileTypeDetector.class.getName()
+ " ");
}
}
The above program is checking based on file extension only. How do I make it to check content type also(mime) and then determine the type. I am using tika-app-1.8.jar in netbean 8.0.2. What am I missing?

The code checks the file extension first and returns the MIME type based on that, if it finds a result. If you want it to check the content first, just switch the two statements:
public String probeContentType(Path path) throws IOException {
// Check contents first
String fileContentDetect = tika.detect(path.toFile());
if (!fileContentDetect.equals(MimeTypes.OCTET_STREAM)) {
return fileContentDetect;
}
// Try file name only if content search was not successful
String fileNameDetect = tika.detect(path.toString());
if (!fileNameDetect.equals(MimeTypes.OCTET_STREAM)) {
return fileNameDetect;
}
// Specification says to return null if we could not
// conclusively determine the file type
return null;
}
Be aware that this may have huge performance impact.

You can use Files.probeContentType(path)

Jtextfield does not display the text

I am trying to pass values from my database java file to my frame java file.BY creating a function like "frame java . get value (name ,phone, date ) " and receiving the value in the frame java file . I tried to print the passed value in the console it works fine, but when i try to set value to the text field it doesn't display the text in the text filed... I don't what's wrong can any one help me to sort out this issue.
Here is database java function
public void check_room(String roomno, String date) throws SQLException {
String sql = "select * from customerinfo where roomno='" + roomno
+ "'and cdate='" + date + "' ";
System.out.print("search method called \n ");
System.out.print("\n ");
try {
con = DriverManager.getConnection(
"jdbc:oracle:thin:#localhost:1521:xe", "hotel", "hotel");
} catch (SQLException ex) {
// Logger.getLogger(checkout.class.getName()).log(Level.SEVERE,
// null, ex);
}
sate = con.createStatement();
rs = sate.executeQuery(sql);
int tmp = 0;
try {
while (rs.next()) {
System.out.print("search found \n ");
String name = rs.getString("GUEST_NAME");
String phono = rs.getString("GUEST_PHO");
String addr = rs.getString("G_ADDR");
String paid = rs.getString("PAID");
String total = rs.getString("TOTAL");
String balance = rs.getString("BALANCE");
System.out.println(name);
mainmenu menu = new mainmenu();
menu.getvalue(name, phono, addr, paid, total, balance);
tmp++;
}
} catch (SQLException ex) {
// Logger.getLogger(checkout.class.getName()).log(Level.SEVERE,
// null, ex);
}
if (tmp <= 0) {
JOptionPane.showMessageDialog(null, "no details found ");
}
}
==========================================================================
This is my frame java file
public void getvalue(String name, String phono, String addr, String paid,String total, String balance) {
nam.setText(name);
pho.setText(phono);
// mainmenu();
}
I edited as u said but getting error . I have attached my skeleton code
and the error i get, below
edited code.
database file
public class OracelThinconnection
{
private MainMenu menu;
public OracelThinconnection(MainMenu menu)
{
this.menu=menu;
..................
.................
.............
========================================================================
Error i get
Exception in thread "AWT-EventQueue-0" java.lang.RuntimeException: Uncompilable source code - cannot find symbol
symbol: constructor OracelThinconnection()
location: class bookingapp.OracelThinconnection
at bookingapp.MainMenu.jButton1ActionPerformed(MainMenu.java:1438)
at bookingapp.MainMenu.access$300(MainMenu.java:33)
at bookingapp.MainMenu$4.actionPerformed(MainMenu.java:379)
at javax.swing.AbstractButton.fireActionPerformed(AbstractButton.java:1995)
at javax.swing.AbstractButton$Handler.actionPerformed(AbstractButton.java:2318)
at javax.swing.DefaultButtonModel.fireActionPerformed(DefaultButtonModel.java:387)
at javax.swing.DefaultButtonModel.setPressed(DefaultButtonModel.java:242)
at javax.swing.plaf.basic.BasicButtonListener.mouseReleased(BasicButtonListener.java:236)
at java.awt.Component.processMouseEvent(Component.java:6263)
at javax.swing.JComponent.processMouseEvent(JComponent.java:3267)
at java.awt.Component.processEvent(Component.java:6028)
at java.awt.Container.processEvent(Container.java:2041)
at java.awt.Component.dispatchEventImpl(Component.java:4630)
at java.awt.Container.dispatchEventImpl(Container.java:2099)
at java.awt.Component.dispatchEvent(Component.java:4460)
at java.awt.LightweightDispatcher.retargetMouseEvent(Container.java:4574)
at java.awt.LightweightDispatcher.processMouseEvent(Container.java:4238)
at java.awt.LightweightDispatcher.dispatchEvent(Container.java:4168)
at java.awt.Container.dispatchEventImpl(Container.java:2085)
at java.awt.Window.dispatchEventImpl(Window.java:2478)
at java.awt.Component.dispatchEvent(Component.java:4460)
at java.awt.EventQueue.dispatchEvent(EventQueue.java:599)
at java.awt.EventDispatchThread.pumpOneEventForFilters(EventDispatchThread.java:269)
at java.awt.EventDispatchThread.pumpEventsForFilter(EventDispatchThread.java:184)
at java.awt.EventDispatchThread.pumpEventsForHierarchy(EventDispatchThread.java:174)
at java.awt.EventDispatchThread.pumpEvents(EventDispatchThread.java:169)
at java.awt.EventDispatchThread.pumpEvents(EventDispatchThread.java:161)
at java.awt.EventDispatchThread.run(EventDispatchThread.java:122)

I don't understand the code you've not shown, but a guess as to your problem is that you're creating a new instance of your GUI, one not visualized, and trying to send information to it, while the visualized GUI sits unperturbed and un-updated.
The key is that you're creating a new instance of your mainmenu GUI, one that has no relationship to the one that is displayed. Passing data to this new instance will have no effect on the displayed GUI.
Here:
mainmenu menu= new mainmenu();
menu. getvalue (name, phono, addr, paid, total, balance);
You're creating a new mainmenu object (the class should be named MainMenu).
A solution, give your Database class a MainMenu field, and pass into this field a valid reference to the main gui, and then call methods on the GUI, but only on the Swing event thread. For example:
public class MyDataBase {
private MyGui gui;
// constructor gets passed a reference to the valid displayed GUI
public MyDataBase(MyGui gui) {
this.gui = gui;
}
public void someDataBaseMethod() {
// get database information
// now you can call methods on the actual displayed GUI
// but on the Swing event thread only. The SwingUtilities.invokeLater
// with the Runnable is required to be sure that you don't mess up
// Swing's threading model.
SwingUtilities.invokeLater(new Runnable() {
public void run() {
// this method below is being called on the
// displayed MyGui instance.
gui.updateGuiWithInformation(....); // some update method of the GUI
}
});
}
The key -- don't create a new GUI instance inside of the database code.
OK, you've changed your code and now have a constructor that looks like so:
public OracelThinconnection(MainMenu menu) {
// ....
}
Now you need to look at the line that the error is pointing to, MainMenu.java:1438, line 1438 (1438????) of the MainMenu class. On that line you likely call the OracelThinconnection constructor but are not passing the current MainMenu instance into it. The new constructor will accept a MainMenu reference, but when you call the constructor, you now have to remember to pass that reference in to the constructor.
Specifically, change it from
// not sure what you name the variable below
// but I just gave it thinConnection for now.
OracelThinconnection thinConnection = new OracelThinconnection();
to this:
OracelThinconnection thinConnection = new OracelThinconnection(this);
or if this doesn't work, then
OracelThinconnection thinConnection = new OracelThinconnection(MainMenu.this);
As an addendum, your code obviously has more than 1400 lines which tells you that you've got some huge God class that tries to do any thing and everything. I strongly urge you to refactor this code, to make smaller classes, each with their own sphere of responsibilities, else you will not be able to debug or enhance this program.