We Keep Coding

Scanning texts for specific words

I want to create an algorithm that searches job descriptions for given words (like Java, Angular, Docker, etc). My algorithm works, but it is rather naive. For example, it cannot detect the word Java if it is contained in another word (such as JavaEE). When I check for substrings, I have the problem that, for example, Java is recognized in the word JavaScript, which I want to avoid. I could of course make an explicit case distinction here, but I'm more looking for a general solution.
Are there any particular techniques or approaches that try to solve this problem?
Unfortunately, I don't have the amount of data necessary for data-driven approaches like machine learning.

Train a simple word2vec language model with your whole job description text data. Then use your own logic to find the keywords. When you find a match, if it's not an exact match use your similar words list.
For example you're searching for Java but find also javascript, use your word vectors to find if there is any similarity between them (in another words, if they ever been used in a similar context). Java and JavaEE probably already used in a same sentence before but java and javascript or Angular and Angularentwicklung been not.
It may seems a bit like over-engineering, but its not :).

I spent some time researching my problem, and I found that identifying certain words, even if they don't match 1:1, is not a trivial problem. You could solve the problem by listing synonyms for the words you are looking for, or you could build a rule-based named entity recognition service. But that is both error-prone and maintenance-intensive.
Probably the best way to solve my problem is to build a named entity recognition service using machine learning. I am currently watching a video series that looks very promising for the given problem. --> https://www.youtube.com/playlist?list=PL2VXyKi-KpYs1bSnT8bfMFyGS-wMcjesM
I will comment on this answer when I am done with my work to give feedback to those who are facing the same problem.

Retrain Stanford coreNLP Lemmatizer

I am trying to work out how the lemmatizer could support the particular use case of identifying e.g. certain regional variants in a corpus. The two possible approaches that I could follow could be:
amend the existing dictionary to use my own material, or
retrain the lemmatizer
I am wondering if you can point me to documentation that I can follow for either approach. I am aware that the Lemma module works in this way:
PropertiesUtils.asProperties(
"annotators", "tokenize,ssplit,pos,lemma",
"ssplit.isOneSentence", "true",
"tokenize.language", "en"));
Apart from CoreAnnotations.LemmaAnnotation, I wasn't able to track something for what I would like to do. I’d appreciate any help you can provide.
Many thanks

Identify an english word as a thing or product?

Write a program with the following objective -
be able to identify whether a word/phrase represents a thing/product. For example -
1) "A glove comprising at least an index finger receptacle, a middle finger receptacle.." <-Be able to identify glove as a thing/product.
2) "In a window regulator, especially for automobiles, in which the window is connected to a drive..." <- be able to identify regulator as a thing.
Doing this tells me that the text is talking about a thing/product. as a contrast, the following text talks about a process instead of a thing/product -> "An extrusion coating process for the production of flexible packaging films of nylon coated substrates consisting of the steps of..."
I have millions of such texts; hence, manually doing it is not feasible. So far, with the help of using NLTK + Python, I have been able to identify some specific cases which use very similar keywords. But I have not been able to do the same with the kinds mentioned in the examples above. Any help will be appreciated!

What you want to do is actually pretty difficult. It is a sort of (very specific) semantic labelling task. The possible solutions are:
create your own labelling algorithm, create training data, test, eval and finally tag your data
use an existing knowledge base (lexicon) to extract semantic labels for each target word
The first option is a complex research project in itself. Do it if you have the time and resources.
The second option will only give you the labels that are available in the knowledge base, and these might not match your wishes. I would give it a try with python, NLTK and Wordnet (interface already available), you might be able to use synset hypernyms for your problem.

This task is called named entity reconition problem.
EDIT: There is no clean definition of NER in NLP community, so one can say this is not NER task, but instance of more general sequence labeling problem. Anyway, there is still no tool that can do this out of the box.
Out of the box, Standford NLP can only recognize following types:
Recognizes named (PERSON, LOCATION, ORGANIZATION, MISC), numerical
(MONEY, NUMBER, ORDINAL, PERCENT), and temporal (DATE, TIME, DURATION,
SET) entities
so it is not suitable for solving this task. There are some commercial solutions that possible can do the job, they can be readily found by googling "product name named entity recognition", some of them offer free trial plans. I don't know any free ready to deploy solution.
Of course, you can create you own model by hand-annotating about 1000 or so product name containing sentences and training some classifier like Conditional Random Field classifier with some basic features (here is documentation page that explains how to that with stanford NLP). This solution should work reasonable well, while it won't be perfect of course (no system will be perfect but some solutions are better then others).
EDIT: This is complex task per se, but not that complex unless you want state-of-the art results. You can create reasonable good model in just 2-3 days. Here is (example) step-by-step instruction how to do this using open source tool:
Download CRF++ and look at provided examples, they are in a simple text format
Annotate you data in a similar manner
a OTHER
glove PRODUCT
comprising OTHER
...
and so on.
Spilt you annotated data into two files train (80%) and dev(20%)
use following baseline template features (paste in template file)
U02:%x[0,0]
U01:%x[-1,0]
U01:%x[-2,0]
U02:%x[0,0]
U03:%x[1,0]
U04:%x[2,0]
U05:%x[-1,0]/%x[0,0]
U06:%x[0,0]/%x[1,0]
4.Run
crf_learn template train.txt model
crf_test -m model dev.txt > result.txt
Look at result.txt. one column will contain your hand-labeled data and other - machine predicted labels. You can then compare these, compute accuracy etc. After that you can feed new unlabeled data into crf_test and get your labels.
As I said, this won't be perfect, but I will be very surprised if that won't be reasonable good (I actually solved very similar task not long ago) and certanly better just using few keywords/templates
ENDNOTE: this ignores many things and some best-practices in solving such tasks, won't be good for academic research, not 100% guaranteed to work, but still useful for this and many similar problems as relatively quick solution.

simple sentiment analysis with java

I am very new to Sentiment analysis. How can I judge if a given word or sentence is positive or negative. I have to implement it with java. I tried to read something like lingpipe, rapidminer tutorial, but I do not understand. In their examples they use a lot of data. In my case I do not have much data. All I have is a word or a sentence, lets say. I tried to read the questions from stackoverflow too. But they do not help me much.
Thanks in advance.

Computers don't know about a human thing like sentiment unless they learn it from examples that a human has labeled as positive or negative.
The goal of Machine Learning is in fact to make the most informed decision about a new example based on the empirical data of previous examples. Statistically, the more data, the better.
To "judge" the sentiment of a sentence, you'll need to have trained a model or classifier on some sentences labeled for sentiment. The classifier takes an unlabeled sentence as input and outputs a label: positive or negative.
First get training examples. I'm sure you can find some labeled sentiment data in the public domain. One of the best data set repositories is the UCI KDD Archive. You may then train a classifier on the data to judge new examples. There are a host of learning algorithm resources available. My favorites are jBoost, which can output a classifier as Java code, and Rapidminer, which is better for visual analysis.

You could use an existing web-service which is trained from prior data. For example:
Chatterbox Sentiment Detection API
Which has libraries for Java & Android.
(Disclosure: I work for the company that builds this API)

This is not really programming related (neuro-linguistic programming is not programming), and in general there is no reliable solution.
My best idea is to make it work like Google "Pigeon"Rank, i.e. collect words and sentences, and then collect human feedback whether they are positive or negative, and then use Bayesian matching with this data.

Your can try to use Wordnet for searching word's Semantic Orientation based on "distance" calculation between your word and "good" or "bad" words.Shorter distance will give you word's SO. Results seems will be a bit weak but not a lot of data(or time) is necessary for this approach.

Where do I start for Text Pattern Recognition - Java Based

I am seriously considering doing a Optical Character Recognition program. I am well versed with Java and would love to know about libraries available out there. Basically, I want to convert something like the following to text. I will need to give manual interruption to specify a pattern. For example, I would need to ask user to mark f in this text, so that I know where f occurs.
I am a newbie to this entirely, so I dont mind learning from scratch as well. Need guidance.

If you are thinking of coding an OCR program from scratch, reading up on techniques may be useful. I found an OCR Survey from 1996 which reviews some of the popular techniques from a decade and a half ago. Reading that might be helpful; track down papers it cites or papers which cite it.
Usually the process goes as follows:
find text
find characters in the text
extract features from the characters found
do pattern matching
report suspected character
While getting a user to annotate text is fun and exciting, finding a collection of handwriting which is already annotated might save you a lot of time, that way you can focus on the nuts and bolts of doing OCR rather than building your own database of annotated text.
To start with a slightly easier task you might want to consider building a system to detect handwritten digits. The USPS produced a corpus for developing systems to do this for zip code processing. The link was something I found with a quick search.

If you want to use/look at a library, you could try the Google-endorsed Tesseract.