I have to get 'tags' from the database and store them in an array so I could check if my document contains them. Due to the number of tag categories (customers, system_dependencies, keywords) I have multiple arrays to compare my document with. Is there an easy way to simplify and make my code look nicer?
This is my approach but it looks terrible with all the repetitive for loops.
ArrayList<String> KEYWORDS2 = new ArrayList<String>();
ArrayList<String> CUSTOMERS = new ArrayList<String>();
ArrayList<String> SYSTEM_DEPS = new ArrayList<String>();
ArrayList<String> MODULES = new ArrayList<String>();
ArrayList<String> DRIVE_DEFS = new ArrayList<String>();
ArrayList<String> PROCESS_IDS = new ArrayList<String>();
while (resultSet2.next()) {
CUSTOMERS.add(resultSet2.getString(1));
}
sql = "SELECT da_tag_name FROM da_tags WHERE da_tag_type_id = 6";
stmt = conn.prepareStatement(sql);
resultSet2 = stmt.executeQuery();
while (resultSet2.next()) {
SYSTEM_DEPS.add(resultSet2.getString(1));
}
while (resultSet.next()) {
String da_document_id = resultSet.getString(1);
String file_name = resultSet.getString(2);
try {
if(file_name.endsWith(".docx") || file_name.endsWith(".docm")) {
System.out.println(file_name);
XWPFDocument document = new XWPFDocument(resultSet.getBinaryStream(3));
XWPFWordExtractor wordExtractor = new XWPFWordExtractor(document);
//Return what's inside the document
System.out.println("Keywords found in the document:");
for (String keyword : KEYWORDS) {
if (wordExtractor.getText().contains(keyword)) {
System.out.println(keyword);
}
}
System.out.println("\nCustomers found in the document:");
for (String customer : CUSTOMERS) {
if (wordExtractor.getText().contains(customer)) {
System.out.println(customer);
}
}
System.out.println("\nSystem dependencies found in the document:");
for (String systemDeps : SYSTEM_DEPS) {
if (wordExtractor.getText().contains(systemDeps)) {
System.out.println(systemDeps);
}
}
System.out.println("Log number: " + findLogNumber(wordExtractor));
System.out.println("------------------------------------------");
wordExtractor.close();
}
As you can see there are 3 more to come and this doesn't look good already. Maybe there's a way to compare all of them at the same time.
I have made another attempt at this creating this method:
public void genericForEachLoop(ArrayList<String> al, POITextExtractor te) {
for (String item : al) {
if (te.getText().contains(item)) {
System.out.println(item);
}
}
}
Then calling it like so: genericForEachLoop(MODULES, wordExtractor);
Any better solutions?
I've got two ideas to shorten this: first of all you can write a general for-loop in a separate method that has an ArrayList as a parameter. Then you pass it each of your ArrayLists successively, which would mean that at least you do not have to repeat the for-loops. Secondly, you can create an ArrayList of type ArrayList and store your ArrayLists inside it. Then you can iterate over the whole thing. Only apparent disadvantage of both ideas (or a combination of them) would be, that you need to name the variable for your query string alike for the search of each ArrayList.
What you could do is use a Map and an enum like this:
enum TagType {
KEYWORDS2(2), // or whatever its da_tag_type_id is
CUSTOMERS(4),
SYSTEM_DEPS(6),
MODULES(8),
DRIVE_DEFS(10),
PROCESS_IDS(12);
public final daTagTypeId; // this will be used in queries
TagType(int daTagTypeId) {
this.daTagTypeId = daTagTypeId;
}
}
Map<TagType, List<String>> tags = new HashMap<>();
XWPFDocument document = new XWPFDocument(resultSet.getBinaryStream(3));
XWPFWordExtractor wordExtractor = new XWPFWordExtractor(document);
for(TagType tagType : TagType.values()) {
tags.put(tagType, new ArrayList<>()); // initialize
String sql = String.format("SELECT da_tag_name FROM da_tags WHERE da_tag_type_id = %d", tagType.daTagTypeId); // build query
stmt = conn.prepareStatement(sql);
resultSet2 = stmt.executeQuery();
while(resultSet2.next()) { // fill from DB
tags.get(tagType).add(.add(resultSet2.getString(1)));
}
System.out.println(String.format("%s found in the document:", tags.get(tagType).name());
for (String tag : tags.get(tagType)) { // search in text
if (wordExtractor.getText().contains(tag)) {
System.out.println(keyword);
}
}
}
But at this point I'm not sure you need those lists at all:
enum TagType {
KEYWORDS2(2), // or whatever its da_tag_type_id is
CUSTOMERS(4),
SYSTEM_DEPS(6),
MODULES(8),
DRIVE_DEFS(10),
PROCESS_IDS(12);
public final daTagTypeId; // this will be used in queries
TagType(int daTagTypeId) {
this.daTagTypeId = daTagTypeId;
}
}
XWPFDocument document = new XWPFDocument(resultSet.getBinaryStream(3));
XWPFWordExtractor wordExtractor = new XWPFWordExtractor(document);
for(TagType tagType : TagType.values()) {
String sql = String.format("SELECT da_tag_name FROM da_tags WHERE da_tag_type_id = %d", tagType.daTagTypeId); // build query
stmt = conn.prepareStatement(sql);
resultSet2 = stmt.executeQuery();
System.out.println(String.format("%s found in the document:", tags.get(tagType).name());
while(result2.next()) {
String tag = resultSet2.getString(1);
if (wordExtractor.getText().contains(tag)) {
System.out.println(keyword);
}
}
}
This given I don't know where those resultSet is declared and initialised, nor where that resultSet2 is initialised.
Basically you just fetch tags for each type from DB and then directly search them in the text without storing them at first and then re-iterating the stored ones... I mean that's what the DB is there for.
Related
I'm trying to concatenate 2 array lists of objects into one but i can't figure out how to do it. I've tried with addAll and add but those methods won't really do what i want.
Basically, i have one array list with values like this:
SearchResult1 [title=null, url=null, price=19 690 EUR]
And another one with values like this:
SearchResult2 [title=Ford Car, url=http://www.something.com, price=null]
How can i combine those 2 arrays into one with values like this:
SearchResult3 [title=Ford Car, url=http://www.something.com, price=19 690 EUR]
This is the code so far:
public List searchMethod() {
try {
final String query = "ford";
final Document page = Jsoup.connect("link" + URLEncoder.encode(query, "UTF-8")).userAgent(USER_AGENT).get();
List<SearchResult> resultList1 = new ArrayList<SearchResult>();
List<SearchResult> resultList2 = new ArrayList<SearchResult>();
List<SearchResult> resultList3 = new ArrayList<SearchResult>();
for(Element searchResult : page.select(".offer-price")) {
String price = searchResult.text();
resultList1.add(new SearchResult(price));
}
for(Element searchResult : page.select(".offer-title__link")) {
String title = searchResult.text();
String url = searchResult.attr("href");
resultList2.add(new SearchResult(title, url));
}
resultList3.addAll(resultList1);
resultList3.addAll(resultList2);
return resultList3;
}catch(Exception e) {
e.printStackTrace();
}
return Collections.emptyList();
}
The values that i put in those arrays are extracted from a web page
Thanks for helping!
From the comment, you have said that you just want to correlate/merge the objects from both lists by each index.
You can simply loop through the list, constructing a new SearchResult (assuming you have getters for the fields)
for(int i = 0; i < resultList1.size(); i++) {
resultList3.add(new SearchResult(resultList1.get(i).getPrice(),
resultList2.get(i).getTitle(),
resultList2.get(i).getUrl()));
}
You may have to change the order of the passed arguments to the SearchResult constructor taking price, title and url as you haven't shown it.
why don't you do it in one shot?
List<SearchResult> resultList1 = new ArrayList<SearchResult>();
for(Element searchResult : page.select(".offer-title__link")) {
String title = searchResult.text();
String url = searchResult.attr("href");
resultList1.add(new SearchResult(title, url));
}
int index = 0;
for(Element searchResult : page.select(".offer-price")) {
String price = searchResult.text();
//since you have already assumed
//that price will come in the same order and title and url.
resultList1.get(index++).setPrice(price);
}
return resultList1;
My question is when i store the data into array from sqlite database, how can i get it from specific position let say, my database contain "food, drinks,snack" how can i get the string "snack" from array.
String CatNameQuery = "SELECT * FROM Category";
db = new DBController(MainActivity.this);
SQLiteDatabase db3 = db.getReadableDatabase();
final Cursor cursor2 = db3.rawQuery(CatNameQuery, null);
{
List<String> array = new ArrayList<String>();
while(cursor2.moveToNext()){
String uname = cursor2.getString(cursor2.getColumnIndex("CategoryName"));
array.add(uname);
}
You need to iterate through the list in order to find the item you are looking for.
For example:
for (String s : array) {
if (s.equals("snack")) {
System.out.println("Found snack");
}
}
You can also use the contains method to check if the list contains "snack."
if (array.contains("snack")) {
System.out.println("Found snack");
}
Resource: ArrayList
Use the WHERE clause within your SELECT query. For example:
"SELECT * FROM Category WHERE CategoryName='snacks'"
This will fill your array with only items under the category 'snacks'.
List<String> array = new ArrayList<String>();
array.add("food");
array.add("drinks");
array.add("snack");
String result="";
if (array.contains("snack")) // avoid null pointer exception
{
int index =array.indexOf("snack") //find the index of arraylist
result=array.get(index);
}
You can find it by looping the array
List<String> arrobj= new ArrayList<String>();
arrobj.add("food");
arrobj.add("drinks");
arrobj.add("snack");
for (String value : arrobj) {
if (value.equals("snack")) {
System.out.println("Here is the snack");
}
}
if (array.size() > 0) {
int index = 0;
if (array.contains("Snacks")) {
index = array.indexOf("Snacks");
System.out.println(array.get(index));
}
}
I want to modify an existing *.rptdesign file and save it under a new name.
The existing file contains a Data Set with a template SQL select statement and several DS parameters.
I'd like to use an actual SQL select statement which uses only part of the DS parameters.
However, the following code results in the exception:
Exception in thread "main" `java.lang.RuntimeException`: *The structure is floating, and its handle is invalid!*
at org.eclipse.birt.report.model.api.StructureHandle.getStringProperty(StructureHandle.java:207)
at org.eclipse.birt.report.model.api.DataSetParameterHandle.getName(DataSetParameterHandle.java:143)
at org.eclipse.birt.report.model.api.DataSetHandle$DataSetParametersPropertyHandle.removeParamBindingsFor(DataSetHandle.java:851)
at org.eclipse.birt.report.model.api.DataSetHandle$DataSetParametersPropertyHandle.removeItems(DataSetHandle.java:694)
--
OdaDataSetHandle dsMaster = (OdaDataSetHandle) report.findDataSet("Master");
HashSet<String> bindVarsUsed = new HashSet<String>();
...
// find out which DS parameters are actually used
HashSet<String> bindVarsUsed = new HashSet<String>();
...
ArrayList<OdaDataSetParameterHandle> toRemove = new ArrayList<OdaDataSetParameterHandle>();
for (Iterator iter = dsMaster.parametersIterator(); iter.hasNext(); ) {
OdaDataSetParameterHandle dsPara = (OdaDataSetParameterHandle)iter.next();
String name = dsPara.getName();
if (name.startsWith("param_")) {
String bindVarName = name.substring(6);
if (!bindVarsUsed.contains(bindVarName)) {
toRemove.add(dsPara);
}
}
}
PropertyHandle paramsHandle = dsMaster.getPropertyHandle( OdaDataSetHandle.PARAMETERS_PROP );
paramsHandle.removeItems(toRemove);
What is wrong here?
Has anyone used the DE API to remove parameters from an existing Data Set?
I had similar issue. Resolved it by calling 'removeItem' multiple times and also had to re-evaluate parametersIterator everytime.
protected void updateDataSetParameters(OdaDataSetHandle dataSetHandle) throws SemanticException {
int countMatches = StringUtils.countMatches(dataSetHandle.getQueryText(), "?");
int paramIndex = 0;
do {
paramIndex = 0;
PropertyHandle odaDataSetParameterProp = dataSetHandle.getPropertyHandle(OdaDataSetHandle.PARAMETERS_PROP);
Iterator parametersIterator = dataSetHandle.parametersIterator();
while(parametersIterator.hasNext()) {
Object next = parametersIterator.next();
paramIndex++;
if(paramIndex > countMatches) {
odaDataSetParameterProp.removeItem(next);
break;
}
}
if(paramIndex < countMatches) {
paramIndex++;
OdaDataSetParameter dataSetParameter = createDataSetParameter(paramIndex);
odaDataSetParameterProp.addItem(dataSetParameter);
}
} while(countMatches != paramIndex);
}
private OdaDataSetParameter createDataSetParameter(int paramIndex) {
OdaDataSetParameter dataSetParameter = StructureFactory.createOdaDataSetParameter();
dataSetParameter.setName("param_" + paramIndex);
dataSetParameter.setDataType(DesignChoiceConstants.PARAM_TYPE_INTEGER);
dataSetParameter.setNativeDataType(1);
dataSetParameter.setPosition(paramIndex);
dataSetParameter.setIsInput(true);
dataSetParameter.setIsOutput(false);
dataSetParameter.setExpressionProperty("defaultValue", new Expression("<evaluation script>", ExpressionType.JAVASCRIPT));
return dataSetParameter;
}
How to load values from an Oracle db to a JComboBox to make it easier for the user To Choose from I Have tried this:
Connection dbcon = null;
try {
Class.forName("oracle.jdbc.driver.OracleDriver");
dbcon = DriverManager.getConnection("
jdbc:oracle:thin:#localhost:1521:XE", "USERNAME", "PASSWORD");
Statement st = dbcon.createStatement();
String combo = "Select EMP_IDNUM from employees";
ResultSet res = st.executeQuery(combo);
while(res.next()){
String ids = res.getString("EMP_IDNUM");
String [] str = new String[]{ids};
cboId = new JComboBox(str);
}
} catch (Exception d) {
System.out.println(d);
}
This is Only getting me the first Value Into the JComboBox cboID. What Is the Best way to Load the entire Field Data (EMP_IDNUM) into the Jcombobox??
String [] str = new String[]{ids};
It means your String array is having only one ids value which you have loaded String ids = res.getString("EMP_IDNUM");
if(rs.getRow()>0){
String [] str = new String[res.getRow()];
int i=0;
while(res.next()){
str[i++] = res.getString("EMP_IDNUM");
}
}
JComboBox jcb = new JComboBox(str);
Instead of array you can use Vector also to create JComboBox.
you can use this code:
Vector v = new Vector();
while(res.next()){
String ids = res.getString("EMP_IDNUM");
v.add(ids)
}
JComboBox jcb = new JComboBox(v);
In this code you create a Vector with your Strings, and then invoke directly the JComboBox(Vector items) constructor of JComboBox
there are three important areas
a) close all JDBC Objects in the finally block, because these Object aren't, never GC'ed
try {
} catch (Exception d) {
System.out.println(d);
} finally {
try {
st.close()
res.close()
dbcon.close()
} catch (Exception d) {
//not important
}
}
b) don't to create any Objects inside try - catch - finally, prepare that before
meaning cboId = new JComboBox(str);
c) put all data from JDBC to the ComboBoxModel, prepare that before
i have a collection of raw text in a table in database, i need to replace some words in this collection using a set of words.
i put all the term to be replace and its substitutes in a text file as below
min=admin
lelet=lambat
lemot=lambat
nii=nih
ntu=itu
and so on.
i have successfully initiate a variabel of File and Scanner to read the collection of the term and its substitutes.
i loop all the dataset and save the raw text in a string
in the same loop
i loop all the term collection and save its row to a string name 'pattern', and split the pattern into two string named 'term' and 'replacer'
in this loop i initiate a new string which its value is the string from the dataset modified by replaceAll(term,replacer)
end loop for term collection
then i insert the new string to another table in database
end loop for dataset
i do it manualy as below
replaceAll("min","admin")
and its works but its really something to code it manually for almost 2000 terms to be replace it.
anyone ever face this kind of really something..
i really need a help now desperate :(
package sentimenrepo;
import javax.swing.*;
import java.sql.*;
import java.io.*;
//import java.util.HashMap;
import java.util.Scanner;
//import java.util.Map;
/**
*
* #author herman
*/
public class synonimReplaceV2 extends SwingWorker {
protected Object doInBackground() throws Exception {
new skripsisentimen.sentimenttwitter().setVisible(true);
Integer row = 0;
File synonimV2 = new File("synV2/catatan_kata_sinonim.txt");
String newTweet = "";
DB db = new DB();
Connection conn = db.dbConnect("jdbc:mysql://localhost:3306/tweet", "root", "");
try{
Statement select = conn.createStatement();
select.executeQuery("select * from synonimtweet");
ResultSet RS = select.getResultSet();
Scanner scSynV2 = new Scanner(synonimV2);
while(RS.next()){
row++;
String no = RS.getString("no");
String tweet = " "+ RS.getString("tweet");
String published = RS.getString("published");
String label = RS.getString("label");
clean2 cleanv2 = new clean2();
newTweet = cleanv2.cleanTweet(tweet);
try{
Statement insert = conn.createStatement();
insert.executeUpdate("INSERT INTO synonimtweet_v2(no,tweet,published,label) values('"
+no+"','"+newTweet+"','"+published+"','"+label+"')");
String current = skripsisentimen.sentimenttwitter.txtAreaResult.getText();
skripsisentimen.sentimenttwitter.txtAreaResult.setText(current+"\n"+row+"original : "+tweet+"\n"+newTweet+"\n______________________\n");
skripsisentimen.sentimenttwitter.lblStat.setText(row+" tweet read");
skripsisentimen.sentimenttwitter.txtAreaResult.setCaretPosition(skripsisentimen.sentimenttwitter.txtAreaResult.getText().length() - 1);
}catch(Exception e){
skripsisentimen.sentimenttwitter.lblStat.setText(e.getMessage());
}
skripsisentimen.sentimenttwitter.lblStat.setText(e.getMessage());
}
}catch(Exception e){
skripsisentimen.sentimenttwitter.lblStat.setText(e.getMessage());
}
return row;
}
class clean2{
public clean2(){}
public String cleanTweet(String tweet){
File synonimV2 = new File("synV2/catatan_kata_sinonim.txt");
String pattern = "";
String term = "";
String replacer = "";
String newTweet="";
try{
Scanner scSynV2 = new Scanner(synonimV2);
while(scSynV2.hasNext()){
pattern = scSynV2.next();
term = pattern.split("=")[0];
replacer = pattern.split("=")[1];
newTweet = tweet.replace(term, replacer);
}
}catch(Exception e){
e.printStackTrace();
}
System.out.println(newTweet+"\n"+tweet);
return newTweet;
}
}
}
update
ive just realize that the code actually works but only for the first row in database, the second row and so on stand still. here is i update the newest code i ve build
public class synonimReplaceV2 extends SwingWorker {
protected Object doInBackground() throws Exception {
new skripsisentimen.sentimenttwitter().setVisible(true);
Integer row = 0;
String newTweet = "";
DB db = new DB();
Connection conn = db.dbConnect("jdbc:mysql://localhost:3306/tweet", "root", "");
try{
Statement select = conn.createStatement();
select.executeQuery("select * from synonimtweet limit 2,10");
ResultSet RS = select.getResultSet();
FileReader readSyn = new FileReader("synV2/catatan_kata_sinonim.txt");
BufferedReader buffSyn = new BufferedReader(readSyn);
while(RS.next()){
row++;
String no = RS.getString("no");
String tweet = " "+ RS.getString("tweet");
String published = RS.getString("published");
String label = RS.getString("label");
String pattern = "";
while((pattern=buffSyn.readLine())!=null){
String patternTerm = pattern.split("=")[0];
String patternSubs = pattern.split("=")[1];
tweet = tweet.replaceAll("\\s"+patternTerm, patternSubs);
}
try{
Statement insert = conn.createStatement();
insert.executeUpdate("INSERT INTO synonimtweet_v2(no,tweet,published,label) values('"
+no+"','"+tweet+"','"+published+"','"+label+"')");
String current = skripsisentimen.sentimenttwitter.txtAreaResult.getText();
skripsisentimen.sentimenttwitter.txtAreaResult.setText(current+"\n"+row+"original : "+tweet+"\n"+newTweet+"\n______________________\n");
skripsisentimen.sentimenttwitter.lblStat.setText(row+" tweet read");
skripsisentimen.sentimenttwitter.txtAreaResult.setCaretPosition(skripsisentimen.sentimenttwitter.txtAreaResult.getText().length() - 1);
}catch(Exception e){
skripsisentimen.sentimenttwitter.lblStat.setText(e.getMessage());
}
}
}catch(Exception e){
skripsisentimen.sentimenttwitter.lblStat.setText(e.getMessage());
// System.out.println(e.getMessage());
}
Thread.sleep(100);
return row;
}
}
Opening the synonym file and iterating over 2,000 lines for every row in your ResultSet is a bit wasteful.
Load your synonyms into an in-memory Map once, keyed by unique misspelt term, then do a lookup on the map for every row in your result set, and replace as necessary.
Let us use both solutions to build a single solution for you:
First, you create a HashMap with all your keys:
public static HashMap<String, String> getMap() {
//your version would read from the file
HashMap<String,String> myMap=new HashMap<String,String>();
myMap.put("min", "admin");
myMap.put("lelet", "lambat");
myMap.put("lemot", "lambat");
myMap.put("nii", "nih");
myMap.put("ntu", "itu");
return(myMap);
}
Second, you create a pattern that contains all the keys in your hashmap:
public static String getPattern(HashMap<String,String> mapReplacement) {
String pattern="";
for (String s : mapReplacement.keySet()) {
if (!pattern.isEmpty()) {
pattern=pattern+"|";
}
pattern=pattern+s;
}
return(pattern);
}
Next, you can create a cleanTweet method that uses both structures you created:
public static String cleanTweet(String tweet, Pattern pattern,HashMap<String, String> myMap) {
String newTweet=tweet;
Matcher matcher = pattern.matcher(newTweet);
int start=0;
while (matcher.find()) {
String key=matcher.group();
String replacement=myMap.get(key);
if (replacement!=null) {
newTweet=newTweet.replace(key, replacement );
}
}
return(newTweet);
}
This might require some tweaking to perfect (I onyl tested a few cases), but the point is that you are going to iterate a single time in your keys and then iterate only on your tweets.
I hope it helps.
I didn't try, but it seems to me that you've almost got it - just replace this line:
newTweet = tweet.replace(term, replacer);
with this:
tweet = tweet.replaceAll(term, replacer);
As you're not using newTweet any more, return tweet:
return tweet;
You should also delete the newTweet declaration.
Also, you shouldn't read Scanner to read lines. Use FileReader instead.
thanks folks
i ve found the answer why the code is not working,
the txt file containing terms and its substitutes should be initiated each time the program read a row from database.
the code would be like this
public class synonimReplaceV2 extends SwingWorker {
protected Object doInBackground() throws Exception {
new skripsisentimen.sentimenttwitter().setVisible(true);
Integer row = 0;
String newTweet = "";
DB db = new DB();
Connection conn = db.dbConnect("jdbc:mysql://localhost:3306/tweet", "root", "");
try{
Statement select = conn.createStatement();
select.executeQuery("select * from synonimtweet limit 2,10");
ResultSet RS = select.getResultSet();
while(RS.next()){
row++;
FileReader readSyn = new FileReader("synV2/catatan_kata_sinonim.txt");
BufferedReader buffSyn = new BufferedReader(readSyn);
String no = RS.getString("no");
String tweet = " "+ RS.getString("tweet");
String published = RS.getString("published");
String label = RS.getString("label");
String pattern = "";
while((pattern=buffSyn.readLine())!=null){
String patternTerm = pattern.split("=")[0];
String patternSubs = pattern.split("=")[1];
tweet = tweet.replaceAll("\\s"+patternTerm, patternSubs);
}
try{
Statement insert = conn.createStatement();
insert.executeUpdate("INSERT INTO synonimtweet_v2(no,tweet,published,label) values('"
+no+"','"+tweet+"','"+published+"','"+label+"')");
String current = skripsisentimen.sentimenttwitter.txtAreaResult.getText();
skripsisentimen.sentimenttwitter.txtAreaResult.setText(current+"\n"+row+"original : "+tweet+"\n"+newTweet+"\n______________________\n");
skripsisentimen.sentimenttwitter.lblStat.setText(row+" tweet read");
skripsisentimen.sentimenttwitter.txtAreaResult.setCaretPosition(skripsisentimen.sentimenttwitter.txtAreaResult.getText().length() - 1);
}catch(Exception e){
skripsisentimen.sentimenttwitter.lblStat.setText(e.getMessage());
}
}
}catch(Exception e){
skripsisentimen.sentimenttwitter.lblStat.setText(e.getMessage());
// System.out.println(e.getMessage());
}
Thread.sleep(100);
return row;
}
}
but im actually want to apply the code in which rlinden made above, but cant figure it out how to call the cleanTweet function.