I am currently calculating values to fill a database of 15 milion records. The first 7 mill went just fin,e however now my update query starts giving problems :
Now & then a random letter changes into some jibberish.
In java I generate the query doing :
String updateSql = "UPDATE VanNaar SET time = CASE ID ";
for (int i = 0; i < routes.size(); i++) {
updateSql += " WHEN " + routes.get(i).ID + " THEN " + routes.get(i).driveTime;
}
updateSql += " END, ";
updateSql += " distance = CASE ID ";
for (int i = 0; i < routes.size(); i++) {
updateSql += " WHEN " + routes.get(i).ID + " THEN " + routes.get(i).distance;
}
updateSql += " END WHERE id IN (";
for (int i = 0; i < routes.size(); i++) {
updateSql += routes.get(i).ID + ",";
}
updateSql = updateSql.substring(0, updateSql.length() - 1);
updateSql += ");";
Which works just fine, as mentioned before. Here is what Java trows at me now:
...MySQL server version for the right syntax to use near '×HEN 8284022 THEN 999.999 WHEN 8284023 THEN 3791.0 WHEN 8284024 THEN 378...
Or
...MySQL server version for the right syntax to use near 'WÈEN 7468574 THEN 2273.0 WHEN 7468575 THEN 2410.0 WHEN 7468576 THEN 2472.0 W' at line 1
Notice the weirdisch Ã^ or Ã- , a final exmpale, mind you the bold tekst:
...MySQL server version for the right syntax to use near **'Â** WHEN 7228125 THEN 48590.0 WHEN 7228126 THEN 47910.0 WHEN 7228127 THEN...
...
Update:
It seems to be getting worse..:
Unknown column '9°22331' in 'where clause'
I'm no Java expert but shouldn't you be using a StringBuilder in the first place? Possibly even use a prepared statement? You could buld the prepared statement with a stringbuilder but instead of
updateSql += " WHEN " + routes.get(i).ID + " THEN " + routes.get(i).driveTime;
everywhere you'd do something like
myStrngBldr.append(" WHEN ? THEN ?");
or
myStrngBldr.append(" WHEN #foo1 THEN #foo2");
if named parameters are supported (don't know) and later add the actual parameters:
myPrepdStmt = myConn.prepareStatement(myStrngBldr.toString());
myPrepdStmt.setInt(1, routes.get(i).ID);
myPrepdStmt.setFloat(2, routes.get(i).driveTime);
...
myPrepdStmt.executeUpdate();
This page should help you.
What is actually causing the 'strange malformed strings': I don't know. My best guess would be the you'd have to use something like .ToString() on all those ID's and other non-string values since you're concatenating a string. Maybe, somehow, the values are interpreted as charcodes (because they're not explicitly casted as string) and thus causing weird characters.
Another guess would be: are you actually building 15 million queries in-memory before sending them to the database? Or does each query get sent to the DB seperately? Maybe the fact that you're trying to store a huge-ass string in-memory causes some problems (although it shouldn't cause the problem you're describing here).
You might want to consider replacing the gigantic update with two CASE selectors and one IN with individual UPDATE statements for each row. I know that the problem is not caused by this, but it is probably a cleaner and more efficient solution. If your DB connector supports multiple statements per execution, you could do something like the following:
int batchSize = 0;
StringBuilder sb = new StringBuilder();
for (Route r: routes) {
sb.append("UPDATE VanNaar SET")
.append(" time = '").append(r.driveTime).append("',")
.append(" distance = '").append(r.distance).append("'")
.append(" WHERE ID = '").append(r.ID).append("'; ");
if (++batchSize == BATCH_SIZE) {
connector.exec(sb.toString());
sb = new StringBuilder();
batchSize = 0;
}
}
if (sb.length() > 0) {
connector.exec(sb.toString());
}
This will let StringBuilder take care of transforming whatever the underlying value type is into a String. If driveTime, distance, or ID are already strings, then you will have to escape them properly using the appropriate JDBC methods.
You'd probably be better off using prepared statements for this sort of thing as suggested by #RobIII since they would take care of the SQL injection problems automatically.
Related
I'm trying to get query results from realm:
RealmResults<Line> results = realm.where(Line.class).lessThan("id",5).findAll();
Line[] lines = results.toArray(new Line[results.size()]);
for(int i=0;i<lines.length;i++)
Log.e("erralert", String.valueOf(lines[i].id) + "-" + lines[i].text + "-" + lines[i].localId);
the results are like:
0-null-0
0-null-0
0-null-0
0-null-0
But actually, I'm sure that the data in the database are correct and changing query .less than("id",5) shows that it has actually found 4 items (those with id less than 5). changing 5 to 10, returns 9 results, but the taken results are always empty.
The following code also have the same result:
RealmResults<Line> lines = realm.where(Line.class).lessThan("id",20).findAll();
for(int i=0;i<lines.size();i++)
Log.e("erralert", String.valueOf(lines.get(i).id) + "-" + lines.get(i).text + "-" + lines.get(i).localId);
P.S.: id field is Primary Key, so obviously we can't have records like that.
for reasons, i must use tfs java sdk, so i always need workitem.open() to load revisions for each workitem from workitemclient.Query(),like below code:
WorkItemCollection workItemCollection = workItemClient.query(wiql);
for(int i = 0; i < workItemCollection.size(); i++)
{
WorkItem workItem = workItems.get(i);
workItem.open();
workItem.getRevisions()
.forEach(r -> {
//..."System.History").getValue());
//..."System.History").getOriginalValue());
});
}
if don't use open(), Revisions must be empty;
that open() method cost 55 millseconds for each workitem,that's unacceptable。
so any suggestion for better efficiency ？ Thanks a lot。
/////////////////////////////////////////////////////////////////////make it/////////////////////////////////////////////////////////////////////
i make it that cut the time.
use SQL like blow :
String sql = "select [System.Id],[System.ChangedBy],[System.ChangedDate],[words]"
+ " from Tfs_DefaultCollection.dbo.WorkItemsEverable"
+ " left join Tfs_DefaultCollection.dbo.WorkItemLongTexts on [System.Id] = [ID] and [System.Rev] = [Rev]"
+ " where FldID=54" // behalf History field
+ " and"+conditionSql;
conditionSql : " id in ('12','1324','1')"
now use this method, average cost mills for retrieve history(changedby,changedDate,History) just 2~4 mills（test case is 30 ~ 1000 workitems）
/////////////////////////////////////////////////////////////////////still/////////////////////////////////////////////////////////////////////
any advise for better code/efficiency would be appreciate。
The revision history can only be retrieved for work item fields available in the collection WorkItem.Fields, so if you used the below code to get the workitem revision history you will NOT see the history but end up reloading the current workitem object again and again.
foreach (Revision revision in wi.Revisions)
{
Debug.Write(revision.WorkItem);
}
You may try to get value of fields in the work item revision:
// Returns a work item object
var wi = GetWorkItemDetails(id);
// Get All work item revisions
foreach (Revision revision in wi.Revisions)
{
// Get value of fields in the work item revision
foreach (Field field in wi.Fields)
{
Debug.Write(revision.Fields[field.Name].Value);
}
}
Helpful blog: http://geekswithblogs.net/TarunArora/archive/2011/08/21/tfs-sdk-work-item-history-visualizer-using-tfs-api.aspx
I am really new to SOAP web services and to Netsuite ERP and I am trying to generate a report in my company where I need to obtain all the Clients and their Invoices using the data available in Netsuite ERP. I followed the Java and Axis tutorial they offer with their sample app for the ERP and I successfully created a Java project in Eclipse that consumes the WSDL for netsuite 2015-2 and compiles the needed classes to run the sample app. So, I followed an example found in their CRM exapmle app to obtain a Client's information but the only problem is that their example method needs you to introduce the Client's ID. Here is the sample code:
public int getCustomerList() throws RemoteException,
ExceededUsageLimitFault, UnexpectedErrorFault, InvalidSessionFault,
ExceededRecordCountFault, UnsupportedEncodingException {
// This operation requires a valid session
this.login(true);
// Prompt for list of internalIds and put in an array
_console
.write("\ninternalIds for records to retrieved (separated by commas): ");
String reqKeys = _console.readLn();
String[] internalIds = reqKeys.split(",");
return getCustomerList(internalIds, false);
}
private int getCustomerList(String[] internalIds, boolean isExternal)
throws RemoteException, ExceededUsageLimitFault,
UnexpectedErrorFault, InvalidSessionFault, ExceededRecordCountFault {
// Build an array of RecordRef objects and invoke the getList()
// operation to retrieve these records
RecordRef[] recordRefs = new RecordRef[internalIds.length];
for (int i = 0; i < internalIds.length; i++) {
RecordRef recordRef = new RecordRef();
recordRef.setInternalId(internalIds[i]);
recordRefs[i] = recordRef;
recordRefs[i].setType(RecordType.customer);
}
// Invoke getList() operation
ReadResponseList getResponseList = _port.getList(recordRefs);
// Process response from get() operation
if (!isExternal)
_console.info("\nRecords returned from getList() operation: \n");
int numRecords = 0;
ReadResponse[] getResponses = getResponseList.getReadResponse();
for (int i = 0; i < getResponses.length; i++) {
_console.info("\n Record[" + i + "]: ");
if (!getResponses[i].getStatus().isIsSuccess()) {
_console.errorForRecord(getStatusDetails(getResponses[i]
.getStatus()));
} else {
numRecords++;
Customer customer = (Customer) getResponses[i].getRecord();
_console.info(" internalId="
+ customer.getInternalId()
+ "\n entityId="
+ customer.getEntityId()
+ (customer.getCompanyName() == null ? ""
: ("\n companyName=" + customer
.getCompanyName()))
+ (customer.getEntityStatus() == null ? ""
: ("\n status=" + customer.getEntityStatus().getName()))
+ (customer.getEmail() == null ? ""
: ("\n email=" + customer.getEmail()))
+ (customer.getPhone() == null ? ""
: ("\n phone=" + customer.getPhone()))
+ "\n isInactive="
+ customer.getIsInactive()
+ (customer.getDateCreated() != null ? ""
: ("\n dateCreated=" + customer
.getDateCreated().toString())));
}
}
return numRecords;
}
So as you can see, this method needs the internal ID of each Customer which I find not useful as I have a many Customers and I don't want to pass each Customer's ID. I read their API docs (which I find hard to navigate and kind of useless) and I found a web service called getAll() that gives all the records given a getAllRecord object which requires a getAllRecordType object. However, the getAllRecordType object does not support Customer entities, so I can't obtain all the customers on the ERP this way.
Is there an easy way to obtain all the Customers in my Netsuite ERP (maybe using other thing rather than the SOAP Web Services they offer? I am desperate about this situation as understanding how Netsuite's Web Services API has been really troublesome.
Thanks!
You would normally use a search to select a list of customers. On a large account you would not normally get all customers on any regular basis. If you are trying to get the invoices you might just find it more practical to get those with a search.
You wrote "in your company". Are you trying to write an application of some sort? If this is an internal project (and even if it's not) you'll probably find using SuiteScripts much more efficient in terms of your time and frustration level.
I made it using the following code on my getCustomerList method:
CustomerSearch customerSrch = new CustomerSearch();
SearchResult searchResult = _port.search(customerSrch);
System.out.println(searchResult.getTotalRecords());
RecordList rl = searchResult.getRecordList();
for (int i = 0; i <searchResult.getTotalRecords()-1; i++) {
Record r = rl.getRecord(i);
System.out.println("Customer # " + i);
Customer testcust = (Customer)r;
System.out.println("First Name: " + testcust.getFirstName());
}
I have wirtten a program to manage tv series and I am stuck at an issue with lightcouch and a specific database query. This is what I have so far. To setup the database views I used the following lines:
MapReduce get_numberOfSeasonsMR = new MapReduce();
get_numberOfSeasonsMR.setMap(
"function(doc) { "
+ " emit(doc.seriesName, doc.season)"
+ "}");
get_numberOfSeasonsMR.setReduce(
"function (key, values, rereduce) {"
+ "return Math.max.apply({}, values)"
+ "}");
map.put("get_numberOfSeasons", get_numberOfSeasonsMR);
In Futon everything appears normal (see http://i.stack.imgur.com/1hgSJ.png).
However, when I try to execute the following line, I get an exception, instead of the results that appear in Futon.
int nr = client.view("design/get_numberOfSeasons").key("Arrow").queryForInt();
Exception:
org.lightcouch.NoDocumentException: Expecting exactly a single result of this view query, but was: 0
org.lightcouch.View.queryValue(View.java:246)
org.lightcouch.View.queryForInt(View.java:219)
....db.Server.getNumberOfSeasons(Server.java:237)
...
I tried to emit Strings in my map() function instead on ints, but it did not make any difference. What am I doing wrong? Or can someone post an example of a successful lightcouch map()+reduce() operation? The tutorials I found only used map() without reduce().
Thanks in advance ;)
Nothing seems wrong with your code, here is the full version:
CouchDbClient dbClient = new CouchDbClient();
DesignDocument designDocument = new DesignDocument();
designDocument.setId("_design/mydesign");
designDocument.setLanguage("javascript");
MapReduce get_numberOfSeasonsMR = new MapReduce();
get_numberOfSeasonsMR.setMap(
"function(doc) { "
+ " emit(doc.seriesName, doc.season)"
+ "}");
get_numberOfSeasonsMR.setReduce(
"function (key, values, rereduce) {"
+ "return Math.max.apply({}, values)"
+ "}");
Map<String, MapReduce> view = new HashMap<>();
view.put("get_numberOfSeasons", get_numberOfSeasonsMR);
designDocument.setViews(view);
dbClient.design().synchronizeWithDb(designDocument);
int count = dbClient.view("mydesign/get_numberOfSeasons").key("Arrow").queryForInt();
I believe that most of you would be thinking that this is the same question you have heard multiple times (and answered ) about string concatenation in Java. But trust me, it is different. In fact, so different that I am even hesitant in posting it here. But anyways, here it is. I have some piece of code which goes like:
public void handleSuccess(String result)
{
result = result.trim();
MessageBox.alert("Information","Result after trimming: '" + result + "'");
result = result.substring(result.indexOf('\n') + 1);
MessageBox.alert("Information","Result after substring: '" + result + "'");
String returns = getReturns();
MessageBox.alert("Information","Returns: '" + returns + "'");
String action = getAction();
MessageBox.alert("Information","Action: '" + action + "'");
String finalResult = result + returns + action;
MessageBox.alert("Information","Final result: '" + finalResult + "'");
}
Now the situation here is that, all of these : getReturns(), result and getAction() return non blank values, and in fact the string finalResult contains the concatenated value after the last line is executed.
So, at Line 1, "result" contains "12/03/2013|04-AERTY|". The value of result remains same at end of line 1,2. getReturns() returns value 12.4724. So at end of line 3, finalResult contains "12/03/2013|04-AERTY|12.4724". getAction() returns "expt". So, at end of line 5, finalResult contains "12/03/2013|04-AERTY|12.4724|expt"
This is , when I debug or run the application in eclipse. As soon as build the same application on a UNIX system to generate a "war" file, and deploy the war on a tomcat server, the problem rears it's ugly head. When I run the application on the deployed war, the last line does not contain the concatenated value. So at the end of line 5, finalResult contains just "12/03/2013|04-AERTY|12.4724". I expected it to contain "12/03/2013|04-AERTY|12.4724|expt" as it does while running in eclipse.
I have tried stringbuffer, stringbuilder and the "+" operator as well, but nothing seems to work. I am not even getting an exception.
Can somebody help me in fixing this or at least enlightening me in what I might be doing wrong here?
Just to stress again, the code on eclipse(which is on a windows machine) and UNIX machine are exactly same. I have done a diff on them.
Here is what I get after putting the message-boxes:
Message-box 1: "Result after trimming: '12/03/2013|04-AERTY|'"
Message-box 2: "Result after substring: '12/03/2013|04-AERTY|'"
Message-box 3:"Returns: '12.4724'"
Message-box 4:"Action: '|expt'"
Message-box 5:"Final result: '12/03/2013|04-AERTY|12.4724|expt'"
Message-box 5 output is the one I receive when I execute code using eclipse
When running on deployed war, Message-box 1-4 have the same output as above, but Message-box 5 says: "Final result: '12/03/2013|04-AERTY|12.4724"
It's not clear where the extra "|" is meant to come from - if getAction() just returns expt, the result would be 12/03/2013|04-AERTY|12.4724|expt.
Anyway, I think it's safe to say that string concatenation will be working fine, and something else is wrong. You should add more diagnostics, logging everything:
public void handleSuccess(String result) {
result = result.trim();
log.info("Result after trimming: '" + result + "'");
result = result.substring(result.indexOf('\n') + 1);
log.info("Result after substring: '" + result + "'");
String returns = getReturns();
log.info("Returns: '" + returns + "'");
String action = getAction();
log.info("Action: '" + action + "'");
// It's not clear what this is meant to do. I suggest you remove it and
// use logging instead.
MessageBox.alert("Information", "The selected action is " + action, null);
String finalResult = result + returns + action;
log.info("Final result: '" + finalResult + "'");
I suspect you'll find that action is an empty string in the broken case.
Note that I've added quotes round each of the logged values, very deliberately. That means that if there's some unprintable character at the end of a string which causes problems, you should be able to detect that in the logging.
EDIT: As per the comment thread, when these were turned into message boxes (as it turns out this is running in GWT) it looks like there's something wrong with the early strings, as the closing ' isn't seen in diagnostics, in the broken case. The OP is going to investigate further.