We Keep Coding

Get lines of code in src folder skipping comments and empty lines

Hi I already have something to get the lines of code but still it out puts the count having empty lines and comments counted.
git ls-files | grep "\.java$" | xargs wc -l
Can you modify this to skip comments and blank lines..?
Thanks in advance

Try CLOC, it can lists numbers in great detail.
You need to install CLOC first using syntax brew install cloc
cloc $(git ls-files)
Sample output for reference:
20 text files.
20 unique files.
6 files ignored.
http://cloc.sourceforge.net v 1.62 T=0.22 s (62.5 files/s, 2771.2 lines/s)
-------------------------------------------------------------------------------
Language files blank comment code
-------------------------------------------------------------------------------
Javascript 2 13 111 309
JSON 3 0 0 58
HTML 2 7 12 50
Handlebars 2 0 0 37
CoffeeScript 4 1 4 12
SASS 1 1 1 5
-------------------------------------------------------------------------------
SUM: 14 22 128 471
-------------------------------------------------------------------------------

Most test coverage tools also calculate LOC. For instance, Jacoco spits this out to Jenkins.

How to generate report from huge mongodb document [duplicate]

Using the code:
all_reviews = db_handle.find().sort('reviewDate', pymongo.ASCENDING)
print all_reviews.count()
print all_reviews[0]
print all_reviews[2000000]
The count prints 2043484, and it prints all_reviews[0].
However when printing all_reviews[2000000], I get the error:
pymongo.errors.OperationFailure: database error: Runner error: Overflow sort stage buffered data usage of 33554495 bytes exceeds internal limit of 33554432 bytes
How do I handle this?

You're running into the 32MB limit on an in-memory sort:
https://docs.mongodb.com/manual/reference/limits/#Sort-Operations
Add an index to the sort field. That allows MongoDB to stream documents to you in sorted order, rather than attempting to load them all into memory on the server and sort them in memory before sending them to the client.

As said by kumar_harsh in the comments section, i would like to add another point.
You can view the current buffer usage using the below command over the admin database:
> use admin
switched to db admin
> db.runCommand( { getParameter : 1, "internalQueryExecMaxBlockingSortBytes" : 1 } )
{ "internalQueryExecMaxBlockingSortBytes" : 33554432, "ok" : 1 }
It has a default value of 32 MB(33554432 bytes).In this case you're running short of buffer data so you can increase buffer limit with your own defined optimal value, example 50 MB as below:
> db.adminCommand({setParameter: 1, internalQueryExecMaxBlockingSortBytes:50151432})
{ "was" : 33554432, "ok" : 1 }
We can also set this limit permanently by the below parameter in the mongodb config file:
setParameter=internalQueryExecMaxBlockingSortBytes=309715200
Hope this helps !!!
Note:This commands supports only after version 3.0 +

solved with indexing
db_handle.ensure_index([("reviewDate", pymongo.ASCENDING)])

If you want to avoid creating an index (e.g. you just want a quick-and-dirty check to explore the data), you can use aggregation with disk usage:
all_reviews = db_handle.aggregate([{$sort: {'reviewDate': 1}}], {allowDiskUse: true})
(Not sure how to do this in pymongo, though).

JavaScript API syntax for the index:
db_handle.ensureIndex({executedDate: 1})

In my case, it was necessary to fix nessary indexes in code and recreate them:
rake db:mongoid:create_indexes RAILS_ENV=production
As the memory overflow does not occur when there is a needed index of field.
PS Before this I had to disable the errors when creating long indexes:
# mongo
MongoDB shell version: 2.6.12
connecting to: test
> db.getSiblingDB('admin').runCommand( { setParameter: 1, failIndexKeyTooLong: false } )
Also may be needed reIndex:
# mongo
MongoDB shell version: 2.6.12
connecting to: test
> use your_db
switched to db your_db
> db.getCollectionNames().forEach( function(collection){ db[collection].reIndex() } )

neo4j upload csv UnmarshalException

I am uploading a 100MB csv file to neo4j containing transactional data. I am getting a java error that I cannot seem to trace to a setting or something that I can change.
neo4j-sh (?)$ CREATE CONSTRAINT ON (a:Account) ASSERT a.id IS UNIQUE;
+-------------------+
| No data returned. |
+-------------------+
Constraints added: 1
48 ms
neo4j-sh (?)$ USING PERIODIC COMMIT
> LOAD CSV FROM
> "file:/somepath/findata.csv"
> AS line
> FIELDTERMINATOR ','
> MERGE (a1:Account { id: toString(line[3]) })
> MERGE (a2:Account { id: toString(line[4]) })
> CREATE (a1)-[:LINK { value: toFloat(line[0]), date: line[5] } ]->(a2);
java.rmi.UnmarshalException: Error unmarshaling return header; nested exception is:
java.io.EOFException
at sun.rmi.transport.StreamRemoteCall.executeCall(StreamRemoteCall.java:228)
at sun.rmi.server.UnicastRef.invoke(UnicastRef.java:161)
at java.rmi.server.RemoteObjectInvocationHandler.invokeRemoteMethod(RemoteObjectInvocationHandler.java:194)
at java.rmi.server.RemoteObjectInvocationHandler.invoke(RemoteObjectInvocationHandler.java:148)
at com.sun.proxy.$Proxy1.interpretLine(Unknown Source)
at org.neo4j.shell.impl.AbstractClient.evaluate(AbstractClient.java:110)
at org.neo4j.shell.impl.AbstractClient.evaluate(AbstractClient.java:94)
at org.neo4j.shell.impl.AbstractClient.grabPrompt(AbstractClient.java:74)
at org.neo4j.shell.StartClient.grabPromptOrJustExecuteCommand(StartClient.java:357)
at org.neo4j.shell.StartClient.startRemote(StartClient.java:303)
at org.neo4j.shell.StartClient.start(StartClient.java:175)
at org.neo4j.shell.StartClient.main(StartClient.java:120)
Caused by: java.io.EOFException
at java.io.DataInputStream.readByte(DataInputStream.java:267)
at sun.rmi.transport.StreamRemoteCall.executeCall(StreamRemoteCall.java:214)
... 11 more
I've tried the command twice and it gives me the same error. Sofar google has not helped me to figure out what I can do to circumvent this error. What is happening in neo4j and how could I solve this?

Maybe this isn't the problem, but your path to your CSV file may be malformed. This would explain the java.rmi.UnmarshalException. The path should be 'file://', where would be something like '/home/cantdutchthis/findata.csv' on a linux system. On a linux or Mac machine, this means that there will be 3 '/'s - 'file:///home/cantdutchthis/findata.csv'.
Grace and peace,
Jim

Replication of solr Lucene index failing with following error message

I am getting below error when running solr replication
2013-12-27 05:03:32,391 [explicit-fetchindex-cmd] ERROR org.apache.solr.handler.ReplicationHandler- SnapPull failed :org.apache.solr.common.SolrException: Index fetch failed :
at org.apache.solr.handler.SnapPuller.fetchLatestIndex(SnapPuller.java:485)
at org.apache.solr.handler.ReplicationHandler.doFetch(ReplicationHandler.java:319)
at org.apache.solr.handler.ReplicationHandler$1.run(ReplicationHandler.java:220)
Caused by: java.io.EOFException: read past EOF: MMapIndexInput(path="/apps/search/data/customers/solr/solr/adidas-archive/data/index.20131227050332242/segments_a")
at org.apache.lucene.store.ByteBufferIndexInput.readByte(ByteBufferIndexInput.java:78)
at org.apache.lucene.store.ChecksumIndexInput.readByte(ChecksumIndexInput.java:41)
at org.apache.lucene.store.DataInput.readInt(DataInput.java:84)
at org.apache.lucene.index.SegmentInfos.read(SegmentInfos.java:320)
at org.apache.lucene.index.SegmentInfos$1.doBody(SegmentInfos.java:380)
at org.apache.lucene.index.SegmentInfos$FindSegmentsFile.run(SegmentInfos.java:812)
at org.apache.lucene.index.SegmentInfos$FindSegmentsFile.run(SegmentInfos.java:663)
at org.apache.lucene.index.SegmentInfos.read(SegmentInfos.java:376)
at org.apache.lucene.index.IndexWriter.<init>(IndexWriter.java:711)
at org.apache.solr.update.SolrIndexWriter.<init>(SolrIndexWriter.java:77)
at org.apache.solr.update.SolrIndexWriter.create(SolrIndexWriter.java:64)
at org.apache.solr.update.DefaultSolrCoreState.createMainIndexWriter(DefaultSolrCoreState.java:267)
at org.apache.solr.update.DefaultSolrCoreState.newIndexWriter(DefaultSolrCoreState.java:179)
at org.apache.solr.update.DirectUpdateHandler2.newIndexWriter(DirectUpdateHandler2.java:632)
at org.apache.solr.handler.SnapPuller.fetchLatestIndex(SnapPuller.java:469)
... 2 more
My step-up is Master on 3.x and Slave on 4.x.
This happened when I copied very large index 100 g+. What does this error means ? Does it means that index has got corrupt and if so what can be done to fix it, any thoughts ?
I ran the checkindex utility, but is again giving the error :
ERROR: could not read any segments file in directory
java.io.EOFException: read past EOF: MMapIndexInput(path="/apps/search/data/customers/solr/solr/adidas-archive/data/index.20131227051833263/segments_a") at org.apache.lucene.store.ByteBufferIndexInput.readByte(ByteBufferIndexInput.java:78)
at org.apache.lucene.store.ChecksumIndexInput.readByte(ChecksumIndexInput.java:41)

while upgrading the version of solr3.x to 4.x to upgrade you can use any of the following methods
1 . rebuild the index from scratch (time taking)
OR
2. optimize the index
But What i found in upgrading from 3.6 to 4.2 was that if your index size in more than 100 gigs (mine was over 170 gigs in solr 3.6 ) .It would be more fruitful to rebuild the index from scratch as new compression techniques are implemented now . (my index size got reduced from 170 gigs to 115 gigs in solr 4.2). other wise after optimization your index would be accepted by solr 4.x . but index size would remain same .
Please don't use different versions in a same replication system .
If you have been using it in past please share the details it would be really helpful
regards
rajat

BCP exits with code 0

When running BCP from my Java application it exits with status code 0 when 1 is expected.
I run bcp with an invalid combination of data and formatting file and bcp gives the following error:
Starting copy...
SQLState = 22005, NativeError = 0
Error = [Microsoft][SQL Server Native Client 10.0]Invalid character value for cast specification
BCP copy in failed
BCP however exits with exit code 0 and not 1, as i suspect. Now it is extremely difficult to see if something failed while running BCP. Exitting with the right code works once they match to some degree (like same delimiters).
Command
PS C:\Users\feh\Desktop> bcp integrate_test.dbo.AS_LOADER_DELIMITED in .\data.dat -S "10.0.0.161\SQL2K5,1048" -U user -P pass -f .\formatting.ctl -m 1
Starting copy...
SQLState = S1000, NativeError = 0
Error = [Microsoft][SQL Server Native Client 10.0]Unexpected EOF encountered in BCP data-file
0 rows copied.
Network packet size (bytes): 4096
Clock Time (ms.) Total : 1
PS C:\Users\feh\Desktop> $lastexitcode
0
How can i validate a formatting file to the data and get a exit code 1 when they do not match?

If the bcp is "in" the consider using BULK INSERT. This does the same as bcp but can be trapped using SQL Server TRY/CATCH or normal exception handling
Otherwise, are you trapping %ERRORLEVEL% and not some other code?