full code
public class SolrToMongodb {
private static final Logger LOGGER = LoggerFactory.getLogger(SolrToMongodb.class);
public static void main(String[] args) throws IOException, SolrServerException {
SolrToMongodb main = new SolrToMongodb();
main.run();
}
public void run() throws IOException, SolrServerException {
SparkConfig config = new SparkConfig();
JavaSparkContext jsc = new JavaSparkContext(config.sparkConf("admiralty-stream"));
SolrClient client = new HttpSolrClient(Constant.SOLR_STREAMING);
SolrQuery q = new SolrQuery();
q.set("q","*:*");
q.set("indent","on");
q.set("wt", "json");
client.query(q);
try {
CloudSolrClient cloudSolrClient = new CloudSolrClient(Constant.ZOOKEEPER_SOLR);
SolrJavaRDD solrRDD = SolrJavaRDD.get(cloudSolrClient.getZkHost(), "admiraltyStream", jsc.sc());
JavaRDD<SolrDocument> resultsRDD = solrRDD.queryShards(q);
JavaRDD<Object> objectJavaRDD = resultsRDD.map(new Function<SolrDocument, Object>() {
#Override
public Object call(SolrDocument v1) throws Exception {
System.out.println(v1.getFieldValueMap());
return v1.getFieldValueMap();
}
});
}
catch (Exception e){
System.out.println("Exception here : "+e.getMessage());
}
}}
ERROR LOG :
2017-08-02 10:02:58,709 [main] ERROR CloudSolrClient - Request to collection admiraltyStream failed due to (0) java.net.ConnectException: Connection refused (Connection refused), retry? 0
2017-08-02 10:02:59,688 [main] ERROR CloudSolrClient - Request to collection admiraltyStream failed due to (0) java.net.ConnectException: Connection refused (Connection refused), retry? 1
2017-08-02 10:03:01,630 [main] ERROR CloudSolrClient - Request to collection admiraltyStream failed due to (0) java.net.ConnectException: Connection refused (Connection refused), retry? 2
2017-08-02 10:03:02,579 [main] ERROR CloudSolrClient - Request to collection admiraltyStream failed due to (0) java.net.ConnectException: Connection refused (Connection refused), retry? 3
2017-08-02 10:03:03,540 [main] ERROR CloudSolrClient - Request to collection admiraltyStream failed due to (0) java.net.ConnectException: Connection refused (Connection refused), retry? 4
2017-08-02 10:03:04,484 [main] ERROR CloudSolrClient - Request to collection admiraltyStream failed due to (0) java.net.ConnectException: Connection refused (Connection refused), retry? 5
Exception :
Exception here : No live SolrServers available to handle this request:[http://xxx.xxx.ph:8983/solr/admiraltyStream, http://xxx.xxx.ph:8983/solr/admiraltyStream, http://xxx.xxx.ph:8983/solr/admiraltyStream]
Using the CloudSolrClient instead of HttpSolrClient allows solrj to do a round-robin load balancing between the available solr servers, and of course is recommended in a SolrCloud context. The "No live SolrServers” message indicates a problem with the collection admiraltyStream.
Specifically, behind the scenes, SolrJ is using LBHttpSolrClient (which is using a set of HttpSolrClient instances) for round-robin requests between shards. I think that your problem is actually this: some shard is not available (i.e. the leader and the replicas).
I would review the currently online replicas (http://solr.server:8983/#/~cloud): there you should see if all the replicas for your collection online.
There must be at least one replica per shard; and I think that in your case:
the node you’re trying to connect directly using HttpSolrClient is up and running
when using CloudSolrClient (i.e. Zookeeper -> Solr), there’s something wrong in your cluster state: Solr believes that there are no replicas for at least one shard, so the list of the available HttpSolrClient instances within a given instance of LBHttpSolrClient is empty
Related
Hope you are doing well. Basically I am calling Java API from .NET .
Most of the time it is working absolutely fine but the thing it sometimes once or twice in a day it gives me Socket exception. I am putting down the calling code and the exception stack trace. Any suggestion to fix this once and for all will be helpful.
Thanks in advance.!!
Code :
var client = new HttpClient{Timeout = TimeSpan.FromMinutes(30), BaseAddress = new Uri(AppSettingsConfiguration.GetConfig<string>("appSettings:Generic"))};
client.DefaultRequestHeaders.Accept.Clear();
client.DefaultRequestHeaders.Accept.Add(new MediaTypeWithQualityHeaderValue("application/json"));
client.DefaultRequestHeaders.Add("key", AppSettingsConfiguration.GetConfig<string>("appSettings:JavaAPIKey"));
var response = client.PostAsync(AppSettingsConfiguration.GetConfig<string>("appSettings:Generic"), new StringContent(JsonConvert.SerializeObject(order), Encoding.UTF8, "application/json")).Result;
Stack Trace :
Error reported : One or more errors occurred. Inner Exception
System.Net.Http.HttpRequestException: An error occurred while sending the request. --->
System.Net.WebException: Unable to connect to the remote server --->
System.Net.Sockets.SocketException: A connection attempt failed because the connected party did
not properly respond after a period of time, or established connection failed because connected host has failed to respond {IP} at
System.Net.Sockets.Socket.InternalEndConnect(IAsyncResult asyncResult) at System.Net.Sockets.Socket.EndConnect(IAsyncResult asyncResult)
at System.Net.ServicePoint.ConnectSocketInternal(Boolean connectFailure, Socket s4, Socket s6,
Socket& socket, IPAddress& address, ConnectSocketState state, IAsyncResult asyncResult, Exception& exception) --- End of inner exception stack trace --- at
System.Net.HttpWebRequest.EndGetRequestStream(IAsyncResult asyncResult, TransportContext& context)
at System.Net.Http.HttpClientHandler.GetRequestStreamCallback(IAsyncResult ar) --- End of inner
exception stack trace --- Stack trace :
After 30 mins of inactivity or so I start getting the below error when I try to insert into mongo, When I try again it starts to work. Error Below. I'm on Azure:
[INFO ] 2018-09-10T12:00:43,188 [http-nio-8080-exec-6] connection - Closed connection [connectionId{localValue:3, serverValue:26}] to XX.XX.XX.XX:27017 because there was a socket exception raised by this connection.
[ERROR] 2018-09-10T12:00:43,189 [http-nio-8080-exec-6] [dispatcherServlet] - Servlet.service() for servlet [dispatcherServlet] in context with path [] threw exception [Request processing failed; nested exception is org.springframework.data.mongodb.UncategorizedMongoDbException: Timeout while receiving message; nested exception is com.mongodb.MongoSocketReadTimeoutException: Timeout while receiving message] with root cause
java.net.SocketTimeoutException: Read timed out
at java.net.SocketInputStream.socketRead0(Native Method) ~[?:1.8.0_181]
at java.net.SocketInputStream.socketRead(SocketInputStream.java:116) ~[?:1.8.0_181]
ava.net.SocketTimeoutException: Read timed out
t java.net.SocketInputStream.read(SocketInputStream.java:171) ~[?:1.8.0_181]
at java.net.SocketInputStream.read(SocketInputStream.java:141) ~[?:1.8.0_181]
Here is how I initialize my mongo Template:
#Bean
public MongoDbFactory mongoDbFactory() {
String[] addresses = mongoUri.split(",");
List<ServerAddress> servers = new ArrayList<>();
for (String address : addresses) {
String[] split = address.trim().split(":");
servers.add(new ServerAddress(split[0].trim(), Integer.parseInt(split[1].trim())));
}
MongoClientOptions.Builder mongoOperations = MongoClientOptions.builder();
mongoOperations.socketTimeout(1000 * 20); // I tried to increase the socket timeout to see if it helps but no help either
mongoOperations.connectTimeout(1000 * 10);
MongoClient mongoClient = new MongoClient(servers, MongoCredential.createCredential(userName, dbName, password.toCharArray()), mongoOperations.build());
return new SimpleMongoDbFactory(mongoClient, dbName);
}
#Bean
public MongoTemplate getMongoTemplate() {
return new MongoTemplate(mongoDbFactory());
}
My mongod version is 3.6.4 and I'm using the same version of java driver.
I tried to increase/decrease the tcp_keepalive_time setting as provided in the docs using
sudo sysctl -w net.ipv4.tcp_keepalive_time=120 but no help either.
ohk. so we found that the mongo java driver jar was older than the mongo server we were using which was causing this issue. make sure that the driver supports the version of mongo server.
I am loading data in Solr from MongoDB.
SolrInputDocument doc = new SolrInputDocument();
BasicDBObject record = (BasicDBObject) cursor.next();
doc.addField("_id", record.get("_id"));
// add more fields
server.add(doc);
server.commit();
I am getting below exception.
Exception in thread "main" org.apache.solr.client.solrj.SolrServerException: java.net.ConnectException: Connection timed out: connect
at org.apache.solr.client.solrj.impl.CommonsHttpSolrServer.request(CommonsHttpSolrServer.java:483)
at org.apache.solr.client.solrj.impl.CommonsHttpSolrServer.request(CommonsHttpSolrServer.java:244)
at org.apache.solr.client.solrj.request.AbstractUpdateRequest.process(AbstractUpdateRequest.java:105)
at org.apache.solr.client.solrj.SolrServer.add(SolrServer.java:64)
Caused by: java.net.ConnectException: Connection timed out: connect
This code worked for me before. I did not change anything in the code. But now its giving error after adding few documents. What's wrong here?
I have the following problem:
I try to connect to an ActiveMQ broker (which is now down) using the following piece of code
connectionFactory = new ActiveMQConnectionFactory(this.url + "?timeout=2000");
connection = connectionFactory.createConnection();
connection.start();
session = connection.createSession(false, Session.AUTO_ACKNOWLEDGE);
LOGGER.info("Connected to " + this.url);
The problem is that the timeout does not have any effect
connection.start()
is blocked forever.
I inspected ActiveMQ log and found the following info:
2013-12-20 01:49:03,149 DEBUG [ActiveMQ Task-1] (FailoverTransport.java:786) - urlList connectionList:[tcp://localhost:61616?timeout=2000], from: [tcp://localhost:61616?timeout=2000]
2013-12-20 01:49:03,149 DEBUG [ActiveMQ Task-1] (FailoverTransport.java:1040) - Connect fail to: tcp://localhost:61616?timeout=2000, reason: java.lang.IllegalArgumentException: Invalid connect parameters: {timeout=2000}
The timeout parameter is specified here http://activemq.apache.org/cms/configuring.html
Has anybody any idea how to pass timeout argument to ActiveMQConnectionFactory?
Or how to set a timeout for connection.start() ?
Thank you!
Update: I found this on Stackoverflow: ActiveMQ - CreateSession failover timeout after a connection is resumed . I tried it but the following exception is thrown:
javax.jms.JMSException: Could not create Transport. Reason: java.lang.IllegalArgumentException: Invalid connect parameters: {transport.timeout=5000}
at org.apache.activemq.util.JMSExceptionSupport.create(JMSExceptionSupport.java:35)
I use ActiveMQ 5.8.0 from maven repo
It appears that your url is invalid still in both cases when attempting to set the timeout property.
If you're trying to have a failover URL, which it looks like you are since it is getting in to the Failover code then you're probably looking for initialReconnectDelay (and possibly maxReconnectAttempts which would throw an exception if the server is still down after the number of attempts is reached).
ActiveMQConnectionFactory connectionFactory = new ActiveMQConnectionFactory("failover://(tcp://localhost:61616)?initialReconnectDelay=2000&maxReconnectAttempts=2");
I am trying to catch exception for an ActiveMQ connection which could not be established because of the broker being down.
With following code:
String url = ActiveMQConnection.DEFAULT_BROKER_URL;
ConnectionFactory connectionFactory = new ActiveMQConnectionFactory(url);
Connection connection = connectionFactory.createConnection();
connection.start();
The attempt to connect to the broker goes into infinite loop, if the broker is down. If I change url to
String url = "failover:(tcp://127.0.0.1:61616/)?startupMaxReconnectAttempts=2";
It makes 2 attempts and then throws an exception.(which is what I want.)
Now if I initialize the connection object using Spring Bean with the following:
<bean id="jmsFactory" class="org.apache.activemq.ActiveMQConnectionFactory">
<property name="brokerURL">
<!--<value>tcp://0.0.0.0:61616</value>-->
<value>failover:(tcp://127.0.0.1:61616/)?startupMaxReconnectAttempts=2</value>
</property>
</bean>
I get an error message for failure to connect in 2 attempts but then, it still tries to connect again after every 5 seconds giving the same error message again and going on in infinite loop.
ERROR transport.failover.FailoverTransport - Failed to connect to [tcp://127.0.0.1:61616/] after: 2 attempt(s)
WARN jms.listener.DefaultMessageListenerContainer - Could not refresh JMS Connection for destination 'destinationQueue' - retrying in 5000 ms. Cause: Connection refused
ERROR transport.failover.FailoverTransport - Failed to connect to [tcp://127.0.0.1:61616/] after: 2 attempt(s)
WARN jms.listener.DefaultMessageListenerContainer - Could not refresh JMS Connection for destination 'destinationQueue' - retrying in 5000 ms. Cause: Connection refused
ERROR transport.failover.FailoverTransport - Failed to connect to [tcp://127.0.0.1:61616/] after: 2 attempt(s)
WARN jms.listener.DefaultMessageListenerContainer - Could not refresh JMS Connection for destination 'destinationQueue' - retrying in 5000 ms. Cause: Connection refused
ERROR transport.failover.FailoverTransport - Failed to connect to [tcp://127.0.0.1:61616/] after: 2 attempt(s)
WARN jms.listener.DefaultMessageListenerContainer - Could not refresh JMS Connection for destination 'destinationQueue' - retrying in 5000 ms. Cause: Connection refused
these messages repeat!!
I want to know how to stop this infinite polling and catch an exception (may be using PostInit) in case of failure.
You could implement ExceptionListener in your listener class and override onException message. There you could handle your notification logic. While it try to reconnect automatically.
public class QueueListener implements MessageListener,ExceptionListener{
public void onMessage(Message message) {
}
public void onException(JMSException jsme) {
// Send my notifcation here.
}
}