Cannot resolve ${url} in applicationcontext-ws.xml while running test - java

An url is set with something like http:xxx.com in ..test/java/resources/application-local.properties and ..main/java/resources/application-local.properties
A file called applicationcontext-ws.xml used ${url} :
<jaxws:client id="xxx"
serviceClass="RemoteService"
address="${url}">
<jaxws:outInterceptors>
<ref bean="headerInterceptor"/>
</jaxws:outInterceptors>
</jaxws:client>
When running test, I got an exception:
Caused by: java.net.URISyntaxException: Illegal character in path at
index 1: ${url} at java.net.URI$Parser.fail(URI.java:2848) at
java.net.URI$Parser.checkChars(URI.java:3021) at
java.net.URI$Parser.parseHierarchical(URI.java:3105) at
java.net.URI$Parser.parse(URI.java:3063) at
java.net.URI.(URI.java:588) at
org.apache.cxf.transport.http.HTTPConduit.setupURI(HTTPConduit.java:664)
at
org.apache.cxf.transport.http.HTTPConduit.prepare(HTTPConduit.java:458)
However, when running it normally, i.e., not running test, this problem never occurs.
At first I suspect that it's because ..test/resources/application-local.properties is not loaded. I annotate a field url with #Value(${url}) in my test class, and in debug mode I can see the field's value is exactly the url's content in ..test/resources/application-local.properties, so that's not the cause.
Would you please offer some advice to help me figure out what's the cause and how to resove url while running test? Thanks.

I believe '${url}' itself is going as the url and '{' is being treated as an illegal character. Please debug and check what is the url that is being passed/created at runtime. I believe ${url} is not getting replaced correctly at runtime.

Related

Spring Boot Redisson not able to read clusterServersConfig

Here is the application.yml I am using for my Spring WebFlux project
redis:
redisson:
config: |
clusterServersConfig:
idleConnectionTimeout: 10000
connectTimeout: ${REDISSON_CONNECT_TIMEOUT:20000}
timeout: ${REDISSON_TIMEOUT:3000}
retryAttempts: ${REDISSON_RETRY_ATTEMPTS:3}
retryInterval: ${REDISSON_RETRY_INTERVAL:1500}
subscriptionConnectionPoolSize: ${REDISSON_SUBSCRIPTION_POOL_SIZE:50}
slaveConnectionMinimumIdleSize: ${REDISSON_SLAVE_MIN_IDLE_SIZE:24}
slaveConnectionPoolSize: ${REDISSON_SLAVE_POOL_SIZE:48}
masterConnectionMinimumIdleSize: ${REDISSON_MASTER_MIN_IDLE_SIZE:24}
masterConnectionPoolSize: ${REDISSON_MASTER_POOL_SIZE:48}
nodeAddresses:
- "rediss://${APPS_REDIS:-}:${APPS_REDIS_PORT:6379}"
password: ${APPS_REDIS_SECRET:-}
threads: ${REDISSON_THREADS:16}
nettyThreads: ${REDISSON_NETTY_THREADS:96}
But whenever I am starting the project in my laptop, this error comes up
Caused by: com.fasterxml.jackson.core.JsonParseException: Unrecognized token 'clusterServersConfig': was expecting (JSON String, Number, Array, Object or token 'null', 'true' or 'false')
I am not sure why it is saying clusterServersConfig is an unrecognized token. In the official doc also, it is mentioned and here is an example of this.
At first I thought it might be because I am running redis locally in my M1 Mac so redis-clusters aren't generated by default. I even tried to enable clusters in redis.conf and run a redis clusters with 3 nodes using redis-cli but still this happens. I have tried almost everything I could think of or search on the net. Any help appreciated :)

Gitchangelog plugin error "bjurr.gitchangelog.api.exceptions.GitChangelogIntegrationException"

Error while posting content from jenkin Git changelog plugin to mediawiki
bjurr.gitchangelog.api.exceptions.GitChangelogIntegrationException
Full error:
atse.bjurr.gitchangelog.internal.integrations.mediawiki.MediaWikiClient.createMediaWikiPage(MediaWikiClient.java:95)
at
se.bjurr.gitchangelog.api.GitChangelogApi.toMediaWiki(GitChangelogApi.java:134)
at
org.jenkinsci.plugins.gitchangelog.perform.RemoteCallable.call(RemoteCallable.java:167)
at
org.jenkinsci.plugins.gitchangelog.perform.RemoteCallable.call(RemoteCallable.java:29)
at hudson.FilePath.act(FilePath.java:1085) at
org.jenkinsci.plugins.gitchangelog.perform.GitChangelogPerformer.performerPerform(GitChangelogPerformer.java:40)
at
org.jenkinsci.plugins.gitchangelog.GitChangelogRecorder.perform(GitChangelogRecorder.java:52)
at
hudson.tasks.BuildStepCompatibilityLayer.perform(BuildStepCompatibilityLayer.java:81)
at hudson.tasks.BuildStepMonitor$1.perform(BuildStepMonitor.java:20)
at
hudson.model.AbstractBuild$AbstractBuildExecution.perform(AbstractBuild.java:730)
at
hudson.model.AbstractBuild$AbstractBuildExecution.performAllBuildSteps(AbstractBuild.java:676)
at hudson.model.Build$BuildExecution.post2(Build.java:186) at
hudson.model.AbstractBuild$AbstractBuildExecution.post(AbstractBuild.java:621)
at hudson.model.Run.execute(Run.java:1760) at
hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:43) at
hudson.model.ResourceController.execute(ResourceController.java:97) at
hudson.model.Executor.run(Executor.java:415) Caused by:
com.jayway.jsonpath.PathNotFoundException: Missing property in path
$['login'] at
com.jayway.jsonpath.internal.path.PathToken.handleObjectProperty(PathToken.java:72)
at
com.jayway.jsonpath.internal.path.PropertyPathToken.evaluate(PropertyPathToken.java:77)
at
com.jayway.jsonpath.internal.path.RootPathToken.evaluate(RootPathToken.java:62)
at
com.jayway.jsonpath.internal.path.CompiledPath.evaluate(CompiledPath.java:53)
at
com.jayway.jsonpath.internal.path.CompiledPath.evaluate(CompiledPath.java:61)
at com.jayway.jsonpath.JsonPath.read(JsonPath.java:187) at
com.jayway.jsonpath.internal.JsonContext.read(JsonContext.java:164) at
com.jayway.jsonpath.internal.JsonContext.read(JsonContext.java:151) at
com.jayway.jsonpath.JsonPath.read(JsonPath.java:502) at
se.bjurr.gitchangelog.internal.integrations.mediawiki.MediaWikiClient.getWikiToken(MediaWikiClient.java:192)
at
se.bjurr.gitchangelog.internal.integrations.mediawiki.MediaWikiClient.doAuthenticate(MediaWikiClient.java:177)
at
se.bjurr.gitchangelog.internal.integrations.mediawiki.MediaWikiClient.createMediaWikiPage(MediaWikiClient.java:86)
There is an issue here: https://github.com/tomasbjerre/git-changelog-lib/issues/51
And also asked here: Cannot authenticate with MediaWiki 1.28 API
I'm not sure if this functionality works or not. As far as I know I do follow the API correctly but I have not been able to actually create a page during the last year.

Web service call returns "Fault string, and possibly fault code, not set"

I have a java application (A) that calls the webservice.
1)Then I have another app (B) that uses the A jar.
The call is successful and needed data are returned.
2)Then I have web application that also uses the A jar.
In this case when the web service method is called from A app, an exception is thrown.
Since the class that executes the webservice call is the same in both cases, I'm not sure where the problem is.
The webservice is on https server. But I authenticate using credentials.
I think maybe something should be set in the web application, but I don't know what..
The exception is:
09:21:48.009 [http-nio-8084-exec-33] WARN o.a.cxf.phase.PhaseInterceptorChain - Interceptor for {http://www.myserver.com/ci}ci2#{http://www.myserver.com/ci}GetCertPath has thrown exception, unwinding now
java.lang.UnsupportedOperationException: null
at java.util.AbstractMap.put(AbstractMap.java:209) ~[na:1.8.0_92]
at org.apache.cxf.binding.soap.interceptor.SoapPreProtocolOutInterceptor.setSoapAction(SoapPreProtocolOutInterceptor.java:122) ~[cxf-rt-bindings-soap-2.7.16.jar:2.7.16]
javax.xml.ws.soap.SOAPFaultException: Fault string, and possibly fault code, not set
at org.apache.cxf.binding.soap.interceptor.SoapPreProtocolOutInterceptor.handleMessage(SoapPreProtocolOutInterceptor.java:63) ~[cxf-rt-bindings-soap-2.7.16.jar:2.7.16]
at org.apache.cxf.jaxws.JaxWsClientProxy.invoke(JaxWsClientProxy.java:158)
at org.apache.cxf.binding.soap.interceptor.SoapPreProtocolOutInterceptor.handleMessage(SoapPreProtocolOutInterceptor.java:47) ~[cxf-rt-bindings-soap-2.7.16.jar:2.7.16]
at org.apache.cxf.phase.PhaseInterceptorChain.doIntercept(PhaseInterceptorChain.java:272) ~[cxf-api-2.7.16.jar:2.7.16]
at org.apache.cxf.endpoint.ClientImpl.doInvoke(ClientImpl.java:572) [cxf-api-2.7.16.jar:2.7.16]
at org.apache.cxf.endpoint.ClientImpl.invoke(ClientImpl.java:481) [cxf-api-2.7.16.jar:2.7.16]
at org.apache.cxf.endpoint.ClientImpl.invoke(ClientImpl.java:382) [cxf-api-2.7.16.jar:2.7.16]
at com.sun.proxy.$Proxy318.getCertPath(Unknown Source)
at org.apache.cxf.endpoint.ClientImpl.invoke(ClientImpl.java:335) [cxf-api-2.7.16.jar:2.7.16]
..... more lines here :) .....
at java.util.AbstractMap.put(AbstractMap.java:209)
at org.apache.cxf.binding.soap.interceptor.SoapPreProtocolOutInterceptor.setSoapAction(SoapPreProtocolOutInterceptor.java:122)
at org.apache.cxf.binding.soap.interceptor.SoapPreProtocolOutInterceptor.handleMessage(SoapPreProtocolOutInterceptor.java:63)
Fault string, and possibly fault code, not set
at org.apache.cxf.binding.soap.interceptor.SoapPreProtocolOutInterceptor.handleMessage(SoapPreProtocolOutInterceptor.java:47)
at org.apache.cxf.phase.PhaseInterceptorChain.doIntercept(PhaseInterceptorChain.java:272)
at org.apache.cxf.endpoint.ClientImpl.doInvoke(ClientImpl.java:572)
at org.apache.cxf.endpoint.ClientImpl.invoke(ClientImpl.java:481)
at org.apache.cxf.endpoint.ClientImpl.invoke(ClientImpl.java:382)
at org.apache.cxf.endpoint.ClientImpl.invoke(ClientImpl.java:335)
at org.apache.cxf.frontend.ClientProxy.invokeSync(ClientProxy.java:96)
at org.apache.cxf.jaxws.JaxWsClientProxy.invoke(JaxWsClientProxy.java:136)
... 59 more
Any ideas?
Thank you.
EDIT:
Does it have to have any headers set? When connecting in case 1, there is nothing specific set and it's working.
Ok, so I have found the issue. The request should have included the content-type. Which is strange that it didn't threw an exception in the first case...
I have added Content-Type: application/soap+xml; charset=utf-8 and the web application is working too.
As I found out the javax.xml.ws.soap.SOAPFaultException: Fault string, and possibly fault code, not set is mostly connected to missing content-type..
In my case, this was due to a simple badly formatted URL in a placeholder. http:/xxx.yyu (note the single /)
CXF wasn't very clear running in JDK6 but running in JDK8 was more explicit

File name contains # caused Spring Batch StaxEventItemReader failed to open

I have an input file with # in the name (UPX#L.xxxxx001.xml). Spring Batch StaxEventItemReader failed to open this file. Is there anyway to escape this special character? Please advise.
This is my bean
<bean id="myItemReader" parent="myItemReaderParent" scope="step">
<property name="resource" value="file:#{jobParameters['toolf.input.file.name']}"/>
</bean>
<bean id="myItemReaderParent" class="org.springframework.batch.item.xml.StaxEventItemReader" scope="prototype">
<property name="fragmentRootElementName" value="MyRec"/>
<property name="unmarshaller" ref="myUnmarshaller"/>
</bean>
[INFO ]org.springframework.jms.listener.DefaultMessageListenerContainer#0-1[2015-02-05 10:27:23,487] - Sending notification for start of JobExecution: id=52760, version=1, startTime=Thu Feb 05 10:27:23 EST
2015, endTime=null, lastUpdated=Thu Feb 05 10:27:23 EST 2015, status=STARTED, exitStatus=exitCode=UNKNOWN;exitDescription=, job=[JobInstance: id=52759, version=0, JobParameters=[
{toolf.institutionSymbol=YDX , toolf.projectId=013832, toolf.orderId=678161, toolf.transmissionDate=20150205 10:27:23.392 EST, toolf.input.file.name=/batchdata/dataload/xxx/in/tool/UPX#L.xxxxx001.xml, toolf.out.root.directory=/batchdata/output, toolf.out.input.file.name.without.path=YDX.013832.678161.YbpUsSI#Picklists.d20150204.t140001.xml, toolf.out.working.directory/batchdata/dataload/xxx/in/tool/, timestamp=1423150043392}
], Job=[timeOfOrderLabelJob]]
[INFO ]org.springframework.jms.listener.DefaultMessageListenerContainer#0-1[2015-02-05 10:27:23,576] - Executing step: [toolPrologue]
[ERROR]org.springframework.jms.listener.DefaultMessageListenerContainer#0-1[2015-02-05 10:27:23,607] - Encountered an error executing the step
org.springframework.batch.item.ItemStreamException: Failed to initialize the reader
at org.springframework.batch.item.support.AbstractItemCountingItemStreamItemReader.open(AbstractItemCountingItemStreamItemReader.java:137)
at sun.reflect.GeneratedMethodAccessor260.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:309)
at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:183)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:150)
at org.springframework.aop.support.DelegatingIntroductionInterceptor.doProceed(DelegatingIntroductionInterceptor.java:131)
at org.springframework.aop.support.DelegatingIntroductionInterceptor.invoke(DelegatingIntroductionInterceptor.java:119)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172)
at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:202)
at $Proxy9.open(Unknown Source)
at org.springframework.batch.item.support.CompositeItemStream.open(CompositeItemStream.java:93)
at org.springframework.batch.core.step.tasklet.TaskletStep.open(TaskletStep.java:301)
at org.springframework.batch.core.step.AbstractStep.execute(AbstractStep.java:192)
at org.springframework.batch.core.job.SimpleStepHandler.handleStep(SimpleStepHandler.java:135)
at org.springframework.batch.core.job.flow.JobFlowExecutor.executeStep(JobFlowExecutor.java:61)
at org.springframework.batch.core.job.flow.support.state.StepState.handle(StepState.java:60)
at org.springframework.batch.core.job.flow.support.SimpleFlow.resume(SimpleFlow.java:144)
at org.springframework.batch.core.job.flow.support.SimpleFlow.start(SimpleFlow.java:124)
at org.springframework.batch.core.job.flow.FlowJob.doExecute(FlowJob.java:135)
at org.springframework.batch.core.job.AbstractJob.execute(AbstractJob.java:281)
at org.springframework.batch.core.launch.support.SimpleJobLauncher$1.run(SimpleJobLauncher.java:120)
at org.springframework.core.task.SyncTaskExecutor.execute(SyncTaskExecutor.java:48)
at org.springframework.batch.core.launch.support.SimpleJobLauncher.run(SimpleJobLauncher.java:114)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.codehaus.groovy.runtime.callsite.PojoMetaMethodSite$PojoCachedMethodSiteNoUnwrapNoCoerce.invoke(PojoMetaMethodSite.java:229)
at org.codehaus.groovy.runtime.callsite.PojoMetaMethodSite.call(PojoMetaMethodSite.java:52)
at org.codehaus.groovy.runtime.callsite.CallSiteArray.defaultCall(CallSiteArray.java:40)
at org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:117)
at org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:129)
at org.oclc.tool.mule.launch.MuleJobLauncher.launchJob(MuleJobLauncher.groovy:74)
at org.oclc.tool.mule.launch.JobLauncher$launchJob.call(Unknown Source)
at org.codehaus.groovy.runtime.callsite.CallSiteArray.defaultCall(CallSiteArray.java:40)
at org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:117)
at org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:137)
at org.oclc.tool.mule.si.JmsMessageUnwrapper.unwrapMessage(JmsMessageUnwrapper.groovy:42)
at sun.reflect.GeneratedMethodAccessor596.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.springframework.expression.spel.support.ReflectiveMethodExecutor.execute(ReflectiveMethodExecutor.java:69)
at org.springframework.expression.spel.ast.MethodReference.getValueInternal(MethodReference.java:83)
at org.springframework.expression.spel.ast.CompoundExpression.getValueInternal(CompoundExpression.java:57)
at org.springframework.expression.spel.ast.SpelNodeImpl.getTypedValue(SpelNodeImpl.java:102)
at org.springframework.expression.spel.standard.SpelExpression.getValue(SpelExpression.java:102)
at org.springframework.integration.util.AbstractExpressionEvaluator.evaluateExpression(AbstractExpressionEvaluator.java:126)
at org.springframework.integration.util.MessagingMethodInvokerHelper.processInternal(MessagingMethodInvokerHelper.java:225)
at org.springframework.integration.util.MessagingMethodInvokerHelper.process(MessagingMethodInvokerHelper.java:125)
at org.springframework.integration.handler.MethodInvokingMessageProcessor.processMessage(MethodInvokingMessageProcessor.java:73)
at org.springframework.integration.handler.ServiceActivatingHandler.handleRequestMessage(ServiceActivatingHandler.java:64)
at org.springframework.integration.handler.AbstractReplyProducingMessageHandler.handleMessageInternal(AbstractReplyProducingMessageHandler.java:98)
at org.springframework.integration.handler.AbstractMessageHandler.handleMessage(AbstractMessageHandler.java:78)
at org.springframework.integration.dispatcher.UnicastingDispatcher.doDispatch(UnicastingDispatcher.java:110)
at org.springframework.integration.dispatcher.UnicastingDispatcher.dispatch(UnicastingDispatcher.java:97)
at org.springframework.integration.channel.AbstractSubscribableChannel.doSend(AbstractSubscribableChannel.java:61)
at org.springframework.integration.channel.AbstractMessageChannel.send(AbstractMessageChannel.java:157)
at org.springframework.integration.core.MessagingTemplate.doSend(MessagingTemplate.java:288)
at org.springframework.integration.core.MessagingTemplate.send(MessagingTemplate.java:149)
at org.springframework.integration.core.MessagingTemplate.convertAndSend(MessagingTemplate.java:189)
at org.springframework.integration.gateway.MessagingGatewaySupport.send(MessagingGatewaySupport.java:183)
at org.springframework.integration.jms.ChannelPublishingJmsMessageListener$GatewayDelegate.send(ChannelPublishingJmsMessageListener.java:424)
at org.springframework.integration.jms.ChannelPublishingJmsMessageListener.onMessage(ChannelPublishingJmsMessageListener.java:278)
at org.springframework.jms.listener.AbstractMessageListenerContainer.doInvokeListener(AbstractMessageListenerContainer.java:535)
at org.springframework.jms.listener.AbstractMessageListenerContainer.invokeListener(AbstractMessageListenerContainer.java:495)
at org.springframework.jms.listener.AbstractMessageListenerContainer.doExecuteListener(AbstractMessageListenerContainer.java:467)
at org.springframework.jms.listener.AbstractPollingMessageListenerContainer.doReceiveAndExecute(AbstractPollingMessageListenerContainer.java:325)
at org.springframework.jms.listener.AbstractPollingMessageListenerContainer.receiveAndExecute(AbstractPollingMessageListenerContainer.java:263)
at org.springframework.jms.listener.DefaultMessageListenerContainer$AsyncMessageListenerInvoker.invokeListener(DefaultMessageListenerContainer.java:1058)
at org.springframework.jms.listener.DefaultMessageListenerContainer$AsyncMessageListenerInvoker.executeOngoingLoop(DefaultMessageListenerContainer.java:1050)
at org.springframework.jms.listener.DefaultMessageListenerContainer$AsyncMessageListenerInvoker.run(DefaultMessageListenerContainer.java:947)
at java.lang.Thread.run(Thread.java:619)
Caused by: java.lang.IllegalStateException: Input resource must exist (reader is in 'strict' mode)
at org.springframework.batch.item.xml.StaxEventItemReader.doOpen(StaxEventItemReader.java:182)
at org.springframework.batch.item.support.AbstractItemCountingItemStreamItemReader.open(AbstractItemCountingItemStreamItemReader.java:134)
... 73 more
[INFO ]org.springframework.jms.listener.DefaultMessageListenerContainer#0-1[2015-02-05 10:27:23,612] - enter DetailLogStepExecutionListener.afterStep
[INFO ]org.springframework.jms.listener.DefaultMessageListenerContainer#0-1[2015-02-05 10:27:23,612] - exit DetailLogStepExecutionListener.afterStep
In a URL/URI, the # character is reserved, used to delimit the fragment identifier (which would have no meaning to the file: resource scheme). If you want to include this character as part of the path, it must be escaped with %23
Rather than concatenate your value to file: in your bean definition file, I recommend that you specify the parameter as a full URI/URI, rather than a filesystem path:
<property name="resource" value="#{jobParameters['toolf.input.file.url']}"/>
And set toolf.input.file.url to:
file:/batchdata/dataload/xxx/in/tool/UPX%23L.xxxxx001.xml
This will avoid any other escapement pitfalls you might encounter, like with characters #, !, =, etc.
If you must use a filesystem path as a parameter, you should be able to use:
<property name="resource" value="#{new java.io.File(jobParameters['toolf.input.file.name']).toURI().toURL()}"/>
From looking at your error message and the responses, your problem is clear, both Tunaki and Glenn Lane are partially correct, and I'm up-voting Glenn Lane for answering the question you asked, and doing it well.
From your reply to Glenn:
if (fileName.contains("#")) { def (value1, value2) = fileName.split("#") fileName = value1 + "%23#" + value2 } I tried this. Still doesn't work
Escaping will not help the situation.
The character is already being handled somewhere along the execution chain, Tunaki was correct in questioning the type error you were having.
From your error log:
[ERROR]org.springframework.jms.listener.DefaultMessageListenerContainer#0-1[2015-02-05 10:27:23,607] - Encountered an error executing the step
org.springframework.batch.item.ItemStreamException: Failed to initialize the reader
Coupled with the resource state exception Tunaki pointed out, we can conclude that is the case. There is insufficient information here to say why or where this occurring, but here are a two general categories:
Depending on how it's arriving at, or from, the file store the character may already removed from the file at that point in the process or aliased on the Store in a way the File handler can't deal with. For the reasons Glenn pointed out.
Depending on the Message or Job Handling in your execution chain it is being stripped out as it transformed into or out of a URI/URL call. For the reasons Glenn pointed out.
Yes. It is the # problem. Once the # is removed it works fine.
Based on the above statement the simple, quick, and dirty solution would be to be aware of this and handle it by stripping and injecting the # directly from the file or uri as needed. Based on your code snippet, you know how to do this.
The correct way to solve this would be to track down where the resource error is occurring and ensure that your file handling is consistent across the entire Job flow.
Without all the right code here and/or knowing more about your system configuration I can't be specific. There is simply not enough information between the bean and the error log to say exactly where this occurring only that this is exactly your problem.
Here are some possible suggestions:
I am assuming you've checked the file Store directly and hand removed the # and changed the file name in the job and it runs, based on your above statement.
Try to access the file on the store from a different java object using the same identifier and it works; Then check how your item reader implementation is handling the file access.
If it failed or that is not the failure point. Check that "#" value is the job file and if it's there and/or the only # when you perform the split.
If that all checks find out how it is arriving to the file store and that resource can be properly located in the store and that # is not also reserved in your implementation or requires special handling/configuration.
This will help narrow done the root cause.

error when opening a lucene index: java.io.IOException: Map failed

I have a problem which I think is the same as that described here:
Error when opening a lucene index: Map failed
However the solution does not apply in this case so I am providing more details and asking again.
The index is created using Solr 5.3
The line of code causing the exception is:
IndexReader indexReader = DirectoryReader.open(FSDirectory.open(Paths.get("the_path")));
The exception stacktrace is:
Exception in thread "main" java.io.IOException: Map failed: MMapIndexInput(path="/mnt/fastdata/ac1zz/JATE/solr-5.3.0/server/solr/jate/data_aclrd/index/_5t.tvd") [this may be caused by lack of enough unfragmented virtual address space or too restrictive virtual memory limits enforced by the operating system, preventing us to map a chunk of 434505698 bytes. Please review 'ulimit -v', 'ulimit -m' (both should return 'unlimited'), and 'sysctl vm.max_map_count'. More information: http://blog.thetaphi.de/2012/07/use-lucenes-mmapdirectory-on-64bit.html]
at sun.nio.ch.FileChannelImpl.map(FileChannelImpl.java:907)
at org.apache.lucene.store.MMapDirectory.map(MMapDirectory.java:265)
at org.apache.lucene.store.MMapDirectory.openInput(MMapDirectory.java:239)
at org.apache.lucene.codecs.compressing.CompressingTermVectorsReader.<init>(CompressingTermVectorsReader.java:144)
at org.apache.lucene.codecs.compressing.CompressingTermVectorsFormat.vectorsReader(CompressingTermVectorsFormat.java:91)
at org.apache.lucene.index.SegmentCoreReaders.<init>(SegmentCoreReaders.java:120)
at org.apache.lucene.index.SegmentReader.<init>(SegmentReader.java:65)
at org.apache.lucene.index.StandardDirectoryReader$1.doBody(StandardDirectoryReader.java:58)
at org.apache.lucene.index.StandardDirectoryReader$1.doBody(StandardDirectoryReader.java:50)
at org.apache.lucene.index.SegmentInfos$FindSegmentsFile.run(SegmentInfos.java:731)
at org.apache.lucene.index.StandardDirectoryReader.open(StandardDirectoryReader.java:50)
at org.apache.lucene.index.DirectoryReader.open(DirectoryReader.java:63)
at uk.ac.shef.dcs.jate.app.AppATTF.extract(AppATTF.java:39)
at uk.ac.shef.dcs.jate.app.AppATTF.main(AppATTF.java:33)
The suggested solutions as in the exception message do not work in this case because I am running the application on a server and I do not have permissions to change those.
Namely,
ulimit -v unlimited
prints: "-bash: ulimit: virtual memory: cannot modify limit: Operation not permitted"
and
sysctl -w vm.max_map_count=10000000
gives:"error: permission denied on key 'vm.max_map_count'"
Is there anyway I can solve this?
Thanks
I have found a solution and so I am answering myself.
If you really cannot set ulimit or vm.max_map_count, the only solution, according to http://blog.thetaphi.de/2012/07/use-lucenes-mmapdirectory-on-64bit.html, is to configure your solr (or if you work with Lucene api, choose explicitly) to use SimpleFSDirectory (if windows) or NIOFSDirectory, both are slower than the default.
For example
DirectoryReader.open(new NIOFSDirectory(Paths.get("path_to_index"), FSLockFactory.getDefault()))

Categories