I can't seem to get GATK to recognise the number of available threads. I am running GATK (4.2.4.1) in a conda environment which is part of a nextflow (v20.10.0) pipeline I'm writing. For whatever reason, I cannot get GATK to see there is more than one thread. I've tried different node types, increasing and decreasing the number of cpus available, providing java arguments such as -XX:ActiveProcessorCount=16, using taskset, but it always just detects 1.
Here is the command from the .command.sh:
gatk HaplotypeCaller \
--tmp-dir tmp/ \
-ERC GVCF \
-R VectorBase-54_AgambiaePEST_Genome.fasta \
-I AE12A_S24_BP.bam \
-O AE12A_S24_BP.vcf
And here is the top of the .command.log file:
12:10:00.695 INFO HaplotypeCaller - ------------------------------------------------------------
12:10:00.695 INFO HaplotypeCaller - The Genome Analysis Toolkit (GATK) v4.2.4.1
12:10:00.695 INFO HaplotypeCaller - For support and documentation go to https://software.broadinstitute.org/gatk/
12:10:00.696 INFO HaplotypeCaller - Executing on Linux v4.18.0-193.6.3.el8_2.x86_64 amd64
12:10:00.696 INFO HaplotypeCaller - Java runtime: OpenJDK 64-Bit Server VM v11.0.13+7-b1751.21
12:10:00.696 INFO HaplotypeCaller - Start Date/Time: 9 February 2022 at 12:10:00 GMT
12:10:00.696 INFO HaplotypeCaller - ------------------------------------------------------------
12:10:00.696 INFO HaplotypeCaller - ------------------------------------------------------------
12:10:00.697 INFO HaplotypeCaller - HTSJDK Version: 2.24.1
12:10:00.697 INFO HaplotypeCaller - Picard Version: 2.25.4
12:10:00.697 INFO HaplotypeCaller - Built for Spark Version: 2.4.5
12:10:00.697 INFO HaplotypeCaller - HTSJDK Defaults.COMPRESSION_LEVEL : 2
12:10:00.697 INFO HaplotypeCaller - HTSJDK Defaults.USE_ASYNC_IO_READ_FOR_SAMTOOLS : false
12:10:00.697 INFO HaplotypeCaller - HTSJDK Defaults.USE_ASYNC_IO_WRITE_FOR_SAMTOOLS : true
12:10:00.697 INFO HaplotypeCaller - HTSJDK Defaults.USE_ASYNC_IO_WRITE_FOR_TRIBBLE : false
12:10:00.697 INFO HaplotypeCaller - Deflater: IntelDeflater
12:10:00.697 INFO HaplotypeCaller - Inflater: IntelInflater
12:10:00.697 INFO HaplotypeCaller - GCS max retries/reopens: 20
12:10:00.698 INFO HaplotypeCaller - Requester pays: disabled
12:10:00.698 INFO HaplotypeCaller - Initializing engine
12:10:01.126 INFO HaplotypeCaller - Done initializing engine
12:10:01.129 INFO HaplotypeCallerEngine - Tool is in reference confidence mode and the annotation, the following changes will be made to any specified annotations: 'StrandBiasBySample' will be enabled. 'ChromosomeCounts', 'FisherStrand', 'StrandOddsRatio' and 'QualByDepth' annotations have been disabled
12:10:01.143 INFO HaplotypeCallerEngine - Standard Emitting and Calling confidence set to 0.0 for reference-model confidence output
12:10:01.143 INFO HaplotypeCallerEngine - All sites annotated with PLs forced to true for reference-model confidence output
12:10:01.162 INFO NativeLibraryLoader - Loading libgkl_utils.so from jar:file:/home/anaconda3/envs/NF_GATK/share/gatk4-4.2.4.1-0/gatk-package-4.2.4.1-local.jar!/com/intel/gkl/native/libgkl_utils.so
12:10:01.169 INFO NativeLibraryLoader - Loading libgkl_pairhmm_omp.so from jar:file:/home/anaconda3/envs/NF_GATK/share/gatk4-4.2.4.1-0/gatk-package-4.2.4.1-local.jar!/com/intel/gkl/native/libgkl_pairhmm_omp.so
12:10:01.209 INFO IntelPairHmm - Flush-to-zero (FTZ) is enabled when running PairHMM
12:10:01.210 INFO IntelPairHmm - Available threads: 1
12:10:01.210 INFO IntelPairHmm - Requested threads: 4
12:10:01.210 WARN IntelPairHmm - Using 1 available threads, but 4 were requested
12:10:01.210 INFO PairHMM - Using the OpenMP multi-threaded AVX-accelerated native PairHMM implementation
12:10:01.271 INFO ProgressMeter - Starting traversal
I found a thread on the broad institute website suggesting it might be the OMP library, but this is seemingly loaded, and I'm using the version they suggested updating to...
Needless to say, this is a little slow. I can always parallelise by using the -L option, but this doesn't solve that every step in the pipeline will be very slow.
Thanks in advance.
In case anyone else has the same problem, it turned out I had to configure the submission as an MPI job.
So on the HPC I use, here is the nextflow process:
process DNA_HCG {
errorStrategy { sleep(Math.pow(2, task.attempt) * 600 as long); return 'retry' }
maxRetries 3
maxForks params.HCG_Forks
tag { SampleID+"-"+chrom }
executor = 'pbspro'
clusterOptions = "-lselect=1:ncpus=${params.HCG_threads}:mem=${params.HCG_memory}gb:mpiprocs=1:ompthreads=${params.HCG_threads} -lwalltime=${params.HCG_walltime}:00:00"
publishDir(
path: "${params.HCDir}",
mode: 'copy',
)
input:
each chrom from chromosomes_ch
set SampleID, path(bam), path(bai) from processed_bams
path ref_genome
path ref_dict
path ref_index
output:
tuple chrom, path("${SampleID}_${chrom}.vcf") into HCG_ch
path("${SampleID}_${chrom}.vcf.idx") into idx_ch
beforeScript 'module load anaconda3/personal; source activate NF_GATK'
script:
"""
mkdir tmp
n_slots=`expr ${params.GVCF_threads} / 2 - 3`
if [ \$n_slots -le 0 ]; then n_slots=1; fi
taskset -c 0-\${n_slots} gatk --java-options \"-Xmx${params.HCG_memory}G -XX:+UseParallelGC -XX:ParallelGCThreads=\${n_slots}\" HaplotypeCaller \\
--tmp-dir tmp/ \\
--pair-hmm-implementation AVX_LOGLESS_CACHING_OMP \\
--native-pair-hmm-threads \${n_slots} \\
-ERC GVCF \\
-L ${chrom} \\
-R ${ref_genome} \\
-I ${bam} \\
-O ${SampleID}_${chrom}.vcf ${params.GVCF_args}
"""
}
I think I solved this problem (at least for me, it worked well on SLURM). This comes from how GATK is configured for parallelizing jobs: it's based on OpenMP, so you should add to the beginning of your script something like this:
export OMP_NUM_THREADS=$SLURM_CPUS_PER_TASK
source
Related
I'm having some trouble installing apache netbeans 11.2, (and in general any version of netbeans) on my pc because when I run the installer the error, described in the title happens.
There are no lock files.
I have jdk 13.0.1 installed, (and java 8u231)
I'm running it as administrator
(Update) Since there is a 30000 character limit I've replaced the log with the NetBeans\11.2\var\log\messages.txt and removed the Turning on modules: ...
Any help is greatly appreciated.
-------------------------------------------------------------------------------
>Log Session: Monday, December 2, 2019 at 12:11:03 PM Central European Standard Time
>System Info:
Product Version = Apache NetBeans IDE 11.2
Operating System = Windows 10 version 10.0 running on amd64
Java; VM; Vendor = 13.0.1; Java HotSpot(TM) 64-Bit Server VM 13.0.1+9; Oracle Corporation
Runtime = Java(TM) SE Runtime Environment 13.0.1+9
Java Home = C:\Program Files\Java\jdk-13.0.1
System Locale; Encoding = hu_HU (nb); Cp1250
Home Directory = C:\Users\banhi
Current Directory = C:\netbeans\bin
User Directory = C:\Users\banhi\AppData\Roaming\NetBeans\11.2
Cache Directory = C:\Users\banhi\AppData\Local\NetBeans\Cache\11.2
Installation = C:\netbeans\nb
C:\netbeans\ergonomics
C:\netbeans\ide
C:\netbeans\extide
C:\netbeans\java
C:\netbeans\apisupport
C:\netbeans\webcommon
C:\netbeans\websvccommon
C:\netbeans\enterprise
C:\netbeans\profiler
C:\netbeans\php
C:\netbeans\harness
C:\netbeans\groovy
C:\netbeans\javafx
C:\netbeans\platform
Boot & Ext. Classpath =
Application Classpath = C:\netbeans\platform\lib\boot.jar;C:\netbeans\platform\lib\org-openide-modules.jar;C:\netbeans\platform\lib\org-openide-util-lookup.jar;C:\netbeans\platform\lib\org-openide-util-ui.jar;C:\netbeans\platform\lib\org-openide-util.jar
Startup Classpath = C:\netbeans\platform\core\asm-all-5.0.1.jar;C:\netbeans\platform\core\core-base.jar;C:\netbeans\platform\core\core.jar;C:\netbeans\platform\core\org-netbeans-libs-asm.jar;C:\netbeans\platform\core\org-openide-filesystems-compat8.jar;C:\netbeans\platform\core\org-openide-filesystems.jar;C:\netbeans\nb\core\org-netbeans-upgrader.jar;C:\netbeans\nb\core\locale\core_nb.jar
-------------------------------------------------------------------------------
WARNING [org.netbeans.core.startup.NbEvents]: The extension C:\netbeans\ide\modules\ext\jcodings-1.0.18.jar may be multiply loaded by modules: [C:\netbeans\ide\modules\org-netbeans-libs-bytelist.jar, C:\netbeans\ide\modules\org-netbeans-modules-textmate-lexer.jar]; see: http://www.netbeans.org/download/dev/javadoc/org-openide-modules/org/openide/modules/doc-files/classpath.html#class-path
INFO [org.netbeans.modules.netbinox]: Install area set to file:/C:/netbeans/
WARNING [org.netbeans.core.modules]: the modules [org.netbeans.modules.java.editor.lib, org.netbeans.modules.xml.text] use org.netbeans.modules.editor.deprecated.pre65formatting which is deprecated.
WARNING [org.netbeans.core.modules]: the modules [org.netbeans.modules.ide.kit, org.netbeans.modules.xml.text] use org.netbeans.modules.editor.structure which is deprecated.
WARNING [org.netbeans.core.modules]: the modules [org.netbeans.modules.ant.hints, org.netbeans.modules.java.hints, org.netbeans.modules.jshell.support, org.netbeans.modules.maven.hints] use org.netbeans.modules.java.hints.legacy.spi which is deprecated: Use Java Hints SPI (org.netbeans.spi.java.hints) instead.
WARNING [org.openide.filesystems.Ordering]: Found same position 127 for both Services/MIMEResolver/org-netbeans-modules-javascript-nodejs-file-PackageLockJsonDataObject-Registration.xml and Services/MIMEResolver/org-netbeans-modules-javascript-nodejs-file-NpmDebugLogDataObject-Registration.xml
WARNING [org.openide.filesystems.Ordering]: Found same position 127 for both Services/MIMEResolver/org-netbeans-modules-javascript-nodejs-file-PackageLockJsonDataObject-Registration.xml and Services/MIMEResolver/org-netbeans-modules-javascript-nodejs-file-PackageJsonDataObject-Registration.xml
WARNING [org.openide.filesystems.Ordering]: Found same position 127 for both Services/MIMEResolver/org-netbeans-modules-javascript-nodejs-file-NpmDebugLogDataObject-Registration.xml and Services/MIMEResolver/org-netbeans-modules-javascript-nodejs-file-PackageJsonDataObject-Registration.xml
WARNING [org.openide.filesystems.Ordering]: Found same position 127 for both Services/MIMEResolver/org-netbeans-modules-javascript-nodejs-file-PackageLockJsonDataObject-Registration.xml and Services/MIMEResolver/org-netbeans-modules-javascript-nodejs-file-NpmDebugLogDataObject-Registration.xml
WARNING [org.openide.filesystems.Ordering]: Found same position 127 for both Services/MIMEResolver/org-netbeans-modules-javascript-nodejs-file-PackageLockJsonDataObject-Registration.xml and Services/MIMEResolver/org-netbeans-modules-javascript-nodejs-file-PackageJsonDataObject-Registration.xml
WARNING [org.openide.filesystems.Ordering]: Found same position 127 for both Services/MIMEResolver/org-netbeans-modules-javascript-nodejs-file-NpmDebugLogDataObject-Registration.xml and Services/MIMEResolver/org-netbeans-modules-javascript-nodejs-file-PackageJsonDataObject-Registration.xml
WARNING [null]: Last record repeated again.
WARNING [org.openide.filesystems.Ordering]: Found same position 127 for both Services/MIMEResolver/org-netbeans-modules-javascript-nodejs-file-NpmDebugLogDataObject-Registration.xml and Services/MIMEResolver/org-netbeans-modules-javascript-nodejs-file-PackageLockJsonDataObject-Registration.xml
WARNING [org.openide.filesystems.Ordering]: Found same position 127 for both Services/MIMEResolver/org-netbeans-modules-javascript-nodejs-file-PackageJsonDataObject-Registration.xml and Services/MIMEResolver/org-netbeans-modules-javascript-nodejs-file-PackageLockJsonDataObject-Registration.xml
WARNING [org.netbeans.modules.java.j2seplatform.libraries.J2SELibraryTypeProvider]: Can not resolve URL: nbinst://org.netbeans.modules.j2ee.eclipselink/modules/ext/docs/javax.persistence-2.1.0-doc.zip
WARNING [null]: Last record repeated again.
WARNING [org.netbeans.modules.java.j2seplatform.libraries.J2SELibraryTypeProvider]: Can not resolve URL: nbinst://org.netbeans.modules.j2ee.eclipselink/modules/ext/eclipselink/eclipselink.jar
WARNING [null]: Last record repeated again.
WARNING [org.netbeans.modules.java.j2seplatform.libraries.J2SELibraryTypeProvider]: Can not resolve URL: nbinst://org.netbeans.libs.javacapi/modules/ext/nb-javac-api.jar
WARNING [null]: Last record repeated again.
WARNING [org.netbeans.modules.java.j2seplatform.libraries.J2SELibraryTypeProvider]: Can not resolve URL: nbinst://org.netbeans.libs.jaxb/modules/ext/jaxb/jaxb-impl.jar
WARNING [org.netbeans.modules.java.j2seplatform.libraries.J2SELibraryTypeProvider]: Can not resolve URL: nbinst://org.netbeans.libs.jaxb/modules/ext/jaxb/jaxb-xjc.jar
WARNING [org.netbeans.modules.java.j2seplatform.libraries.J2SELibraryTypeProvider]: Can not resolve URL: nbinst://org.netbeans.modules.xml.jaxb.api/modules/ext/jaxb/api/jsr173_1.0_api.jar
WARNING [org.netbeans.modules.java.j2seplatform.libraries.J2SELibraryTypeProvider]: Can not resolve URL: nbinst://org.netbeans.libs.jaxb/modules/ext/jaxb/jaxb-impl.jar
WARNING [org.netbeans.modules.java.j2seplatform.libraries.J2SELibraryTypeProvider]: Can not resolve URL: nbinst://org.netbeans.libs.jaxb/modules/ext/jaxb/jaxb-xjc.jar
WARNING [org.netbeans.modules.java.j2seplatform.libraries.J2SELibraryTypeProvider]: Can not resolve URL: nbinst://org.netbeans.modules.xml.jaxb.api/modules/ext/jaxb/api/jsr173_1.0_api.jar
WARNING [org.netbeans.modules.java.j2seplatform.libraries.J2SELibraryTypeProvider]: Can not resolve URL: nbinst://org.netbeans.modules.j2ee.eclipselink/modules/ext/docs/javax.persistence-2.1.0-doc.zip
WARNING [null]: Last record repeated again.
//Had to remove turning on modules... due to 30000 character limit
INFO [org.netbeans.core.netigso.Netigso]: bundle org.eclipse.mylyn.wikitext.markdown.core#2.6.0.v20150901-2143 resolved
INFO [org.netbeans.core.netigso.Netigso]: bundle org.eclipse.mylyn.wikitext.textile.core#2.6.0.v20150901-2143 resolved
INFO [org.netbeans.core.netigso.Netigso]: bundle org.eclipse.jgit.java7#3.6.2.201501210735-r resolved
INFO [org.netbeans.core.netigso.Netigso]: bundle org.eclipse.mylyn.wikitext.confluence.core#2.6.0.v20150901-2143 resolved
INFO [org.netbeans.core.netigso.Netigso]: bundle net.java.html.sound#1.6.1 resolved
INFO [org.netbeans.core.netigso.Netigso]: bundle net.java.html.boot.script#1.6.1 resolved
INFO [org.netbeans.core.netigso.Netigso]: bundle net.java.html.geo#1.6.1 resolved
INFO [org.netbeans.core.netigso.Netigso]: bundle org.eclipse.osgi#3.9.1.v20140110-1610 started
INFO [org.netbeans.core.network.proxy.NetworkProxyReloader]: System network proxy resolver: Windows
INFO [org.netbeans.core.network.proxy.windows.WindowsNetworkProxy]: Windows system proxy resolver: auto detect
INFO [org.netbeans.core.network.proxy.NetworkProxyReloader]: System network proxy reloading succeeded.
INFO [org.netbeans.core.network.proxy.NetworkProxyReloader]: System network proxy - mode: direct
INFO [org.netbeans.core.network.proxy.NetworkProxyReloader]: System network proxy: fell to default (correct if direct mode went before)
WARNING [org.netbeans.TopSecurityManager]: use of system property netbeans.home has been obsoleted in favor of InstalledFileLocator/Places at org.netbeans.modules.java.j2seplatform.platformdefinition.Util.removeNBArtifacts(Util.java:337)
Diagnostic information
Input arguments:
-Dnetbeans.importclass=org.netbeans.upgrade.AutoUpgrade
-XX:+UseStringDeduplication
-Djdk.lang.Process.allowAmbiguousCommands=true
-Xss2m
-Djdk.gtk.version=2.2
-Dapple.laf.useScreenMenuBar=true
-Dapple.awt.graphics.UseQuartz=true
-Dsun.java2d.noddraw=true
-Dsun.java2d.dpiaware=true
-Dsun.zip.disableMemoryMapping=true
-Dplugin.manager.check.updates=false
-Dnetbeans.extbrowser.manual_chrome_plugin_install=yes
--add-opens=java.base/java.net=ALL-UNNAMED
--add-opens=java.base/java.lang.ref=ALL-UNNAMED
--add-opens=java.base/java.lang=ALL-UNNAMED
--add-opens=java.base/java.security=ALL-UNNAMED
--add-opens=java.base/java.util=ALL-UNNAMED
--add-opens=java.desktop/javax.swing.plaf.basic=ALL-UNNAMED
--add-opens=java.desktop/javax.swing.text=ALL-UNNAMED
--add-opens=java.desktop/javax.swing=ALL-UNNAMED
--add-opens=java.desktop/java.awt=ALL-UNNAMED
--add-opens=java.desktop/java.awt.event=ALL-UNNAMED
--add-opens=java.prefs/java.util.prefs=ALL-UNNAMED
--add-opens=jdk.jshell/jdk.jshell=ALL-UNNAMED
--add-modules=jdk.jshell
--add-exports=java.desktop/sun.awt=ALL-UNNAMED
--add-exports=java.desktop/java.awt.peer=ALL-UNNAMED
--add-exports=java.desktop/com.sun.beans.editors=ALL-UNNAMED
--add-exports=java.desktop/sun.swing=ALL-UNNAMED
--add-exports=java.desktop/sun.awt.im=ALL-UNNAMED
--add-exports=jdk.internal.jvmstat/sun.jvmstat.monitor=ALL-UNNAMED
--add-exports=java.management/sun.management=ALL-UNNAMED
--add-exports=java.base/sun.reflect.annotation=ALL-UNNAMED
--add-exports=jdk.javadoc/com.sun.tools.javadoc.main=ALL-UNNAMED
-XX:+IgnoreUnrecognizedVMOptions
-Djdk.home=C:\Program Files\Java\jdk-13.0.1
-Dnetbeans.home=C:\netbeans\platform
-Dnetbeans.user=C:\Users\banhi\AppData\Roaming\NetBeans\11.2
-Dnetbeans.default_userdir_root=C:\Users\banhi\AppData\Roaming\NetBeans
-XX:+HeapDumpOnOutOfMemoryError
-XX:HeapDumpPath=C:\Users\banhi\AppData\Roaming\NetBeans\11.2\var\log\heapdump.hprof
-Dsun.awt.keepWorkingSetOnMinimize=true
-Dnetbeans.dirs=C:\netbeans\nb;C:\netbeans\ergonomics;C:\netbeans\ide;C:\netbeans\extide;C:\netbeans\java;C:\netbeans\apisupport;C:\netbeans\webcommon;C:\netbeans\websvccommon;C:\netbeans\enterprise;C:\netbeans\mobility;C:\netbeans\profiler;C:\netbeans\python;C:\netbeans\php;C:\netbeans\identity;C:\netbeans\harness;C:\netbeans\cnd;C:\netbeans\cndext;C:\netbeans\dlight;C:\netbeans\groovy;C:\netbeans\extra;C:\netbeans\javacard;C:\netbeans\javafx
exit
Compiler: HotSpot 64-Bit Tiered Compilers
Heap memory usage: initial 256,0MB maximum 4082,0MB
Non heap memory usage: initial 7,3MB maximum -1b
Garbage collector: G1 Young Generation (Collections=21 Total time spent=0s)
Garbage collector: G1 Old Generation (Collections=0 Total time spent=0s)
Classes: loaded=11437 total loaded=11442 unloaded 5
INFO [org.netbeans.core.ui.warmup.DiagnosticTask]: Total memory 17 114 759 168
INFO [org.netbeans.modules.parsing.impl.indexing.RepositoryUpdater]: Resolving dependencies took: 16 ms
INFO [org.netbeans.modules.parsing.impl.indexing.RepositoryUpdater]: Complete indexing of 32 binary roots took: 334 ms
INFO [org.netbeans.modules.parsing.impl.indexing.RepositoryUpdater]: Indexing of: C:\Users\banhi\Documents\NetBeansProjects\NASAInSight\src took: 280 ms (New or modified files: 0, Deleted files: 0) [Adding listeners took: 1 ms]
INFO [org.netbeans.modules.parsing.impl.indexing.RepositoryUpdater]: Indexing of: C:\Users\banhi\Documents\NetBeansProjects\test\src took: 46 ms (New or modified files: 0, Deleted files: 0) [Adding listeners took: 0 ms]
INFO [org.netbeans.modules.parsing.impl.indexing.RepositoryUpdater]: Indexing of: C:\Users\banhi\Documents\NetBeansProjects\NASAInSight\test took: 0 ms (New or modified files: 0, Deleted files: 0) [Adding listeners took: 0 ms]
INFO [org.netbeans.modules.parsing.impl.indexing.RepositoryUpdater]: Indexing of: C:\Users\banhi\Documents\NetBeansProjects\test\test took: 0 ms (New or modified files: 0, Deleted files: 0) [Adding listeners took: 0 ms]
INFO [org.netbeans.modules.parsing.impl.indexing.RepositoryUpdater]: Complete indexing of 4 source roots took: 326 ms (New or modified files: 0, Deleted files: 0) [Adding listeners took: 1 ms]
WARNING [org.netbeans.TopSecurityManager]: use of system property netbeans.user has been obsoleted in favor of InstalledFileLocator/Places at org.netbeans.modules.java.api.common.project.ActionProviderSupport.verifyUserPropertiesFile(ActionProviderSupport.java:927)
WARNING [org.netbeans.modules.options.keymap.LayersBridge]: Invalid shortcut: org.openide.loaders.XMLDataObject#66d7a096[MultiFileObject#286f4280[Actions/Help/master-help.xml]]
WARNING [org.netbeans.modules.options.keymap.LayersBridge]: Invalid shortcut: org.openide.loaders.BrokenDataShadow#b4648e[MultiFileObject#2acdee67[Keymaps/NetBeans/D-BACK_QUOTE.shadow]]
WARNING [null]: Last record repeated again.
INFO [org.netbeans.modules.parsing.impl.indexing.RepositoryUpdater]: Resolving dependencies took: 15 ms
INFO [org.netbeans.modules.parsing.impl.indexing.RepositoryUpdater]: Complete indexing of 0 binary roots took: 1 ms
INFO [org.netbeans.modules.parsing.impl.indexing.RepositoryUpdater]: Indexing of: C:\netbeans\webcommon\jsstubs\reststubs.zip took: 98 ms (New or modified files: 0, Deleted files: 0) [Adding listeners took: 0 ms]
INFO [org.netbeans.modules.parsing.impl.indexing.RepositoryUpdater]: Indexing of: C:\netbeans\webcommon\jsstubs\corestubs.zip took: 46 ms (New or modified files: 0, Deleted files: 0) [Adding listeners took: 0 ms]
INFO [org.netbeans.modules.parsing.impl.indexing.RepositoryUpdater]: Indexing of: C:\netbeans\webcommon\jsstubs\domstubs.zip took: 49 ms (New or modified files: 0, Deleted files: 0) [Adding listeners took: 0 ms]
INFO [org.netbeans.modules.parsing.impl.indexing.RepositoryUpdater]: Complete indexing of 3 source roots took: 193 ms (New or modified files: 0, Deleted files: 0) [Adding listeners took: 0 ms]
INFO [org.netbeans.modules.mercurial]: version: null
INFO [org.netbeans.modules.subversion]: Finished indexing svn cache with 0 entries. Elapsed time: 0 ms.
INFO [org.netbeans.modules.parsing.impl.indexing.RepositoryUpdater]: Resolving dependencies took: 48 ms
INFO [org.netbeans.modules.parsing.impl.indexing.RepositoryUpdater]: Complete indexing of 20 binary roots took: 2 938 ms
INFO [org.netbeans.modules.parsing.impl.indexing.RepositoryUpdater]: Indexing of: C:\Users\banhi\Documents\NetBeansProjects\NASAInSight\src took: 385 ms (New or modified files: 0, Deleted files: 0) [Adding listeners took: 0 ms]
INFO [org.netbeans.modules.parsing.impl.indexing.RepositoryUpdater]: Complete indexing of 1 source roots took: 385 ms (New or modified files: 0, Deleted files: 0) [Adding listeners took: 0 ms]
INFO [org.netbeans.core.netigso.Netigso]: bundle org.eclipse.osgi#3.9.1.v20140110-1610 256
WARNING [org.netbeans.modules.progress.spi.InternalHandle]: Cannot switch to silent mode when not running at org.netbeans.core.ui.warmup.MenuWarmUpTask$NbWindowsAdapter$1HandleBridge.run(MenuWarmUpTask.java:244)
INFO [org.netbeans.core.netigso.Netigso]: bundle org.eclipse.osgi#3.9.1.v20140110-1610 stopped
INFO [null]: Last record repeated again.
WARNING [org.netbeans.modules.progress.spi.InternalHandle]: Cannot switch to silent mode when not running at org.netbeans.core.ui.warmup.MenuWarmUpTask$NbWindowsAdapter$1HandleBridge.run(MenuWarmUpTask.java:244)
WARNING [null]: Last record repeated more than 10 times, further logs of this record are ignored until the log record changes.
I am trying to execute MultiGpuLenetMnistExample.java
and i have received following error
"
...
12:41:24.129 [main] INFO Test - Load data....
12:41:24.716 [main] INFO Test - Build model....
12:41:25.500 [main] INFO org.nd4j.linalg.factory.Nd4jBackend - Loaded [JCublasBackend] backend
ND4J CUDA build version: 10.1.243
CUDA device 0: [Quadro K4000]; cc: [3.0]; Total memory: [3221225472];
12:41:26.692 [main] INFO org.nd4j.nativeblas.NativeOpsHolder - Number of threads used for OpenMP: 32
12:41:26.746 [main] INFO org.nd4j.nativeblas.Nd4jBlas - Number of threads used for OpenMP BLAS: 0
12:41:26.755 [main] INFO org.nd4j.linalg.api.ops.executioner.DefaultOpExecutioner - Backend used: [CUDA]; OS: [Windows 8.1]
12:41:26.755 [main] INFO org.nd4j.linalg.api.ops.executioner.DefaultOpExecutioner - Cores: [24]; Memory: [3,5GB];
12:41:26.755 [main] INFO org.nd4j.linalg.api.ops.executioner.DefaultOpExecutioner - Blas vendor: [CUBLAS]
12:41:26.755 [main] INFO org.nd4j.linalg.jcublas.ops.executioner.CudaExecutioner - Device Name: [Quadro K4000]; CC: [3.0]; Total/free memory: [3221225472]
12:41:26.844 [main] INFO org.deeplearning4j.nn.multilayer.MultiLayerNetwork - Starting MultiLayerNetwork with WorkspaceModes set to [training: ENABLED; inference: ENABLED], cacheMode set to [NONE]
12:41:27.957 [main] DEBUG org.nd4j.jita.allocator.impl.MemoryTracker - Free memory on device_0: 2709856256
Exception in thread "main" java.lang.RuntimeException: cudaGetSymbolAddress(...) failed; Error code: [13]
at org.nd4j.linalg.jcublas.ops.executioner.CudaExecutioner.createShapeInfo(CudaExecutioner.java:2557)
at org.nd4j.linalg.api.shape.Shape.createShapeInformation(Shape.java:3282)
at org.nd4j.linalg.api.ndarray.BaseShapeInfoProvider.createShapeInformation(BaseShapeInfoProvider.java:76)
at org.nd4j.jita.constant.ProtectedCudaShapeInfoProvider.createShapeInformation(ProtectedCudaShapeInfoProvider.java:96)
at org.nd4j.jita.constant.ProtectedCudaShapeInfoProvider.createShapeInformation(ProtectedCudaShapeInfoProvider.java:77)
at org.nd4j.linalg.jcublas.CachedShapeInfoProvider.createShapeInformation(CachedShapeInfoProvider.java:44)
at org.nd4j.linalg.api.ndarray.BaseNDArray.<init>(BaseNDArray.java:211)
at org.nd4j.linalg.jcublas.JCublasNDArray.<init>(JCublasNDArray.java:383)
at org.nd4j.linalg.jcublas.JCublasNDArrayFactory.create(JCublasNDArrayFactory.java:1543)
at org.nd4j.linalg.jcublas.JCublasNDArrayFactory.create(JCublasNDArrayFactory.java:1538)
at org.nd4j.linalg.factory.Nd4j.create(Nd4j.java:4298)
at org.nd4j.linalg.factory.Nd4j.create(Nd4j.java:3986)
at org.deeplearning4j.nn.multilayer.MultiLayerNetwork.init(MultiLayerNetwork.java:688)
at org.deeplearning4j.nn.multilayer.MultiLayerNetwork.init(MultiLayerNetwork.java:604)
at Test.main(Test.java:80)
Process finished with exit code 1 "
is there any workaround about this problem?
2 options here: either build dl4j from sources for your target compute capability (3.0) or wait for next release, since we’re going to bring it back for 1 additional release.
At this point cc 3.0 is just considered deprecated by most frameworks afaik 😞
I'm trying to deploy the following streams:
STREAM_2=:messages > filter --expression="#jsonPath(payload, '$.id')==1" | rabbit --queues=id_1 --host=rabbitmq --routing-key=id_1 --exchange=ex_1 --own-connection=true
STREAM_3=:messages > filter --expression="#jsonPath(payload, '$.id')==2" | rabbit --queues=id_2 --host=rabbitmq --routing-key=id_2 --exchange=ex_1
STREAM_4=:messages > filter --expression="#jsonPath(payload, '$.id')==3" | rabbit --queues=id_3 --host=rabbitmq --routing-key=id_3 --exchange=ex_1
STREAM_1=rabbit --queues=hello_queue --host=rabbitmq > :messages
Visualization:
I'm listening for a queue and then sending the message to a different queue depending on one of the message's attributes.
I'm running a local system, using this docker-compose.yml, but I switched to RabbitMQ instead of Kafka for communication.
When I deploy the streams, it takes a couple of minutes until the dataflow-server container reaches the max memory usage, and finally fails on random streams (and sometimes kills the container).
The logs (both stdout and stderr) don't show errors.
I'm running with the latest versions as follows:
DATAFLOW_VERSION=2.0.1.RELEASE SKIPPER_VERSION=2.0.0.RELEASE docker-compose up
Another thing I noticed, in the logs I keep getting:
2019-03-27 09:35:00.485 WARN 70 --- [| adminclient-1] org.apache.kafka.clients.NetworkClient : [AdminClient clientId=adminclient-1] Connection to node -1 could not be established. Broker may not be available.
although I have nothing related to Kafka in my docker-compose.yml. Any ideas where it's coming from?
Relevant parts from my YAML:
version: '3'
services:
mysql:
image: mysql:5.7.25
environment:
MYSQL_DATABASE: dataflow
MYSQL_USER: root
MYSQL_ROOT_PASSWORD: rootpw
expose:
- 3306
dataflow-server:
image: springcloud/spring-cloud-dataflow-server:${DATAFLOW_VERSION:?DATAFLOW_VERSION is not set!}
container_name: dataflow-server
ports:
- "9393:9393"
environment:
- spring.datasource.url=jdbc:mysql://mysql:3306/dataflow
- spring.datasource.username=root
- spring.datasource.password=rootpw
- spring.datasource.driver-class-name=org.mariadb.jdbc.Driver
- spring.cloud.skipper.client.serverUri=http://skipper-server:7577/api
- spring.cloud.dataflow.applicationProperties.stream.spring.rabbitmq.host=rabbitmq
depends_on:
- rabbitmq
rabbitmq:
image: "rabbitmq:3-management"
ports:
- "5672:5672"
- "15672:15672"
expose:
- "5672"
app-import:
...
skipper-server:
image: springcloud/spring-cloud-skipper-server:${SKIPPER_VERSION:?SKIPPER_VERSION is not set!}
container_name: skipper
ports:
- "7577:7577"
- "9000-9010:9000-9010"
volumes:
scdf-targets:
Looks like I were a victim of the OOM killer. The container was crashing with an exit code of 137.
The easiest solution for me now is giving Docker more memory:
CONTAINER ID NAME CPU % MEM USAGE / LIMIT MEM % NET I/O BLOCK I/O PIDS
9a0e3ff0beb8 dataflow-server 0.18% 1.293GiB / 11.71GiB 11.04% 573kB / 183kB 92.1MB / 279kB 49
2a448b3583a3 scdf_kafka_1 7.00% 291.6MiB / 11.71GiB 2.43% 4.65MB / 3.64MB 40.4MB / 36.9kB 73
eb9a70ce2a0e scdf_rabbitmq_1 2.15% 94.21MiB / 11.71GiB 0.79% 172kB / 92.5kB 41.7MB / 139kB 128
06dd2d6a1501 scdf_zookeeper_1 0.16% 81.72MiB / 11.71GiB 0.68% 77.8kB / 99.2kB 36.7MB / 45.1kB 25
1f1b782ad66d skipper 8.64% 6.55GiB / 11.71GiB 55.93% 3.63MB / 4.73MB 213MB / 0B 324
The skipper container is now using 6.55GiB memory, if someone knows what it could be, I would be grateful.
For now, I'm accepting my answer since it does provide a workaround, although I feel there could be a better solution than increasing the memory limit for Docker.
EDIT:
Looks like this is indeed the solution, from this GitHub issue:
Stream components (parts of the pipe) are deployed as applications. Those applications are deployed into the Skipper container (as well as the Skipper application itself) since skipper deploys streams. The more applications that get deployed (parts of the pipe, streams, etc) the more memory is used.
I'm trying to use the command:
movescu -b MOVESCU:11133 -c AE_ARCH1#192.168.2.63:104 --dest MOVESCU -m AccessionNumber=ZH171217DR027
It doesn't find anything, I'm sure there's a result.
use "movescu -b MOVESCU:11133 -c AE_ARCH1#192.168.2.63:104 --dest MOVESCU -m StudyInstanceUID=1.2.840.113619.186.1441772842175112.20180117144250009.393" it
work well.
use "movescu -b MOVESCU:11133 -c AE_ARCH1#192.168.2.63:104 --dest MOVESCU -m AccessionNumber=ZH171217DR027" log:
E:\dcm4che-5.11.0\bin>movescu -b MOVESCU:11133 -c AE_ARCH1#192.168.2.63:104 -
-dest MOVESCU -m AccessionNumber=ZH171217DR027
19:39:19,102 INFO - Initiate connection from 0.0.0.0/0.0.0.0:0 to 192.168.2.63:
104
19:39:19,117 INFO - Established connection Socket[addr=/192.168.2.63,port=104,l
ocalport=8028]
19:39:19,117 DEBUG - /172.16.23.232:8028->/192.168.2.63:104(1): enter state: Sta
4 - Awaiting transport connection opening to complete
19:39:19,133 INFO - MOVESCU->AE_ARCH1(1) << A-ASSOCIATE-RQ
19:39:19,148 DEBUG - A-ASSOCIATE-RQ[
calledAET: AE_ARCH1
callingAET: MOVESCU
applicationContext: 1.2.840.10008.3.1.1.1 - DICOM Application Context Name
implClassUID: 1.2.40.0.13.1.3
implVersionName: dcm4che-5.11.0
maxPDULength: 16378
maxOpsInvoked/maxOpsPerformed: 0/0
PresentationContext[id: 1
as: 1.2.840.10008.5.1.4.1.2.2.2 - Study Root Query/Retrieve Information Mode
l - MOVE
ts: 1.2.840.10008.1.2 - Implicit VR Little Endian
ts: 1.2.840.10008.1.2.1 - Explicit VR Little Endian
ts: 1.2.840.10008.1.2.2 - Explicit VR Big Endian (Retired)
]
]
19:39:19,148 DEBUG - MOVESCU->AE_ARCH1(1): enter state: Sta5 - Awaiting A-ASSOCI
ATE-AC or A-ASSOCIATE-RJ PDU
19:39:19,164 INFO - MOVESCU->AE_ARCH1(1) >> A-ASSOCIATE-AC
19:39:19,195 DEBUG - A-ASSOCIATE-AC[
calledAET: AE_ARCH1
callingAET: MOVESCU
applicationContext: 1.2.840.10008.3.1.1.1 - DICOM Application Context Name
implClassUID: 1.2.528.1.1001.2.800.5.0.3020.0
implVersionName: EA 5.0.3020.0
maxPDULength: 16384
maxOpsInvoked/maxOpsPerformed: 0/1
PresentationContext[id: 1
result: 0 - acceptance
ts: 1.2.840.10008.1.2.1 - Explicit VR Little Endian
]
]
19:39:19,195 DEBUG - MOVESCU->AE_ARCH1(1): enter state: Sta6 - Association estab
lished and ready for data transfer
19:39:19,195 INFO - MOVESCU->AE_ARCH1(1) << 1:C-MOVE-RQ[pcid=1, prior=0
cuid=1.2.840.10008.5.1.4.1.2.2.2 - Study Root Query/Retrieve Information Model
- MOVE
tsuid=1.2.840.10008.1.2.1 - Explicit VR Little Endian
19:39:19,226 DEBUG - Command:
(0000,0002) UI [1.2.840.10008.5.1.4.1.2.2.2] AffectedSOPClassUID
(0000,0100) US [33] CommandField
(0000,0110) US [1] MessageID
(0000,0600) AE [MOVESCU] MoveDestination
(0000,0700) US [0] Priority
(0000,0800) US [0] CommandDataSetType
19:39:19,242 DEBUG - Dataset:
(0008,0050) SH [ZH171217DR027] AccessionNumber
(0008,0052) CS [STUDY] QueryRetrieveLevel
19:39:19,258 INFO - MOVESCU->AE_ARCH1(1) >> 1:C-MOVE-RSP[pcid=1, status=120H
cuid=1.2.840.10008.5.1.4.1.2.2.2 - Study Root Query/Retrieve Information Model
- MOVE
tsuid=1.2.840.10008.1.2.1 - Explicit VR Little Endian
19:39:19,258 DEBUG - Command:
(0000,0002) UI [1.2.840.10008.5.1.4.1.2.2.2] AffectedSOPClassUID
(0000,0100) US [32801] CommandField
(0000,0120) US [1] MessageIDBeingRespondedTo
(0000,0800) US [257] CommandDataSetType
(0000,0900) US [288] Status
19:39:19,258 INFO - MOVESCU->AE_ARCH1(1) << A-RELEASE-RQ
19:39:19,258 DEBUG - MOVESCU->AE_ARCH1(1): enter state: Sta7 - Awaiting A-RELEAS
E-RP PDU
19:39:19,258 INFO - MOVESCU->AE_ARCH1(1) >> A-RELEASE-RP
19:39:19,258 INFO - MOVESCU->AE_ARCH1(1): close Socket[addr=/192.168.2.63,port=
104,localport=8028]
19:39:19,258 DEBUG - MOVESCU->AE_ARCH1(1): enter state: Sta1 - Idle
use"movescu -b MOVESCU:11133 -c AE_ARCH1#192.168.2.63:104 --dest MOVESCU -m StudyInstanceUID=1.2.840.113619.186.1441772842175112.20180117144250009.393" log :
19:46:31,536 INFO - Initiate connection from 0.0.0.0/0.0.0.0:0 to 192.168.2.63:
104
19:46:31,552 INFO - Established connection Socket[addr=/192.168.2.63,port=104,l
ocalport=8094]
19:46:31,552 DEBUG - /172.16.23.232:8094->/192.168.2.63:104(1): enter state: Sta
4 - Awaiting transport connection opening to complete
19:46:31,552 INFO - MOVESCU->AE_ARCH1(1) << A-ASSOCIATE-RQ
19:46:31,568 DEBUG - A-ASSOCIATE-RQ[
calledAET: AE_ARCH1
callingAET: MOVESCU
applicationContext: 1.2.840.10008.3.1.1.1 - DICOM Application Context Name
implClassUID: 1.2.40.0.13.1.3
implVersionName: dcm4che-5.11.0
maxPDULength: 16378
maxOpsInvoked/maxOpsPerformed: 0/0
PresentationContext[id: 1
as: 1.2.840.10008.5.1.4.1.2.2.2 - Study Root Query/Retrieve Information Mode
l - MOVE
ts: 1.2.840.10008.1.2 - Implicit VR Little Endian
ts: 1.2.840.10008.1.2.1 - Explicit VR Little Endian
ts: 1.2.840.10008.1.2.2 - Explicit VR Big Endian (Retired)
]
]
19:46:31,568 DEBUG - MOVESCU->AE_ARCH1(1): enter state: Sta5 - Awaiting A-ASSOCI
ATE-AC or A-ASSOCIATE-RJ PDU
19:46:31,583 INFO - MOVESCU->AE_ARCH1(1) >> A-ASSOCIATE-AC
19:46:31,646 DEBUG - A-ASSOCIATE-AC[
calledAET: AE_ARCH1
callingAET: MOVESCU
applicationContext: 1.2.840.10008.3.1.1.1 - DICOM Application Context Name
implClassUID: 1.2.528.1.1001.2.800.5.0.3020.0
implVersionName: EA 5.0.3020.0
maxPDULength: 16384
maxOpsInvoked/maxOpsPerformed: 0/1
PresentationContext[id: 1
result: 0 - acceptance
ts: 1.2.840.10008.1.2.1 - Explicit VR Little Endian
]
]
19:46:31,646 DEBUG - MOVESCU->AE_ARCH1(1): enter state: Sta6 - Association estab
lished and ready for data transfer
19:46:31,646 INFO - MOVESCU->AE_ARCH1(1) << 1:C-MOVE-RQ[pcid=1, prior=0
cuid=1.2.840.10008.5.1.4.1.2.2.2 - Study Root Query/Retrieve Information Model
- MOVE
tsuid=1.2.840.10008.1.2.1 - Explicit VR Little Endian
19:46:31,677 DEBUG - Command:
(0000,0002) UI [1.2.840.10008.5.1.4.1.2.2.2] AffectedSOPClassUID
(0000,0100) US [33] CommandField
(0000,0110) US [1] MessageID
(0000,0600) AE [MOVESCU] MoveDestination
(0000,0700) US [0] Priority
(0000,0800) US [0] CommandDataSetType
19:46:31,692 DEBUG - Dataset:
(0008,0052) CS [STUDY] QueryRetrieveLevel
(0020,000D) UI [1.2.840.113619.186.1441772842175112.20180117144250009.393] Stu
19:46:36,762 INFO - MOVESCU->AE_ARCH1(1) >> 1:C-MOVE-RSP[pcid=1, status=0H
cuid=1.2.840.10008.5.1.4.1.2.2.2 - Study Root Query/Retrieve Information Model
- MOVE
tsuid=1.2.840.10008.1.2.1 - Explicit VR Little Endian
19:46:36,762 DEBUG - Command:
(0000,0002) UI [1.2.840.10008.5.1.4.1.2.2.2] AffectedSOPClassUID
(0000,0100) US [32801] CommandField
(0000,0120) US [1] MessageIDBeingRespondedTo
(0000,0800) US [257] CommandDataSetType
(0000,0900) US [0] Status
19:46:36,840 INFO - MOVESCU->AE_ARCH1(1) << A-RELEASE-RQ
19:46:36,840 DEBUG - MOVESCU->AE_ARCH1(1): enter state: Sta7 - Awaiting A-RELEAS
E-RP PDU
19:46:36,840 INFO - MOVESCU->AE_ARCH1(1) >> A-RELEASE-RP
19:46:36,840 INFO - MOVESCU->AE_ARCH1(1): close Socket[addr=/192.168.2.63,port=
104,localport=8094]
19:46:36,840 DEBUG - MOVESCU->AE_ARCH1(1): enter state: Sta1 - Idle
The different places is :
19:39:19,258 INFO - MOVESCU->AE_ARCH1(1) >> 1:C-MOVE-RSP[pcid=1, status=120H
cuid=1.2.840.10008.5.1.4.1.2.2.2 - Study Root Query/Retrieve Information Model
- MOVE
tsuid=1.2.840.10008.1.2.1 - Explicit VR Little Endian
19:39:19,258 DEBUG - Command:
(0000,0002) UI [1.2.840.10008.5.1.4.1.2.2.2] AffectedSOPClassUID
(0000,0100) US [32801] CommandField
(0000,0120) US [1] MessageIDBeingRespondedTo
(0000,0800) US [257] CommandDataSetType
(0000,0900) US [288] Status
and
19:46:36,762 INFO - MOVESCU->AE_ARCH1(1) >> 1:C-MOVE-RSP[pcid=1, status=0H
cuid=1.2.840.10008.5.1.4.1.2.2.2 - Study Root Query/Retrieve Information Model
- MOVE
tsuid=1.2.840.10008.1.2.1 - Explicit VR Little Endian
19:46:36,762 DEBUG - Command:
(0000,0002) UI [1.2.840.10008.5.1.4.1.2.2.2] AffectedSOPClassUID
(0000,0100) US [32801] CommandField
(0000,0120) US [1] MessageIDBeingRespondedTo
(0000,0800) US [257] CommandDataSetType
(0000,0900) US [0] Status
C-MOVE-RSP return value status=0H or status=120H I can't find meaning ，and (0000,0900)[288] may be i lost a parameters
I use dcm4che2 "dcmqr -L MOVESCU AE_ARCH1#192.168.2.63:104 -qAccessionNumber=ZH171231DR039 -cmove MOVESCU " it work
How to use the dcm4che3 movescu to query the PACS server use the AccessionNumber?
I do not know dcm4chee too well. But in DICOM, your request to move by Accession Number is not conformant, so I suspect that this would cause your issue. In DICOM, you would put a STUDY-Level C-FIND request to search for Study Instance UIDs matching the Accession Number and then perform a C-MOVE for each study matched subsequently.
As #kritzel_sw pointed out, doing a C-MOVE on anything else but Instance UID values is not conformant to the standard.
I also found the status code 120H referenced in the dcm4che3 source code on Github.
/**
* Failure: missing Attribute (120H): a required Attribute was not
* supplied.
* Used in N-CREATE-RSP.
* May contain:
* Modification List/Attribute List (no tag)
*/
public static final int MissingAttribute = 0x0120;
It looks like it's not supposed to be used in C-MOVE-RSP, but it makes sense - the necessary UID attributes are missing in the request.
The reason it works with dcmqr is exactly that, it does first the C-FIND and then the C-MOVE based on C-FIND results. dcm4che3 utilities are strictly separated C-FIND and C-MOVE tools.
I'm learning hadoop/pig/hive through running through tutorials on hortonworks.com
I have indeed tried to find a link to the tutorial, but unfortunately it only ships with the ISA image that they provide to you. It's not actually hosted on their website.
batting = load 'Batting.csv' using PigStorage(',');
runs = FOREACH batting GENERATE $0 as playerID, $1 as year, $8 as runs;
grp_data = GROUP runs by (year);
max_runs = FOREACH grp_data GENERATE group as grp,MAX(runs.runs) as max_runs;
join_max_run = JOIN max_runs by ($0, max_runs), runs by (year,runs);
join_data = FOREACH join_max_run GENERATE $0 as year, $2 as playerID, $1 as runs;
dump join_data;
I've copied their code exactly as it was stated in the tutorial and I'm getting this output:
2013-06-14 14:34:37,969 [main] INFO org.apache.pig.Main - Apache Pig version 0.11.1.1.3.0.0-107 (rexported) compiled May 20 2013, 03:04:35
2013-06-14 14:34:37,977 [main] INFO org.apache.pig.Main - Logging error messages to: /hadoop/mapred/taskTracker/hue/jobcache/job_201306140401_0020/attempt_201306140401_0020_m_000000_0/work/pig_1371245677965.log
2013-06-14 14:34:38,412 [main] INFO org.apache.pig.impl.util.Utils - Default bootup file /usr/lib/hadoop/.pigbootup not found
2013-06-14 14:34:38,598 [main] INFO org.apache.pig.backend.hadoop.executionengine.HExecutionEngine - Connecting to hadoop file system at: hdfs://sandbox:8020
2013-06-14 14:34:38,998 [main] INFO org.apache.pig.backend.hadoop.executionengine.HExecutionEngine - Connecting to map-reduce job tracker at: sandbox:50300
2013-06-14 14:34:40,819 [main] WARN org.apache.pig.PigServer - Encountered Warning IMPLICIT_CAST_TO_DOUBLE 1 time(s).
2013-06-14 14:34:40,827 [main] INFO org.apache.pig.tools.pigstats.ScriptState - Pig features used in the script: HASH_JOIN,GROUP_BY
2013-06-14 14:34:41,115 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MRCompiler - File concatenation threshold: 100 optimistic? false
2013-06-14 14:34:41,160 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.CombinerOptimizer - Choosing to move algebraic foreach to combiner
2013-06-14 14:34:41,201 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MRCompiler$LastInputStreamingOptimizer - Rewrite: POPackage->POForEach to POJoinPackage
2013-06-14 14:34:41,213 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MultiQueryOptimizer - MR plan size before optimization: 3
2013-06-14 14:34:41,213 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MultiQueryOptimizer - Merged 1 map-reduce splittees.
2013-06-14 14:34:41,214 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MultiQueryOptimizer - Merged 1 out of total 3 MR operators.
2013-06-14 14:34:41,214 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MultiQueryOptimizer - MR plan size after optimization: 2
2013-06-14 14:34:41,488 [main] INFO org.apache.pig.tools.pigstats.ScriptState - Pig script settings are added to the job
2013-06-14 14:34:41,551 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobControlCompiler - mapred.job.reduce.markreset.buffer.percent is not set, set to default 0.3
2013-06-14 14:34:41,555 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobControlCompiler - Using reducer estimator: org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.InputSizeReducerEstimator
2013-06-14 14:34:41,559 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.InputSizeReducerEstimator - BytesPerReducer=1000000000 maxReducers=999 totalInputFileSize=6398990
2013-06-14 14:34:41,559 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobControlCompiler - Setting Parallelism to 1
2013-06-14 14:34:44,244 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobControlCompiler - creating jar file Job5371236206169131677.jar
2013-06-14 14:34:49,495 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobControlCompiler - jar file Job5371236206169131677.jar created
2013-06-14 14:34:49,517 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobControlCompiler - Setting up multi store job
2013-06-14 14:34:49,529 [main] INFO org.apache.pig.data.SchemaTupleFrontend - Key [pig.schematuple] is false, will not generate code.
2013-06-14 14:34:49,530 [main] INFO org.apache.pig.data.SchemaTupleFrontend - Starting process to move generated code to distributed cacche
2013-06-14 14:34:49,530 [main] INFO org.apache.pig.data.SchemaTupleFrontend - Setting key [pig.schematuple.classes] with classes to deserialize []
2013-06-14 14:34:49,755 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher - 1 map-reduce job(s) waiting for submission.
2013-06-14 14:34:50,144 [JobControl] INFO org.apache.hadoop.mapreduce.lib.input.FileInputFormat - Total input paths to process : 1
2013-06-14 14:34:50,145 [JobControl] INFO org.apache.pig.backend.hadoop.executionengine.util.MapRedUtil - Total input paths to process : 1
2013-06-14 14:34:50,256 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher - 0% complete
2013-06-14 14:34:50,316 [JobControl] INFO com.hadoop.compression.lzo.GPLNativeCodeLoader - Loaded native gpl library
2013-06-14 14:34:50,444 [JobControl] INFO com.hadoop.compression.lzo.LzoCodec - Successfully loaded & initialized native-lzo library [hadoop-lzo rev cf4e7cbf8ed0f0622504d008101c2729dc0c9ff3]
2013-06-14 14:34:50,665 [JobControl] WARN org.apache.hadoop.io.compress.snappy.LoadSnappy - Snappy native library is available
2013-06-14 14:34:50,666 [JobControl] INFO org.apache.hadoop.util.NativeCodeLoader - Loaded the native-hadoop library
2013-06-14 14:34:50,666 [JobControl] INFO org.apache.hadoop.io.compress.snappy.LoadSnappy - Snappy native library loaded
2013-06-14 14:34:50,680 [JobControl] INFO org.apache.pig.backend.hadoop.executionengine.util.MapRedUtil - Total input paths (combined) to process : 1
2013-06-14 14:34:52,796 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher - HadoopJobId: job_201306140401_0021
2013-06-14 14:34:52,796 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher - Processing aliases batting,grp_data,max_runs,runs
2013-06-14 14:34:52,796 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher - detailed locations: M: batting[1,10],runs[2,7],max_runs[4,11],grp_data[3,11] C: max_runs[4,11],grp_data[3,11] R: max_runs[4,11]
2013-06-14 14:34:52,796 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher - More information at: http://sandbox:50030/jobdetails.jsp?jobid=job_201306140401_0021
2013-06-14 14:36:01,993 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher - 50% complete
2013-06-14 14:36:04,767 [main] WARN org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher - Ooops! Some job has failed! Specify -stop_on_failure if you want Pig to stop immediately on failure.
2013-06-14 14:36:04,768 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher - job job_201306140401_0021 has failed! Stop running all dependent jobs
2013-06-14 14:36:04,768 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher - 100% complete
2013-06-14 14:36:05,029 [main] ERROR org.apache.pig.tools.pigstats.SimplePigStats - ERROR 2106: Error executing an algebraic function
2013-06-14 14:36:05,030 [main] ERROR org.apache.pig.tools.pigstats.PigStatsUtil - 1 map reduce job(s) failed!
2013-06-14 14:36:05,042 [main] INFO org.apache.pig.tools.pigstats.SimplePigStats - Script Statistics:
HadoopVersion PigVersion UserId StartedAt FinishedAt Features
1.2.0.1.3.0.0-107 0.11.1.1.3.0.0-107 mapred 2013-06-14 14:34:41 2013-06-14 14:36:05 HASH_JOIN,GROUP_BY
Failed!
Failed Jobs:
JobId Alias Feature Message Outputs
job_201306140401_0021 batting,grp_data,max_runs,runs MULTI_QUERY,COMBINER Message: Job failed! Error - # of failed Map Tasks exceeded allowed limit. FailedCount: 1. LastFailedTask: task_201306140401_0021_m_000000
Input(s):
Failed to read data from "hdfs://sandbox:8020/user/hue/batting.csv"
Output(s):
Counters:
Total records written : 0
Total bytes written : 0
Spillable Memory Manager spill count : 0
Total bags proactively spilled: 0
Total records proactively spilled: 0
Job DAG:
job_201306140401_0021 -> null,
null
2013-06-14 14:36:05,042 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher - Failed!
2013-06-14 14:36:05,043 [main] ERROR org.apache.pig.tools.grunt.Grunt - ERROR 1066: Unable to open iterator for alias join_data
Details at logfile: /hadoop/mapred/taskTracker/hue/jobcache/job_201306140401_0020/attempt_201306140401_0020_m_000000_0/work/pig_1371245677965.log
When switching this part: MAX(runs.runs) to avg(runs.runs) then I am getting a completely different issue:
2013-06-14 14:38:25,694 [main] INFO org.apache.pig.Main - Apache Pig version 0.11.1.1.3.0.0-107 (rexported) compiled May 20 2013, 03:04:35
2013-06-14 14:38:25,695 [main] INFO org.apache.pig.Main - Logging error messages to: /hadoop/mapred/taskTracker/hue/jobcache/job_201306140401_0022/attempt_201306140401_0022_m_000000_0/work/pig_1371245905690.log
2013-06-14 14:38:26,198 [main] INFO org.apache.pig.impl.util.Utils - Default bootup file /usr/lib/hadoop/.pigbootup not found
2013-06-14 14:38:26,438 [main] INFO org.apache.pig.backend.hadoop.executionengine.HExecutionEngine - Connecting to hadoop file system at: hdfs://sandbox:8020
2013-06-14 14:38:26,824 [main] INFO org.apache.pig.backend.hadoop.executionengine.HExecutionEngine - Connecting to map-reduce job tracker at: sandbox:50300
2013-06-14 14:38:28,238 [main] ERROR org.apache.pig.tools.grunt.Grunt - ERROR 1070: Could not resolve avg using imports: [, org.apache.pig.builtin., org.apache.pig.impl.builtin.]
Details at logfile: /hadoop/mapred/taskTracker/hue/jobcache/job_201306140401_0022/attempt_201306140401_0022_m_000000_0/work/pig_1371245905690.log
Anybody know what the issue might be?
I am sure lot of people would have figured this out. I combined Eugene's solution with the original code from Hortonworks such that we get the exact output as specific in the tutorial.
Following code works and produces exact output as specified in the tutorial:
batting = LOAD 'Batting.csv' using PigStorage(',');
runs_raw = FOREACH batting GENERATE $0 as playerID, $1 as year, $8 as runs;
runs = FILTER runs_raw BY runs > 0;
grp_data = group runs by (year);
max_runs = FOREACH grp_data GENERATE group as grp, MAX(runs.runs) as max_runs;
join_max_run = JOIN max_runs by ($0, max_runs), runs by (year,runs);
join_data = FOREACH join_max_run GENERATE $0 as year, $2 as playerID, $1 as runs;
dump join_data;
Note: line "runs = FILTER runs_raw BY runs > 0;" is additional than what has been provided by Hortonworks, thanks to Eugene for sharing working code which I used to modify original Hortonworks code to make it work.
UDFs are case sensitive, so at least to answer the second part of your question - you'll need to use AVG(runs.runs) instead of avg(runs.runs)
It's likely that once you correct your syntax you'll get the original error you reported...
i am having the same exact same issue with exact same log output, but this solution doesn't work because i believe changing MAX with AVG here dumps the whole purpose of this hortonworks.com tutorial - it was to get the MAX runs by playerID for each year.
UPDATE
Finally i got it resolved - you have to either remove the first line in Batting.csv (column names) or edit your Pig Latin code like this:
batting = LOAD ‘Batting.csv’ using PigStorage(‘,’);
runs_raw = FOREACH batting GENERATE $0 as playerID, $1 as year, $8 as runs;
runs = FILTER runs_raw BY runs > 0;
grp_data = group runs by (year);
max_runs = FOREACH grp_data GENERATE group as grp, MAX(runs.runs) as max_runs;
dump max_runs;
After that you should be able to complete tutorial correctly and get the proper result.
It also looks like this is due to the "bug" in the older versions of Pig rhich was used in the tutorial
Please specify appropriate data type for playerID, year & runs like below:
runs = FOREACH batting GENERATE $0 as playerID:int, $1 as year:chararray, $8 as runs:int;
Not, it should work.