We Keep Coding

What is configuartion required to get data from object storage by SWIFT in Spark

I go through document but still it is very much confusing how to get data from swift.
I configured swift in my one linux machine. By using below command I am able to get container list,
swift -A https://acc.objectstorage.softlayer.net/auth/v1.0/ -U
username -K passwordkey list
I seen many blog for blumix(https://console.ng.bluemix.net/docs/services/AnalyticsforApacheSpark/index-gentopic1.html#genTopProcId2) and written the below code
sc.textFile("swift://container.myacct/file.xml")
I am looking to integrate in java spark. Where need to configure object storage credential in java code. Is there any sample code or blog?

This notebook illustrates a number of ways to load data using the Scala language. Scala runs on the JVM. Java and Scala classes can be freely mixed, no matter whether they reside in different projects or in the same. Looking at the mechanics of how Scala code interacts with Openstack Swift object storage should help guide you to craft a Java equivalent.
From the above notebook, here are some steps illustrating how to configure and extract data from an Openstack Swift Object Storage instance using the Stocator library using the Scala language. The swift url decomposes into:
swift2d :// container . myacct / filename.extension
^ ^ ^ ^
stocator name of namespace object storage
protocol container filename
Imports
import org.apache.spark.SparkContext
import scala.util.control.NonFatal
import play.api.libs.json.Json
val sqlctx = new SQLContext(sc)
val scplain = sqlctx.sparkContext
Sample Creds
// #hidden_cell
var credentials = scala.collection.mutable.HashMap[String, String](
"auth_url"->"https://identity.open.softlayer.com",
"project"->"object_storage_3xxxxxx3_xxxx_xxxx_xxxx_xxxxxxxxxxxx",
"project_id"->"6xxxxxxxxxx04fxxxxxxxxxx6xxxxxx7",
"region"->"dallas",
"user_id"->"cxxxxxxxxxxaxxxxxxxxxx1xxxxxxxxx",
"domain_id"->"cxxxxxxxxxxaxxyyyyyyxx1xxxxxxxxx",
"domain_name"->"853255",
"username"->"Admin_cxxxxxxxxxxaxxxxxxxxxx1xxxxxxxxx",
"password"->"""&M7372!FAKE""",
"container"->"notebooks",
"tenantId"->"undefined",
"filename"->"file.xml"
)
Helper Method
def setRemoteObjectStorageConfig(name:String, sc: SparkContext, dsConfiguration:String) : Boolean = {
try {
val result = scala.util.parsing.json.JSON.parseFull(dsConfiguration)
result match {
case Some(e:Map[String,String]) => {
val prefix = "fs.swift2d.service." + name
val hconf = sc.hadoopConfiguration
hconf.set("fs.swift2d.impl","com.ibm.stocator.fs.ObjectStoreFileSystem")
hconf.set(prefix + ".auth.url", e("auth_url") + "/v3/auth/tokens")
hconf.set(prefix + ".tenant", e("project_id"))
hconf.set(prefix + ".username", e("user_id"))
hconf.set(prefix + ".password", e("password"))
hconf.set(prefix + "auth.method", "keystoneV3")
hconf.set(prefix + ".region", e("region"))
hconf.setBoolean(prefix + ".public", true)
println("Successfully modified sparkcontext object with remote Object Storage Credentials using datasource name " + name)
println("")
return true
}
case None => println("Failed.")
return false
}
}
catch {
case NonFatal(exc) => println(exc)
return false
}
}
Load the Data
val setObjStor = setRemoteObjectStorageConfig("sparksql", scplain, Json.toJson(credentials.toMap).toString)
val data_rdd = scplain.textFile("swift2d://notebooks.sparksql/" + credentials("filename"))
data_rdd.take(5)

Accessing PowerDesigner Repository Models via COM

I have been trying to get live models directly from the PowerDesigner repository using the COM API without success. Here's what I've been trying in VBA:
Set pd = CreateObject("PowerDesigner.Application")
Set conn = pd.RepositoryConnection
conn.Open "", "", "ShhMahPW"
Set model = conn.FindChildByPath("Program/Project/Logical Models/MahLOM", PdOOM_Classes.cls_Model)
MsgBox model.ShortDescription 'This fails because model is null!
Similarly, I've been trying the same thing in Eclipse with the Java COM bridge:
Application pd = this.getApplicationHook();
//Make live connection to proxy repository
RepositoryConnection conn = new RepositoryConnection( pd.GetRepositoryConnection() );
conn.Open( "", "", ConnectionParams.PASSWORD );
BaseObject model = conn.FindChildByPath( "Program/Project/Logical Models/MahLOM",
PdOOM_Classes.cls_Model );
//Null model, COMException: "Action can not be performed. result = -2147467259"
System.out.println( model.GetShortDescription() )
Can someone please suggest a good way of diving into the repository? I have been able to confirm that I have a connection to the repo and then list the children at that top level. I am struggling to dig into folders beyond the root level. Thanks!

I knew that the model I was looking to pull down from the repo already existed in my local workspace. Really this was a refresh of the local workspaces models. To perform this, the method UpdateFromRepository() can be used!
So what I can do then is get a handle to the local PowerDesigner model and then call for an update before retrieving children. Note the casting from BaseObject to BaseModel for the sake of the refresh...
private BaseObject getModel(){
Application pd = this.getApplicationHook();
model = pd.OpenModel(this.basePath + this.modelName);
System.out.println( "Retrieving model updates from repository... ");
RepositoryConnection conn = new RepositoryConnection( pd.GetRepositoryConnection() );
conn.Open( "", "", ConnectionParams.PASSWORD);
boolean success = new BaseModel(model).UpdateFromRepository();
if( success )
System.out.println( "Update successful!" );
else
System.out.println( "Update failed. Check PowerDesigner settings." );
return this.model;
}

Your main problem is that the search ChildKind should be Cls_RepositoryModel, instead of PdOOM_Class.cls_Model.
option explicit
' assuming we're already connected
if RepositoryConnection.Connected then
Descent RepositoryConnection,""
end if
dim c
set c = RepositoryConnection.FindChildByPath("Folder_7/ConceptualDataModel_1", Cls_RepositoryModel)
if not c is nothing then
output "*** found object " & c.classname
end if
sub Descent(obj,ofs)
output ofs & obj.name & " - " & obj.ObjectType & " - " & obj.ClassName
if obj.ObjectType = "RepositoryModel" then exit sub
if obj.PermanentID = 3 then exit sub ' to save time, don't enter Library
if not obj.HasCollection("ChildObjects") then exit sub
dim c
for each c in obj.ChildObjects
Descent c,ofs & " "
next
end sub

VSAM file locking when writing to it using Java JDBC

This is my first time trying to read and write to a VSAM file. What I did was:
Created a Map for the File using VSE Navigator
Added the Java beans VSE Connector library to my eclipse Java project
Use the code show below to Write and Read to the KSDS file.
Reading the file is not a problem but when I tried to write to the file it only works if I go on the mainframe and close the File before running my java program but it locks the file for like an hour. You cannot open the file on the mainframe or do anything to it.
Anybody can help with this problem. Is there a special setting that I need to set up for the file on the mainframe ? Why do you first need to close the file on CICS to be able to write to it ? And why does it locks the file after writing to it ?
import java.io.BufferedReader;
import java.io.InputStreamReader;
import java.sql.*;
public class testVSAM {
public static void main(String argv[]){
Integer test = Integer.valueOf(2893);
String vsamCatalog = "VSESP.USER.CATALOG";
String FlightCluster = "FLIGHT.ORDERING.FLIGHTS";
String FlightMapName = "FLIGHT.TEST2.MAP";
try{
String ipAddr = "10.1.1.1";
String userID = "USER1";
String password = "PASSWORD";
java.sql.Connection jdbcCon;
java.sql.Driver jdbcDriver = (java.sql.Driver) Class.forName(
"com.ibm.vse.jdbc.VsamJdbcDriver").newInstance();
// Build the URL to use to connect
String url = "jdbc:vsam:"+ipAddr;
// Assign properties for the driver
java.util.Properties prop = new java.util.Properties();
prop.put("port", test);
prop.put("user", userID);
prop.put("password", password);
// Connect to the driver
jdbcCon = DriverManager.getConnection(url,prop);
try {
java.sql.PreparedStatement pstmt = jdbcCon.prepareStatement(
"INSERT INTO "+vsamCatalog+"\\"+FlightCluster+"\\"+FlightMapName+
" (RS_SERIAL1,RS_SERIAL2,RS_QTY1,RS_QTY2,RS_UPDATE,RS_UPTIME,RS_EMPNO,RS_PRINTFLAG,"+
"RS_PART_S,RS_PART_IN_A_P,RS_FILLER)"+" VALUES(?,?,?,?,?,?,?,?,?,?,?)");
//pstmt.setString(1, "12345678901234567890123003");
pstmt.setString(1, "1234567890");
pstmt.setString(2,"1234567890123");
pstmt.setInt(3,00);
pstmt.setInt(4,003);
pstmt.setString(5,"151209");
pstmt.setString(6, "094435");
pstmt.setString(7,"09932");
pstmt.setString(8,"P");
pstmt.setString(9,"Y");
pstmt.setString(10,"Y");
pstmt.setString(11," ");
// Execute the query
int num = pstmt.executeUpdate();
System.out.println(num);
pstmt.close();
}
catch (SQLException t)
{
System.out.println(t.toString());
}
try
{
// Get a statement
java.sql.Statement stmt = jdbcCon.createStatement();
// Execute the query ...
java.sql.ResultSet rs = stmt.executeQuery(
"SELECT * FROM "+vsamCatalog+"\\"+FlightCluster+"\\"+FlightMapName);
while (rs.next())
{
System.out.println(rs.getString("RS_SERIAL1") + " " + rs.getString("RS_SERIAL2")+ " " + rs.getString("RS_UPTIME")+ " " + rs.getString("RS_UPDATE"));
}
rs.close();
stmt.close();
}
catch (SQLException t)
{
}
}
catch (Exception e)
{
// do something appropriate with the exception, *at least*:
e.printStackTrace();
}
}
}
Note: the OS is z/VSE

The short answer to your original question is that KSDS VSAM is not a DBMS.
As you have discovered, you can define the VSAM file such that you can update it both from batch and from CICS, but as #BillWoodger points out, you must serialize your updates yourself.
Another approach would be to do all updates from the CICS region, and have your Java application send a REST or SOAP or MQ message to CICS to request its updates. This does require there be a CICS program to catch the requests from the Java application and perform the updates.

The IBM Mainframe under z/VSE has different partitions that run different jobs. For example partition F7 CICS, partition F8 Batch Jobs, ETC.
When you define a new VSAM file you have to set the SHAREOPTIONS of the file. When I define the file I set the SHAREOPTIONS (2 3). 2 Means that only one partition can write to the file.
So when the batch program (in a different partition to the CICS partition) which is called from Java was trying to write to the file it was not able to write to the file unless I close the file in CICS first.
To fix it I REDEFINE the CICS file with SHAREOPTIONS (4 3). 4 Means that multiple partitions of the Mainframe can write to it. Fixing the problem
Below is a part of the definition code where you set the SHAREOPTION:
* $$ JOB JNM=DEFFI,CLASS=9,DISP=D,PRI=9
* $$ LST CLASS=X,DISP=H,PRI=2,REMOTE=0,USER=JAVI
// JOB DEFFI
// EXEC IDCAMS,SIZE=AUTO
DEFINE CLUSTER -
( -
NAME (FLIGHT.ORDERING.FLIGHTS) -
RECORDS (2000 1000) -
INDEXED -
KEYS (26 0) -
RECORDSIZE (128 128) -
SHAREOPTIONS (4 3) -
VOLUMES (SYSWKE) -
) -
.
.
.

Getting output parameter value set by VBScript (WMI) method in java via JACOB

Am trying to convert a VBScript to java using JACOB - Java COM bridge library.
'Create' method in VBScript accepts a [out] param in it's method and it sets it upon method execution and i couldn't figure out how to retrieve it back via JACOB.
VBScript in question:
Function CreateProcess(strComputer, strCommand)
Dim objWMIService, objProcess
Set objWMIService = GetObject("winmgmts:" & "{impersonationLevel=impersonate}!\\" & strComputer & "\root\cimv2")
Set objProcess = objWMIService.Get("Win32_Process")
errReturn = objProcess.Create (strCommand, Null, Null, intProcessID)
Set objWMIService = Nothing
Set objProcess = Nothing
CreateProcess = intProcessID
End Function
intProcessID is [out] param set after method execution. (Create API contract)
Converted java code(incomplete and modified slightly for demonstration):
public static void createProcess() {
String host = "localhost";
String connectStr = String
.format("winmgmts:{impersonationLevel=impersonate}!\\\\%s\\root\\CIMV2",
host);
ActiveXComponent axWMI = new ActiveXComponent(connectStr);
Variant vCollection = axWMI.invoke("get", new Variant("Win32_Process"));
Dispatch d = vCollection.toDispatch();
Integer processId = null;
int result = Dispatch.call(d, "Create", "notepad.exe", null, null, processId)
.toInt();
System.out.println("Result:" + result);
// WORKS FINE until here i.e. notepad launches properly, however processId still seems to be null. Following commented code is wrong - doesn't work
//Variant v = Dispatch.get(d, "processId"); // even ProcessId doesn't work
//int pId = v.getInt();
//System.out.println("process id:"
// + pId);
// what is the right way to get the process ID set by 'Create' method?
}
Would be great if you could provide some pointers or relevant code. Ask me more if needed. Thanks in advance.

Replacing
Integer processId = null;
with
Variant processId = new Variant(0, true);
should solve the problem. You should then have process ID of the notepad.exe process in the processId variant, and it can be fetched by
processId.getIntRef()

Batch file renaming – inserting text from a list (in Python or Java)

I'm finishing a business card production flow (excel > xml > indesign > single page pdfs) and I would like to insert the employees' names in the filenames.
What I have now:
BusinessCard_01_Blue.pdf
BusinessCard_02_Blue.pdf
BusinessCard_03_Blue.pdf (they are gonna go up to the hundreds)
What I need (I can manipulate the name list with regex easily):
BusinessCard_01_CarlosJorgeSantos_Blue.pdf
BusinessCard_02_TaniaMartins_Blue.pdf
BusinessCard_03_MarciaLima_Blue.pdf
I'm a Java and Python toddler. I've read the related questions, tried this in Automator (Mac) and Name Mangler, but couldn't get it to work.
Thanks in advance,
Gus

Granted you have a map where to look at the right name you could do something like this in Java:
List<Files> originalFiles = ...
for( File f : originalFiles ) {
f.renameTo( new File( getNameFor( f ) ) );
}
And define the getNameFor to something like:
public String getNameFor( File f ) {
Map<String,String> namesMap = ...
return namesMap.get( f.getName() );
}
In the map you'll have the associations:
BusinessCard_01_Blue.pdf => BusinessCard_01_CarlosJorgeSantos_Blue.pdf
Does it make sense?

In Python (tested):
#!/usr/bin/python
import sys, os, shutil, re
try:
pdfpath = sys.argv[1]
except IndexError:
pdfpath = os.curdir
employees = {1:'Bob', 2:'Joe', 3:'Sara'} # emp_id:'name'
files = [f for f in os.listdir(pdfpath) if re.match("BusinessCard_[0-9]+_Blue.pdf", f)]
idnumbers = [int(re.search("[0-9]+", f).group(0)) for f in files]
filenamemap = zip(files, [employees[i] for i in idnumbers])
newfiles = [re.sub('Blue.pdf', e + '_Blue.pdf', f) for f, e in filenamemap]
for old, new in zip(files, newfiles):
shutil.move(os.path.join(pdfpath, old), os.path.join(pdfpath, new))
EDIT: This now alters only those files that have not yet been altered.
Let me know if you want something that will build the the employees dictionary automatically.

If you have a list of names in the same order the files are produced, in Python it goes like this untested fragment:
#!/usr/bin/python
import os
f = open('list.txt', 'r')
for n, name in enumerate(f):
original_name = 'BusinessCard_%02d_Blue.pdf' % (n + 1)
new_name = 'BusinessCard_%02d_%s_Blue.pdf' % (
n, ''.join(name.title().split()))
if os.path.isfile(original_name):
print "Renaming %s to %s" % (original_name, new_name),
os.rename(original_name, new_name)
print "OK!"
else:
print "File %s not found." % original_name

Python:
Assuming you have implemented the naming logic already:
for f in os.listdir(<directory>):
try:
os.rename(f, new_name(f.name))
except OSError:
# fail
You will, of course, need to write a function new_name which takes the string "BusinessCard_01_Blue.pdf" and returns the string "BusinessCard_01_CarlosJorgeSantos_Blue.pdf".