We Keep Coding

find amazon categories using python

I would like to get the categories of the amazon ,I am planning to scrap not to use API.
I have scrapped the http://www.amazon.com.I have scraped all the categories and sub-categories under Shop By Department drop down .I have created a web service to do this The code is here
#route('/hello')
def hello():
text=list();
link=list();
req = urllib2.Request("http://www.amazon.com",
headers={"Content-Type": "application/json"})
html=urllib2.urlopen(req).read()
soup = BeautifulSoup(html)
last_page = soup.find('div', id="nav_subcats")
for elm in last_page.findAll('a'):
texts = elm.text
links = elm.get('href')
links = links.partition("&node=")[2]
text.append(texts)
link.append(links)
alltext=list();
for i,j in zip(text,link):
alltext.append({"name":i,"id":j})
response.content_type = 'application/json'
print(alltext)
return dumps(alltext)
run(host='localhost', port=8080, debug=True)
I am passing the category name and category id as a JSON object to one of my members to pass it to the API to get the product listing for each category
It is written in JAVA.Here is the code
for (int pageno = 1; pageno <= 10; pageno++) {
String page = String.valueOf(pageno);
String category_string = selectedOption.get("category_name").toString();
String category_id = selectedOption.get("category_id").toString();
final Map<String, String> params = new HashMap<String, String>(3);
params.put(AmazonClient.Op.PARAM_OPERATION, "ItemSearch");
params.put("SearchIndex", category_string);
params.put("BrowseNodeId", category_id);
params.put("Keywords", category_string);
params.put("ItemPage", page);
System.out.println(client.documentToString(client.getXml(params)));
DocumentBuilderFactory dbf = DocumentBuilderFactory.newInstance();
Document doc = null;
DocumentBuilder db = dbf.newDocumentBuilder();
InputStream is = client.getInputStream(params);
doc = db.parse(is);
NodeList itemList = doc.getElementsByTagName("Items");
But i am getting this error when i pass the category id as the BrowseNodeId and category name as keyword and search index.
For example
Search Index and Keyword -Amazon Instant Video
BrowseNodeId-2858778011
The value you specified for SearchIndex is invalid. Valid values include [ 'All','Apparel',...................................reless','WirelessAccessories' ].
I would like to know from which amazon url i will get all the categories and its browse nodes
Thank you

I have never looked at Amazon's API before, so this is just a guess but, based on the error message it would seem that "Amazon Instant Video" is not a valid search index. Just because it is there in the drop-down list, doesn't necessarily mean that it is a valid search index.
Here's a list of search indices for US: http://docs.aws.amazon.com/AWSECommerceService/latest/DG/USSearchIndexParamForItemsearch.html . I don't know how up to date it is, but "Amazon Instant Video" does not appear on the list. The error message does include a list of valid search index values, and these do appear to correspond to the above list.
For other locales look here : http://docs.aws.amazon.com/AWSECommerceService/latest/DG/APPNDX_SearchIndexParamForItemsearch.html
I don't think that this is a coding problem per se.
You might like to take a look at python-amazon-product-api. The API might be useful to you, and the documentation might give you some ideas.

Efficient way of parsing XML in Java

I have to parse an XML file with following structure:
<root>
<object_1>
<pro1> abc </pro1>
<pro2> pqr </pro2>
<pro3> xyz </pro3>
<children>
<object_a>
<pro1> abc </pro1>
<pro2> pqr </pro2>
<pro3> xyz </pro3>
<children>
.
.
.
</children>
</object_a>
</children>
</object_1>
<object_2>
.
.
.
</object_n>
</root>
Aim is to parse this multilevel nesting. A few classes are defined in Java.
Class Object_1
Class Object_2
.
.
.
Class Object_N
with their respective properties.
The following code is working for me, but then this is not the best way of doing things.
File file = new File(fileName);
DocumentBuilderFactory dbFactory = DocumentBuilderFactory.newInstance();
DocumentBuilder dBuilder = dbFactory.newDocumentBuilder();
Document doc = dBuilder.parse(file);
doc.getDocumentElement().normalize();
if(doc ==null) return;
Node node = doc.getFirstChild();
NodeList lst = node.getChildNodes();
Node children = null ;
int len = lst.getLength();
for(int index=0;index<len;index++)
{
Node child = lst.item(index);
String name = child.getNodeName();
if(name=="Name")
name = child.getNodeValue();
else if(name=="Comment")
comment = child.getNodeValue());
else if(name=="children")
children = child;
}
if(children==null) return;
lst = children.getChildNodes();
len = lst.getLength();
Class<?> obj=null;
AbsModel model = null;
for(int index=0;index<len;index++)
{
Node childNode = lst.item(index);
String modelName = childNode.getNodeName();
try {
obj = Class.forName(modelName);
} catch (ClassNotFoundException e) {
e.printStackTrace();
}
if(obj!=null)
model = (AbsModel) obj.newInstance();
else
model = new GenericModel();
model.restoreDefaultPropFromXML(childNode);
addChild(model);
}
}
Is there a better way of parsing this XML.

Consider using JAXB, which is part of Java since version 6. You should be able to parse (“unmarshall”) your XML file into your own classes with almost no code, just adding a few annotations expliciting the mapping between your object structure and your XML structure.

StAX and or JAXB is almost always the way to go.
If the XML is really dynamic (like attributes specify the property name) ie <prop name="property" value="" /> then you will need to use StAX only or live with what JAXB will map it to (a POJO with name and value properties) and post process.
Personally I find combining StAX and JAXB the best. I parse to the elements I want and then use JAXB to turn the element into a POJO.
See Also:
My own utility library that will turn an XML Stream into an iterator of objects.
Parsing very large XML files and marshalling to Java Objects
http://tedone.typepad.com/blog/2011/06/unmarshalling-benchmark-in-java-jaxb-vs-stax-vs-woodstox.html

While JAXB may be the best choice I'd also like to mention jOOX which provides a JQuery-like API and makes working with XML documents really pleasant.

ready made parser in java

i have some user defined tag. for example data here , jssj .I have a file(not xml) which contains some data embeded in tags.I need a parser for this which will identify my tags and will extract the data in proper format.
Eg
<newpage> thix text </newpage>
<tagD>
<tagA> kk</tagA>
</tagD>
tags can also have some attributes as simlar to html tags. Eg
<mytag height="f" width ="d" > bla bla bla </mytag>
<mytag attribute="val"> bla bla bla</mytag>

You could look at a parser generator like antlr.
Unless your tag syntax can be represented with a (simple) regular grammar (in which case you could try to scan the file with regexes), you will need a proper parser. It is actually not very hard to do at all - just the first time tastes like biting bullets...

You can use JAXB, already included in Java. It's quite simple.
First you need to create a binding to your XML code. The binding provides a map between Java objects and the XML code.
An example would be:
#XmlRootElement(name = "YourRootElement", namespace ="http://someurl.org")
#XmlAccessorType(XmlAccessType.FIELD)
#XmlType(name = "", propOrder = {
"intValue",
"stringArray",
"stringValue"}
)
public class YourBindingClass {
protected int intValue;
#XmlElement(nillable = false)
protected List<String> stringArray;
#XmlElement(name = "stringValue", required = true)
protected String stringValue;
public int getIntValue() {
return intValue;
}
public void setIntValue(int value) {
this.intValue = value;
}
public List<String> getStringArray() {
if (stringArray == null) {
stringArray = new ArrayList<String>();
}
return this.stringArray;
}
public String getStringValue() {
return stringValue;
}
public void setStringValue(String value) {
this.stringValue = value;
}
}
Then, to encode your Java objects into XML, you can use:
YourBindingClass yourBindingClass = ...;
JAXBContext jaxbContext = JAXBContext.newInstance(YourBindingClass.class);
Marshaller marshaller = jaxbContext.createMarshaller();
marshaller.setProperty(Marshaller.JAXB_FORMATTED_OUTPUT, true);
marshaller.setProperty(Marshaller.JAXB_FRAGMENT, false);
/** If you need to specify a schema */
SchemaFactory sf = SchemaFactory.newInstance(XMLConstants.W3C_XML_SCHEMA_NS_URI);
Schema schema = sf.newSchema(new URL("http:\\www.someurl.org"));
marshaller.setSchema(schema);
marshaller.setProperty(Marshaller.JAXB_SCHEMA_LOCATION, true);
ByteArrayOutputStream stream = new ByteArrayOutputStream();
marshaller.marshal(yourBindingClass, stream);
System.out.println(stream);
To parse your XML back to objects:
InputStream resourceAsStream = ... // Your XML, File, etc.
JAXBContext jaxbContext = JAXBContext.newInstance(YourBindingClass.class);
Unmarshaller unmarshaller = jaxbContext.createUnmarshaller();
Object r = unmarshaller.unmarshal(resourceAsStream);
if (r instanceof YourBindingClass) ...
Example starting from a Java object:
YourBindingClass s = new YourBindingClass();
s.setIntValue(1);
s.setStringValue("a");
s.getStringArray().add("b1");
s.getStringArray().add("b2");
// marshal ...
Result:
<?xml version="1.0" encoding="UTF-8" standalone="yes"?>
<ns2:YourRootElement xmlns:ns2="http://someurl.org">
<intValue>1</intValue>
<stringArray>b1</stringArray>
<stringArray>b2</stringArray>
<stringValue>a</stringValue>
</ns2:YourRootElement>
If you don't know the input format, that means you probably don't have a XML schema. If you don't have a schema you don't have some it's benefits such as:
It is easier to describe allowable document content
It is easier to validate the correctness of data
It is easier to define data facets (restrictions on data)
It is easier to define data patterns (data formats)
It is easier to convert data between different data types
Anyway, the previous code also works with XML code that contains 'unknown' tags. However your XML code still have to present the required fields and follow the declared patterns.
So the following XML code is also valid. The only restriction is: the tag 'stringValue' should be there. Note that 'stringArrayQ' was not previously declared.
<?xml version="1.0" encoding="UTF-8" standalone="yes"?>
<ns2:YourRootElement xmlns:ns2="http://someurl.org">
<stringValue>a</stringValue>
<stringArrayQ>b1</stringArrayQ>
</ns2:YourRootElement>

Are these XML tags? If so, look into one of the many Java XML libraries already available. If they're some kind of custom tagging format, then you're just going to have to write it yourself.

For xml tags - use DOM parser or SAX parser.

You example is XML with this modification:
<root>
<newpage> thix text </newpage>
<tagD>
<tagA> kk</tagA>
</tagD>
</root>
You can use any XML parser you want to parse it.
Edit:
Attributes are a normal part of XML.
<root>
<newpage> thix text </newpage>
<tagD>
<tagA> kk</tagA>
</tagD>
<mytag height="f" width ="d" > bla bla bla </mytag>
<mytag attribute="val"> bla bla bla</mytag>
</root>
Every XML parser can deal with them.
Edit:
If you were able to use Python, you could do something like this:
import lxml.etree
doc = lxml.etree.parse("foo.xml")
print doc.xpath("//mytag[1]/#width")
# => ['d']
That's what i call simple.

NamespaceContext and using namespaces with XPath

Resolving an xpath that includes namespaces in Java appears to require the use of a NamespaceContext object, mapping prefixes to namespace urls and vice versa. However, I can find no mechanism for getting a NamespaceContext other than implementing it myself. This seems counter-intuitive.
The question: Is there any easy way to acquire a NamespaceContext from a document, or to create one, or failing that, to forgo prefixes altogether and specify the xpath with fully qualified names?

It is possible to get a NamespaceContext instance without writing your own class. Its class-use page shows you can get one using the javax.xml.stream package.
String ctxtTemplate = "<data xmlns=\"http://base\" xmlns:foo=\"http://foo\" />";
NamespaceContext nsContext = null;
XMLInputFactory factory = XMLInputFactory.newInstance();
XMLEventReader evtReader = factory
.createXMLEventReader(new StringReader(ctxtTemplate));
while (evtReader.hasNext()) {
XMLEvent event = evtReader.nextEvent();
if (event.isStartElement()) {
nsContext = ((StartElement) event)
.getNamespaceContext();
break;
}
}
System.out.println(nsContext.getNamespaceURI(""));
System.out.println(nsContext.getNamespaceURI("foo"));
System.out.println(nsContext
.getNamespaceURI(XMLConstants.XMLNS_ATTRIBUTE));
System.out.println(nsContext
.getNamespaceURI(XMLConstants.XML_NS_PREFIX));
Forgoing prefixes altogether is likely to lead to ambiguous expressions - if you want to drop namespace prefixes, you'd need to change the document format. Creating a context from a document doesn't necessarily make sense. The prefixes have to match the ones used in the XPath expression, not the ones in any document, as in this code:
String xml = "<data xmlns=\"http://base\" xmlns:foo=\"http://foo\" >"
+ "<foo:value>"
+ "hello"
+ "</foo:value>"
+ "</data>";
String expression = "/stack:data/overflow:value";
class BaseFooContext implements NamespaceContext {
#Override
public String getNamespaceURI(String prefix) {
if ("stack".equals(prefix))
return "http://base";
if ("overflow".equals(prefix))
return "http://foo";
throw new IllegalArgumentException(prefix);
}
#Override
public String getPrefix(String namespaceURI) {
throw new UnsupportedOperationException();
}
#Override
public Iterator<String> getPrefixes(
String namespaceURI) {
throw new UnsupportedOperationException();
}
}
XPathFactory factory = XPathFactory.newInstance();
XPath xpath = factory.newXPath();
xpath.setNamespaceContext(new BaseFooContext());
String value = xpath.evaluate(expression,
new InputSource(new StringReader(xml)));
System.out.println(value);
Neither the implementation returned by the StAX API nor the one above implement the full class/method contracts as defined in the doc. You can get a full, map-based implementation here.

I've just been working through using xpath and NamespaceContexts myself. I came across a good treatment of the issue on developerworks.

I found a convenient implementation in "Apache WebServices Common Utilities" called NamespaceContextImpl.
You can use the following maven dependency to obtain this class:
<dependency>
<groupId>org.apache.ws.commons</groupId>
<artifactId>ws-commons-util</artifactId>
<version>1.0.1</version>
</dependency>
I've use it in the following manner (I know its built for sax, but after reading the code, its o.k):
NamespaceContextImpl nsContext = new NamespaceContextImpl();
nsContext.startPrefixMapping("foo", "my.name.space.com");
You don't need to called endPrefixMapping.

If you are using the Spring framework you can reuse their NamespaceContext implementation
org.springframework.util.xml.SimpleNamespaceContext
This is a similar answer like the one from Asaf Mesika. So it doesn't give you automatic a NamespaceContext based on your document. You have to construct it yourself. Still it helps you because it at least gives you an implementation to starts with.
When we faced a similar problem, Both the spring SimpleNamespaceContext and the "Apache WebServices Common Utilities" worked. We wanted to avoid to the addition jar dependency on Apache WebServices Common Utilities and used the Spring one, because our application is Spring based.

If you are using Jersey 2 and only have a default XML namespace (xmlns="..."), you can use SimpleNamespaceResolver:
<?xml version="1.0" encoding="UTF-8"?>
<Outer xmlns="http://host/namespace">
<Inner />
</Outer>
DocumentBuilderFactory docBuilderFactory = DocumentBuilderFactory.newInstance();
docBuilderFactory.setNamespaceAware(true);
DocumentBuilder docBuilder = docBuilderFactory.newDocumentBuilder();
Document document = docBuilder.parse(new File("document.xml"));
String query = "/t:Outer/t:Inner";
XPath xpath = XPathFactory.newInstance().newXPath();
String xmlns = document.getDocumentElement().getAttribute("xmlns");
xpath.setNamespaceContext(new SimpleNamespaceResolver("t", xmlns));
NodeList nodeList = (NodeList) xpath.evaluate(query, document, XPathConstants.NODESET);
//nodeList will contain the <Inner> element
You can also specify xmlns manually if you want.

How to insert/replace XML tag in XmlDocument?

I have a XmlDocument in java, created with the Weblogic XmlDocument parser.
I want to replace the content of a tag in this XMLDocument with my own data, or insert the tag if it isn't there.
<customdata>
<tag1 />
<tag2>mfkdslmlfkm</tag2>
<location />
<tag3 />
</customdata>
For example I want to insert a URL in the location tag:
<location>http://something</location>
but otherwise leave the XML as is.
Currently I use a XMLCursor:
XmlObject xmlobj = XmlObject.Factory.parse(a.getCustomData(), options);
XmlCursor xmlcur = xmlobj.newCursor();
while (xmlcur.hasNextToken()) {
boolean found = false;
if (xmlcur.isStart() && "schema-location".equals(xmlcur.getName().toString())) {
xmlcur.setTextValue("http://replaced");
System.out.println("replaced");
found = true;
} else if (xmlcur.isStart() && "customdata".equals(xmlcur.getName().toString())) {
xmlcur.push();
} else if (xmlcur.isEnddoc()) {
if (!found) {
xmlcur.pop();
xmlcur.toEndToken();
xmlcur.insertElementWithText("schema-location", "http://inserted");
System.out.println("inserted");
}
}
xmlcur.toNextToken();
}
I tried to find a "quick" xquery way to do this since the XmlDocument has an execQuery method, but didn't find it very easy.
Do anyone have a better way than this? It seems a bit elaborate.

How about an XPath based approach? I like this approach as the logic is super-easy to understand. The code is pretty much self-documenting.
If your xml document is available to you as an org.w3c.dom.Document object (as most parsers return), then you could do something like the following:
// get the list of customdata nodes
NodeList customDataNodeSet = findNodes(document, "//customdata" );
for (int i=0 ; i < customDataNodeSet.getLength() ; i++) {
Node customDataNode = customDataNodeSet.item( i );
// get the location nodes (if any) within this one customdata node
NodeList locationNodeSet = findNodes(customDataNode, "location" );
if (locationNodeSet.getLength() > 0) {
// replace
locationNodeSet.item( 0 ).setTextContent( "http://stackoverflow.com/" );
}
else {
// insert
Element newLocationNode = document.createElement( "location" );
newLocationNode.setTextContent("http://stackoverflow.com/" );
customDataNode.appendChild( newLocationNode );
}
}
And here's the helper method findNodes that does the XPath search.
private NodeList findNodes( Object obj, String xPathString )
throws XPathExpressionException {
XPath xPath = XPathFactory.newInstance().newXPath();
XPathExpression expression = xPath.compile( xPathString );
return (NodeList) expression.evaluate( obj, XPathConstants.NODESET );
}

How about an object oriented approach? You could deserialise the XML to an object, set the location value on the object, then serialise back to XML.
XStream makes this really easy.
For example, you would define the main object, which in your case is CustomData (I'm using public fields to keep the example simple):
public class CustomData {
public String tag1;
public String tag2;
public String location;
public String tag3;
}
Then you initialize XStream:
XStream xstream = new XStream();
// if you need to output the main tag in lowercase, use the following line
xstream.alias("customdata", CustomData.class);
Now you can construct an object from XML, set the location field on the object and regenerate the XML:
CustomData d = (CustomData)xstream.fromXML(xml);
d.location = "http://stackoverflow.com";
xml = xstream.toXML(d);
How does that sound?

If you don't know the schema the XStream solution probably isn't the way to go. At least XStream is on your radar now, might come in handy in the future!

You should be able to do this with query
try
fn:replace(string,pattern,replace)
I am new to xquery myself and I have found it to be a painful query language to work with, but it does work quiet well once you get over the initial learning curve.
I do still wish there was an easier way which was as efficient?