We Keep Coding

Best way to get Amazon page and product information

I want to get Amazon page and product information from their website so I work on a future project. I have no experience with APIs but also saw that I would need to pay in order to use Amazon's. My current plan was to use a WebRequest class which basically takes down the page's raw text and then parse through it to get what I need. It pulls down HTML from all the websites I have tried except amazon. When I try and use it for amazon I get text like this...
??èv~-1?½d!Yä90û?¡òk6??ªó?l}L??A?{í??j?ì??ñF Oü?ª[D ú7W¢!?É?L?]â  v??ÇJ???t?ñ?j?^,Y£>O?|?I`OöN??Q?»bÇJPy1·¬Ç??RtâU??Q%vB??^íè|??ª?
Can someone explain to me why this happens? Or even better if you could point me towards a better way of doing this? Any help is appreciated.
This is the class I mentioned...
public class WebRequest {
protected String url;
protected ArrayList<String> pageText;
public WebRequest() {
url = "";
pageText = new ArrayList<String>();
}
public WebRequest(String url) {
this.url = url;
pageText = new ArrayList<String>();
load();
}
public boolean load() {
boolean returnValue = true;
try {
URL thisURL = new URL(url);
BufferedReader reader = new BufferedReader(new InputStreamReader(thisURL.openStream()));
String line;
while ((line = reader.readLine()) != null) {
pageText.add(line);
}
reader.close();
}
catch (Exception e) {
returnValue = false;
System.out.println("peepee");
}
return returnValue;
}
public boolean load(String url) {
this.url = url;
return load();
}
public String toString() {
String returnString = "";
for (String s : pageText) {
returnString += s + "\n";
}
return returnString;
}
}

It could be that the page is returned using a different character encoding than your platform default. If that's the case, you should specify the appropriate encoding, e.g:
new InputStreamReader(thisURL.openStream(), "UTF-8")
But that data doesn't look like character data at all to me. It's too random. It looks like binary data. Are you sure you're not downloading an image by mistake?
If you want to make more sophisticated HTTP requests, there are quite a few Java libraries, e.g. OkHttp and AsyncHttpClient.
But it's worth bearing in mind that Amazon probably doesn't like people scraping its site, and will have built in detection of malicious or unwanted activity. It might be sending you gibberish on purpose to deter you from continuing. You should be careful because some big sites may block your IP temporarily or permanently.
My advice would be to learn how to use the Amazon APIs. They're pretty powerful—and you won't get yourself banned.

Google OAuth 2.0 Access Token Request Error

I have a small Google App Engine project on GitHub what I "finished" several months ago.
It worked until now (Last use was 2-3 months ago), but it seems the Google changed something and I don't find what the real problem is and how I can solve it.
The problem is in this file (line 59):
File
Code:
public class TokenManager {
static private final String TOKEN_URL = "https://www.googleapis.com/oauth2/v3/token";
static private final String ENC = "UTF-8";
private String _refreshToken;
private String _clientId;
private String _clientSecret;
private final Gson _gson = new Gson();
private TokenData _tokenData = new TokenData();
...
private void refreshAccessToken() throws Exception {
String payload = "client_secret=$client_secret&grant_type=refresh_token&refresh_token=$refresh_token&client_id=$client_id";
payload = payload.replace("$client_secret", URLEncoder.encode(_clientSecret, ENC));
payload = payload.replace("$refresh_token", URLEncoder.encode(_refreshToken, ENC));
payload = payload.replace("$client_id", URLEncoder.encode(_clientId, ENC));
HTTPRequest request = new HTTPRequest(new URL(TOKEN_URL), HTTPMethod.POST);
request.setPayload(payload.getBytes(ENC));
String stringData = new String(URLFetchServiceFactory.getURLFetchService().fetch(request).getContent(), ENC);
_tokenData = _gson.fromJson(stringData, TokenData.class);
}
}
The problem is around the access token request. (I use offline refresh token.)
The response is this:
{
"error": "invalid_grant",
"error_description": "Bad Request",
"error_uri": ""
}
I have tried to modify the TOKEN_URL to "https://www.googleapis.com/oauth2/v4/token"
I have tried to generate new refresh token.
I have tried create new OAuth Client ID.
All attempts had same problem.
Have anyone any idea what is wrong?
Thanks for help!

I found the problem.
Probably my previous refresh token was expired. I tried to generating
a new but I misunderstood the process and I considered "code"
parameter as refresh token.
https://developers.google.com/identity/protocols/OAuth2WebServer#offline

how to send a Photo with caption in telegram using ex3ndr?

I am using ex3ndr for creating a telegram client. now i want to send a message witch has a photo and a caption or description. I send photo using this code snippet:
private static void sendMedia(PeerState peerState, String fileName) {
TLAbsInputPeer inputPeer = peerState.isUser() ? new TLInputPeerContact(peerState.getId()) : new TLInputPeerChat(peerState.getId());
int task = api.getUploader().requestTask(fileName, null);
api.getUploader().waitForTask(task);
int resultState = api.getUploader().getTaskState(task);
Uploader.UploadResult result = api.getUploader().getUploadResult(task);
TLAbsInputFile inputFile;
if (result.isUsedBigFile()) {
inputFile = new TLInputFileBig(result.getFileId(), result.getPartsCount(), "file.jpg");
} else {
inputFile = new TLInputFile(result.getFileId(), result.getPartsCount(), "file.jpg", result.getHash());
}
try {
TLAbsStatedMessage res = api.doRpcCall(new TLRequestMessagesSendMedia(inputPeer, new TLInputMediaUploadedPhoto(inputFile), rnd.nextInt()), 30000);
res.toString();
} catch (IOException e) {
e.printStackTrace();
}
}
but I donot know how can add caption to this photo?(this code snippet is a sample from this url: ex3ndr sample
)

ex3ndr library only support layer 12 of Telegram API where sendMedia method doesn't support captions in photos. That's means this library is not able to send captions with photos, the layer should be updated before being able of doing so (and the repository seems to be abandoned).

PayPal REST API java sdk - custom config file

Good afternoon all!
I use PayPal REST API java sdk and I want to have different configurations for different environments of my application. Here is how I'm trying to do so:
private static boolean IS_PRODUCTION = false;
private static String PAYPAL_ACCESS_TOKEN;
private static void initPayPal() {
InputStream is = null;
try {
is = ApplicationConfig.class.getResourceAsStream(
IS_PRODUCTION? "/my_paypal_sdk_config.properties" : "/my_paypal_sdk_config_test.properties");
PayPalResource.initConfig(is);
String clientID = ConfigManager.getInstance().getConfigurationMap().get("clientID");
String clientSecret = ConfigManager.getInstance().getConfigurationMap().get("clientSecret");
PAYPAL_ACCESS_TOKEN = new OAuthTokenCredential(clientID, clientSecret).getAccessToken();
} catch (Exception e) {
throw new RuntimeException(e);
} finally {
IOUtils.closeQuietly(is);
}
}
and while trying to get the clientID I have
java.io.IOException: Resource 'sdk_config.properties' could not be found
Strange behavior - I thought I've just configured the sdk to use my own properties file.
Please advice how could I set up those settings properly!

So here is the solution I found:
Create an empty sdk_config.properties file in default location
Load your own properties:
private static void initPayPal() {
InputStream is = null;
try {
is = ApplicationConfig.class.getResourceAsStream(
IS_PRODUCTION ? "/my_paypal_sdk_config.properties" : "/my_paypal_sdk_config_test.properties");
Properties props = new Properties();
props.load(is);
PayPalResource.initConfig(props);
ConfigManager.getInstance().load(props);
String clientID = ConfigManager.getInstance().getConfigurationMap().get("clientID");
String clientSecret = ConfigManager.getInstance().getConfigurationMap().get("clientSecret");
PAYPAL_ACCESS_TOKEN = new OAuthTokenCredential(clientID, clientSecret).getAccessToken();
} catch (Exception e) {
throw new RuntimeException(e);
} finally {
IOUtils.closeQuietly(is);
}
}

We have made some good improvements to the PayPal Java SDK on integration steps. We are removing the need for sdk_config.properties file as they do not work as well, specially for multi-configuration settings.
Now, all you do is create an APIContext instance with clientId, clientSecret, and mode. You pass that context object for any API operation from there on.
Here is how the code would look like for different configurations:
APIContext defaultContext = new APIContext(clientId1, clientSecret1, "sandbox");
APIContext sandboxContext = new APIContext(clientId2, clientSecret2, "sandbox");
APIContext someOtherContext = new APIContext(clientId3, clientSecret3, "live");
APIContext liveContext = new APIContext(clientId, clientSecret, "live");
// Now pass any of the above context in these calls, and it would use those configurations.
Payment payment = new Payment();
// Fill in all the details.
payment.create(defaultContext);
// Replace that defaultContext with any of those other contexts.
Here is the wiki page explaining that: https://github.com/paypal/PayPal-Java-SDK/wiki/Making-First-Call

I had the same error with SDK 0.11 version. I use my own properties file, but code still looked for "sdk_config.properties". I put it into root in my CLASSPATH, but still got the same error. Then I made obvious and horrible solution: put empty "sdk_config.properties" into "rest-api-sdk-0.11.0.jar" library. This street magic solved my problem.

generate oauth signature for pentaho client in js or java for twitter

I would like to be able to make a request to Twitter with Pentaho's REST client request however this software does not have any concept of oauth. I found this (Implement OAuth in Java)neat java class that I would like to implement with Pentaho's java class tranformation but I am so new to Pentaho this task will be very difficult. I am hoping someone can help me out with this....

I found this great twitter java library called twitter4J and imported the core classes into the pentaho directory pentaho/design-tools/data-integration/libext and wrote the below custom user java class.
// NO COLLECTION TYPE SAFETY ALLOWED, MUST CAST ALL OBJECTS
import twitter4j.*;
import twitter4j.auth.*;
import twitter4j.conf.*;
//import other libs here
//put your vars here
// Variables
private Twitter twitter = null;
public boolean processRow(StepMetaInterface smi, StepDataInterface sdi) throws KettleException
{
Object[] r = getRow();
if (r==null)
{
setOutputDone();
return false;
}
if (first) {
first=false;
paging = new Paging();
paging.setCount(100);
}
oauth_user_key = get(Fields.In, "oauth_user_key").getString(r);
oauth_user_secret = get(Fields.In, "oauth_user_secret").getString(r);
consumer_key = get(Fields.In, "consumer_key").getString(r);
consumer_secret = get(Fields.In, "consumer_secret").getString(r);
//wierd long/string thing here (pentho compiles java wierd)
user_id = get(Fields.In, "source_user_id").getInteger(r);
Long user = user_id;
ConfigurationBuilder cb = new ConfigurationBuilder();
cb.setDebugEnabled(true)
.setIncludeEntitiesEnabled(true)
.setOAuthConsumerKey(consumer_key)
.setOAuthConsumerSecret(consumer_secret)
.setOAuthAccessToken(oauth_user_key)
.setOAuthAccessTokenSecret(oauth_user_secret);
twitter = new TwitterFactory(cb.build()).getInstance();
try {
//be creative with twitter4j here and output rows with results (may require a loop)
} catch (TwitterException e){
logDebug(e.getMessage());
return true;
}
logBasic("twitter collect done" );
return true;
}