We Keep Coding

Is it possible to download the first 100 lines of a big file with millions of lines from S3?

I have multiple 100MB raw files with series of user activities in CSV format. I only want to download the first 100 lines of the files.
The problem is that each file may have different CSV header columns, and data values, because they are user activities from multiple subdomains using different activity tracking providers. This means that each line can be 50 characters long or 500 characters long and it is unknown until I read them all.
S3 supports getObject API with Range parameter which you can use to download the specific ranges of XX bytes of the file.
https://docs.aws.amazon.com/AmazonS3/latest/API/API_GetObject.html#API_GetObject_RequestSyntax
If I use this API to parse first 1Mb of files, iterate each byte until I see 100 new lines character \n, would that technically work? Is there something that I have to be careful about this approach? (e.g. multibyte chars?)

There is no built-in way, byte-range fetches are the best way forward.
As you're not sure of the header or line length in each case, downloading 1MB chunks until you have 100 line is a safe & efficient approach.
Multibyte chars etc. won't be important as at this level, you're purely looking to stop reading after 100 \n characters. Depending on the source of your files, however, I would be also conscious of \r\n and \r as being valid line endings.
I've written the below Java code for getting the last n bytes, feel free to use it as a starting point for getting the first n bytes:
public String getLastBytesOfObjectAsString(String bucket, String key, long lastBytesCount) {
try {
final ObjectMetadata objectMetadata = client.getObjectMetadata(bucket, key);
final long fileSizeInBytes = objectMetadata.getContentLength();
long rangeStart = fileSizeInBytes - lastBytesCount;
if (rangeStart < 0) {
rangeStart = 0;
}
final GetObjectRequest getObjectRequest =
new GetObjectRequest(bucket, key).withRange(rangeStart);
try (S3Object s3Object = client.getObject(getObjectRequest);
InputStream inputStream = s3Object.getObjectContent()) {
return new String(inputStream.readAllBytes());
}
} catch (Exception ex) {
...
}
}

You can use smart_open like this:
from smart_open import open
with open('s3://bucket/path/file.csv', 'r') as f:
csv_reader = csv.DictReader(f, delimiter=',')
data = ''
for i, row in enumerate(csv_reader):
data += row +'\n'
if i > 100:
store(data)
You will need to open another file in your localmachine with write permission to store a 100 lines or as many as you like. If you want the 1st lines from multiple files, you can do the same but using the boto3 function for listing the files and send the path/file name to a function using smart_open.
s3client = boto3.client('s3')
listObj = s3client.list_objects_v2(Bucket=bucket, Prefix=prefix)
for obj in listObj['Contents']:
smart_function(obj['Key'])
The obj['Key'] contains the path and file name of each file in that Bucket+Path(Prefix)

Espresso image picker - CursorIndexOutOfBoundsException error

I'm developing a test with Espresso to test the profile image change function. I added the following lines in the #Before method in the test.
I create an intent with the image Uri, with my file provider, to return ever that my app goes to the gallery to pick an image.
Intent resultData = new Intent();
String filename = "img1.jpg";
String path = "mnt/sdcard/" + filename;
File f = new File(path);
Context context =InstrumentationRegistry.getInstrumentation().getContext();
Uri contentUri = getUriForFile(context, "com.otsuka.ikigai.fileprovider", f);
resultData.setData(contentUri);
Instrumentation.ActivityResult result = new Instrumentation.ActivityResult(Activity.RESULT_OK,resultData);
intending(not(isInternal())).respondWith(result);
The code of the activity that changes user image, calls the following method when it receives the intent,(I must not change it).
mProfileImage = CommonBitmapUtils.rotate(this, data.getData());
profileEdited = true;
imgUserPhoto.setImageBitmap(mProfileImage);
And I'm getting the following error:
android.database.CursorIndexOutOfBoundsException: Index 0 requested, with a size of 0
Caused by this line in the function rotate of the CommonBitmapUtils class:
path = cursor.getString(cursor.getColumnIndex(MediaStore.Images.Media.DATA));
The cursor has 0 rows don't know why.

Solved it by setting the following path. Without file providers and permissions.
I got the path by debugging the app without testing, and copying the value.
resultData.setData(Uri.parse("content://media/external/images/media/142583"));

How to detect path the file(whether playing from internal or URL stream) being played in Android?

I have a programmed a custom exoplayer videoview to play videos from both internal memory and URL's. Now i want to check whether the video that is played presently in Videoview is from URL Stream or Internal storage. How can i check this?

You can use two methods to determine whether it's internal or remote.
Method 1 - Using Uri
Parse the given URI using Uri.parse(...) then use getScheme() to get the scheme. For (internal) files it will be file.
For example:
String file = "file:///path/of/your/file";
String remote = "https://www.example.com/your_video.mp4";
Uri fileUri = Uri.parse(file);
Uri remoteUri = Uri.parse(remote);
String fileScheme = fileUri.getScheme();
String remoteScheme = remoteUri.getScheme();
Log.d("Scheme", fileScheme); // prints file
Log.d("Scheme", remoteScheme); // prints https
Method 2 - Using File
Create a new File instance from the URI and check if it exists.
For example:
String file = "file:///path/of/your/file";
String remote = "https://www.example.com/your_video.mp4";
File fileFile = new File(file);
File remoteFile = new File(remote);
Log.d("File", "" + fileFile.exists()); // prints true if the file exists
Log.d("File", "" + remoteFile.exists()); // prints false always

convert parse.com file uri to string Android

i have a file that i got back from parse.com and i am trying to get the string stored in that file to a textView. At the moment all i am getting is a url in my text view and not the string i need.
getting the parse file:
ParseFile file = message.getParseFile("file");
Uri fileUri = Uri.parse(file.getUrl());
//passing it to another class
intent.setData(fileUri);
getting the file to display in textview
mDisplay = (TextView)findViewById(R.id.messageDisplay);
Uri textUri = getIntent().getData();
// i have tried this also [ File string = new File(textUri.toString()); ]
String filePath = textUri.getEncodedPath();
mDisplay.setText(filePath);
i have tried:-
1] to get the string bytes before passing string via intent
2] getting url and then converting to string
and 1 or 2 more methods. all are displaying the url and not the string, i feel the issue may lie in this line
Uri fileUri = Uri.parse(file.getUrl());
but i am not sure.

Android: how to parse URL String with spaces to URI object?

I have a string representing an URL containing spaces and want to convert it to an URI object. If I simply try to create it via
String myString = "http://myhost.com/media/File Name that has spaces inside.mp3";
URI myUri = new URI(myString);
it gives me
java.net.URISyntaxException: Illegal character in path at index X
where index X is the position of the first space in the URL string.
How can i parse myString into a URI object?

You should in fact URI-encode the "invalid" characters. Since the string actually contains the complete URL, it's hard to properly URI-encode it. You don't know which slashes / should be taken into account and which not. You cannot predict that on a raw String beforehand. The problem really needs to be solved at a higher level. Where does that String come from? Is it hardcoded? Then just change it yourself accordingly. Does it come in as user input? Validate it and show error, let the user solve itself.
At any way, if you can ensure that it are only the spaces in URLs which makes it invalid, then you can also just do a string-by-string replace with %20:
URI uri = new URI(string.replace(" ", "%20"));
Or if you can ensure that it's only the part after the last slash which needs to be URI-encoded, then you can also just do so with help of android.net.Uri utility class:
int pos = string.lastIndexOf('/') + 1;
URI uri = new URI(string.substring(0, pos) + Uri.encode(string.substring(pos)));
Do note that URLEncoder is insuitable for the task as it's designed to encode query string parameter names/values as per application/x-www-form-urlencoded rules (as used in HTML forms). See also Java URL encoding of query string parameters.

java.net.URLEncoder.encode(finalPartOfString, "utf-8");
This will URL-encode the string.
finalPartOfString is the part after the last slash - in your case, the name of the song, as it seems.

To handle spaces, #, and other unsafe characters in arbitrary locations in the url path, Use Uri.Builder in combination with a local instance of URL as I have described here:
private Uri.Builder builder;
public Uri getUriFromUrl(String thisUrl) {
URL url = new URL(thisUrl);
builder = new Uri.Builder()
.scheme(url.getProtocol())
.authority(url.getAuthority())
.appendPath(url.getPath());
return builder.build();
}

URL url = Test.class.getResource(args[0]); // reading demo file path from
// same location where class
File input=null;
try {
input = new File(url.toURI());
} catch (URISyntaxException e1) {
// TODO Auto-generated catch block
e1.printStackTrace();
}

I wrote this function:
public static String encode(#NonNull String uriString) {
if (TextUtils.isEmpty(uriString)) {
Assert.fail("Uri string cannot be empty!");
return uriString;
}
// getQueryParameterNames is not exist then cannot iterate on queries
if (Build.VERSION.SDK_INT < 11) {
return uriString;
}
// Check if uri has valid characters
// See https://tools.ietf.org/html/rfc3986
Pattern allowedUrlCharacters = Pattern.compile("([A-Za-z0-9_.~:/?\\#\\[\\]#!$&'()*+,;" +
"=-]|%[0-9a-fA-F]{2})+");
Matcher matcher = allowedUrlCharacters.matcher(uriString);
String validUri = null;
if (matcher.find()) {
validUri = matcher.group();
}
if (TextUtils.isEmpty(validUri) || uriString.length() == validUri.length()) {
return uriString;
}
// The uriString is not encoded. Then recreate the uri and encode it this time
Uri uri = Uri.parse(uriString);
Uri.Builder uriBuilder = new Uri.Builder()
.scheme(uri.getScheme())
.authority(uri.getAuthority());
for (String path : uri.getPathSegments()) {
uriBuilder.appendPath(path);
}
for (String key : uri.getQueryParameterNames()) {
uriBuilder.appendQueryParameter(key, uri.getQueryParameter(key));
}
String correctUrl = uriBuilder.build().toString();
return correctUrl;
}