I am designing an API where one of the methods needs to provide a way to get file content.
I do not want the method to return a file location as potentially files will be stored in the database. I also do not want to return file content as Array[Byte] as the files can be quite big.
In Java I would create a method that returns an InputStream.
What is the recommended way of doing it in Scala?
Scala doesn't really have much in the way of I/O. You could return a Scala.io.Source, which is really a wrapper around InputStream (as pretty much anything will have to be, since that's all any JVM program has access to).
There's an interesting Scala I/O library, not a standard library, based on Java 7 stuff, iirc. And there's Scalaz Stream, which is the best thing out there, though it still uses an InputStream under the covers if you are handling files.
The recommended way of doing it in Scala is by using InputStream instead of reinventing the wheel.
Related
I need to parse a large (>800MB) XML file from Jython. The XML is not deeply nested, containing about a million relevant elements. I need to convert these elements into real objects.
I've used nu.xom.* successfully before, but now that I've switched from Java to Jython, the library fails with the following message:
The parser has encountered more than
"64,000" entity expansions in this
document; this is the limit imposed by
the application.
I have not found a way to fix this, so I probably have to look for another XML library. It could be either Java or Jython-compatible Python and should be efficient. Pythonic would be great, nu.xom.* is simple but not very pythonic. Do you have any suggestions?
Sax is the best way to parse large documents.
Sounds like you're hitting the default expansion limit.
See this note:
http://bugs.sun.com/bugdatabase/view_bug.do?bug_id=4843787
You need to set System property "entityExpansionLimit" to change
the default.
(added) see also the answer to this question.
Try using the SAX parser, it is great for streaming large XML files.
Does jython support xml.etree.ElementTree? If so, use the iterparse method to keep your memory size down. Read this and use elem.clear() as described.
there is a lxml python library, that can parse large files, without loading data to memory.
but i don't know if i jython compatible
Basically, I am looking for a simple way to list and access a set of strings in stream form in an abstract manner. The only issue is that Java's file-accessing API can only be used for listing and reading files, and any sort of non-filesystem storage of the data uses a different API. My question is whether there is some common API I could use (whether included in Java or as an external API) so that I could access both in an abstract manner, but also somewhat efficiently.
Essentially I want a set of lazily streamed text files. Something like Set might be reasonable, except on a filesystem, you would have to open the text streams even if you don't end up wanting to access that file.
Some sort of api like
String[] TextStorage.list()
InputStream TextStorage.open(String elementname);
which could abstractly be used to access either filesystems or databases, or some other storage mechanism I invent in the future (maybe fetching something across the internet).
Is there a library which already does this? Can I do this with the already existing Java API? Do I need to write this myself? I'd be surprised if no-one has encountered this problem before, but my google-fu and stackoverflow searches don't seem to find anything.
you might use HSQL
http://hsqldb.org/
Is there a way to easily create Java Input/Output streams for unnamed pipes created in native code?
Motivation: I need my own implementation of the Process class. The native code spawns me a new process with subprocess' IO streams redirected to unnamed pipes.
Problem: The file descriptors for correct ends of those pipes make their way to Java. At this point I get stuck as I cannot create a new FileDescriptor which I could pass to FileInput/FileOutput stream.
I have used reflection to get around the problem and got communication with a simple bouncer sub-process running. However I have a notion that it is not the cleanest way to go. Have you used this approach? Do you see any problems with this approach? (the platform will never change)
Searching around the internets revealed similar solution using native code.
Any thoughts before I dive into heavy testing of this approach are very welcome. I would like to give a shot to existing code before writing my own IO stream implementations...
Thank you.
I have run into this before as well. The only way I know to create FileDescriptor objects is using reflection (or from JNI) to set the private "int" field inside the FileDescriptor class.
But this works fine and is surely how it's done elsewhere in the standard library, so I believe it's as legitimate as you could expect.
The class java.io.FileReader not found in Java ME.
I need this in order to get the file and then parse it with an xml parser.
Anyone know any alternatives for this class?
*added
using CLDC profile. The xml file to be read is in the JAR.
That's because Java ME provides only a limited subset of the java.io package. You need to use the java.microedition.io package instead.
For actual file I/O you'll need to use the FileConnection class provided by JSR-75.
What Java ME profile are you using? The CLDC does not support the concept of files at all.
In general, FileReader is nothing but a convenience class that wraps an InputStreamReader areound a FileInputStream. It's also very broken because it does not allow specifying the encoding, and should therefore almost never be used.
It would be especially wrong to use it to read XML because proper XML data specifies its encoding, and a proper XML parser will handle that, so you really should pass binary data to the XML parser.
So if you're on the CDC profile, just use a FileInputStream directly.
the question is a bit ambiguous. I think Joachim's answer might be only partial if you are trying to read a local file. I'm certainly not sure though.
If the file is stored as a resource in your JAR, you can access it through the getResourceAsStream method in Class.
If the file is a local file on the file system and if I recall correctly, you need JSR-75 support. Over at Sun's developer page there is an introduction to JSR 75 and the fileconnection
API.
I'm working on a project where we're doing a lot of remote object transfer between a Java service and clients written in other various languages. Given our current constraints I've decided to see what it would take to generate code based on an existing Java class. Basically I need to take a .class file (or a collection of them) parse the bytecode to determine all of the data members and perhaps getters/setters and then write something that can output code in a different language to create a class with the same structure.
I'm not looking for standard decompilers such as JAD. I need to be able to take a .class file and create an object model of its data members and methods. Is this possible at all?
I've used BCEL and find it really quite awkward. ASM is much better. It very extensively uses visitors (which can be a little confusing) and does not create an object model. Not creating an object model turns out to be a bonus, as any model you do want to create is unlikely to look like a literal interpretation of all the data.
I have used BCEL in the past and it was pretty easy to use. It was a few years ago so there may be something better now.
Apache Jakarta BCEL
From your description, it sounds like simple reflection would suffice. You can discover all of the static structure of the class, as well as accessing the fields of a particular instance.
I would only move on to BCEL if you are trying to translate method instructions. (And if that's what you're trying to automate, good luck!)
I'm shocked that no one has mentioned ASM yet. It's the best bytecode library your money can buy. Well, ok it's free.
JAD is a java decompiler that doesn't allow programmatic access. It isn't readily available anymore, and probably won't work for newer projects with Java7 bytecodes.
I think javassist might help you too.
http://www.jboss.org/javassist/
I have never had the need of using it, but if you give it a try, would you let us know your comments about it?
Although I think it is more for bytecode manipulation than .class inspection.