We have a requirement to upload large files (may be up to 200 MB) to SharePoint from a Java/J2EE application.
We know there are out of the box SharePoint web services that allows to upload files to SharePoint. However, our main concern is what happens if concurrent users upload files. For example, we would require to read a 200 MB file for each user on Java server (application server), before invoking SharePoint to send that data. Even if there are 5 concurrent users, the memory consumed would be around 1 GB, and there could be high CPU usage too. Are there any suggestions how to handle the server memory, concurrency of file uploads in this scenario?
I think one option is may be to use technologies like Flash/Flex which does not require another server (Java Application server) in between - however, wondering how this can be achieved in J2EE server?
http://servername/sitename/_vti_bin/copy.asmx
Thanks
ok.. so this is what I understood:
You are trying to use Sharepoint Copy Service
And this service requires the stream to be base64encoded in the Soap envelope.
As the file size is huge your SOAP request size becomes huge and demands more memory
I can think of 2 options:
I dont know much about sharepoint, if it is possible to give location of file to be uploaded than sending bytes, then you can ftp/sftp the file to the sharepoint server and then call a webservice with the location of the file.
In Java instead of using the out of the box api for SOAP messages, write a custom api. When user uploads the file save it as base64 encoded file. And then your custom api will create a soap message and stream it instead of loading everything in memory.
For option 2: try if you can send the file content as a soap attachment. its gets a bit complex if you want to send it as part of the message.
Try it out. I am not sure if works.
SharePoint supports WebDAV protocol for reading / writing files.
You can use many WebDAV libraries that would not require loading complete file in memory.
here is the solution
import java.io.ByteArrayOutputStream;
import java.io.File;
import java.io.FileInputStream;
import java.io.FileNotFoundException;
import java.io.IOException;
import java.net.Authenticator;
import java.net.PasswordAuthentication;
import java.security.cert.CertificateException;
import javax.net.ssl.TrustManager;
import javax.net.ssl.X509TrustManager;
import javax.security.cert.X509Certificate;
import javax.xml.ws.Holder;
import org.apache.cxf.configuration.jsse.TLSClientParameters;
import org.apache.cxf.configuration.security.AuthorizationPolicy;
import org.apache.cxf.endpoint.Client;
import org.apache.cxf.frontend.ClientProxy;
import org.apache.cxf.interceptor.LoggingInInterceptor;
import org.apache.cxf.interceptor.LoggingOutInterceptor;
import org.apache.cxf.jaxws.JaxWsProxyFactoryBean;
import org.apache.cxf.transport.http.HTTPConduit;
import org.apache.cxf.transports.http.configuration.HTTPClientPolicy;
import com.microsoft.schemas.sharepoint.soap.copy.CopyResultCollection;
import com.microsoft.schemas.sharepoint.soap.copy.CopySoap;
import com.microsoft.schemas.sharepoint.soap.copy.DestinationUrlCollection;
import com.microsoft.schemas.sharepoint.soap.copy.FieldInformation;
import com.microsoft.schemas.sharepoint.soap.copy.FieldInformationCollection;
import com.microsoft.schemas.sharepoint.soap.copy.FieldType;
public class Upload {
private static String username = "yourusrename";
private static String password = "yourpassword";
private static String targetPath = "https://www.yoursite.target/filename";
private static String sourcePath = "file.txt";
private static String portUrl = "https://www.yoursite.com/_vti_bin/Copy.asmx";
private static CopySoap soapInstance;
public static void main(String[] args) {
activate();
CopySoap sqs = getInstance();
String url = targetPath;
String sourceUrl = sourcePath;
DestinationUrlCollection urls = new DestinationUrlCollection();
urls.getString().add(url);
File file = null;
byte[] content = null;
try {
FileInputStream fileStream = new FileInputStream(file = new File(sourceUrl));
ByteArrayOutputStream bos = new ByteArrayOutputStream();
byte[] buf = new byte[1024];
for (int readNum; (readNum = fileStream.read(buf)) != -1;) {
bos.write(buf, 0, readNum);
}
content = bos.toByteArray();
} catch (FileNotFoundException e) {
e.printStackTrace();
} catch (IOException ex) {
ex.printStackTrace();
}
FieldInformation titleInfo = new FieldInformation();
titleInfo.setDisplayName("testpage");
titleInfo.setType(FieldType.TEXT);
titleInfo.setValue("Test Page");
FieldInformationCollection infos = new FieldInformationCollection();
infos.getFieldInformation().add(titleInfo);
CopyResultCollection results = new CopyResultCollection();
Holder<CopyResultCollection> resultHolder = new Holder<CopyResultCollection>(results);
Holder<Long> longHolder = new Holder<Long>(new Long(-1));
if (content != null) {
sqs.copyIntoItems(sourceUrl, urls, infos, content, longHolder, resultHolder);
}
}
private static void activate() {
JaxWsProxyFactoryBean factory = new JaxWsProxyFactoryBean();
factory.setServiceClass(CopySoap.class);
factory.setAddress(portUrl);
factory.getInInterceptors().add(new LoggingInInterceptor());
factory.getOutInterceptors().add(new LoggingOutInterceptor());
soapInstance = (CopySoap) factory.create();
Authenticator.setDefault(new SPAuthenticator());
Client client = ClientProxy.getClient(soapInstance);
HTTPConduit http = (HTTPConduit) client.getConduit();
HTTPClientPolicy httpClientPolicy = new HTTPClientPolicy();
httpClientPolicy.setConnectionTimeout(10000);
httpClientPolicy.setAllowChunking(false);
HTTPConduit conduit = (HTTPConduit) client.getConduit();
conduit.setClient(httpClientPolicy);
TLSClientParameters tcp = new TLSClientParameters();
tcp.setTrustManagers(new TrustManager[] { (TrustManager) new TrustAllX509TrustManager() });
conduit.setTlsClientParameters(tcp);
}
public static CopySoap getInstance() {
return soapInstance;
}
static class SPAuthenticator extends Authenticator {
public PasswordAuthentication getPasswordAuthentication() {
System.out.println("hitting SP with username and password for " + getRequestingScheme());
return (new PasswordAuthentication(username, password.toCharArray()));
}
}
/**
* This class allow any X509 certificates to be used to authenticate the
* remote side of a secure socket, including self-signed certificates.
*/
public static class TrustAllX509TrustManager implements X509TrustManager {
/** Empty array of certificate authority certificates. */
private static final X509Certificate[] acceptedIssuers = new X509Certificate[] {};
/**
* Always trust for client SSL chain peer certificate chain with any
* authType authentication types.
*
* #param chain
* the peer certificate chain.
* #param authType`enter
* code here` the authentication type based on the client
* certificate.
*/
public void checkClientTrusted(X509Certificate[] chain, String authType) {
}
/**
* Always trust for server SSL chain peer certificate chain with any
* authType exchange algorithm types.
*
* #param chain
* the peer certificate chain.
* #param authType
* the key exchange algorithm used.
*/
public void checkServerTrusted(X509Certificate[] chain, String authType) {
}
/**
* Return an empty array of certificate authority certificates which are
* trusted for authenticating peers.
*
* #return a empty array of issuer certificates.
*/
public java.security.cert.X509Certificate[] getAcceptedIssuers() {
return null;
}
#Override
public void checkClientTrusted(java.security.cert.X509Certificate[] arg0, String arg1)
throws CertificateException {
// TODO Auto-generated method stub
}
#Override
public void checkServerTrusted(java.security.cert.X509Certificate[] arg0, String arg1)
throws CertificateException {
// TODO Auto-generated method stub
}
}
}
May be I missed something... but when you let users upload files to your J2EE server, wouldn't you be writing the uploaded content to a temp directory first and then stream it to the server?
As you write the buffers immediately to the disk, you wouldnt have any issues with memory limitations.
or take object take
#Autowired
ServletContext c;
byte[] bytes = file.getBytes();
String UPLOAD_FOLDEdR=c.getRealPath("/images");
Path path = Paths.get(UPLOAD_FOLDEdR+"/"+file.getOriginalFilename());
Files.write(path, bytes);
String Pic_Name =file.getOriginalFilename() ;
Related
I have a WSDL file which I've turned into Java code by using WSDL2Java inside SoapUI, it works fine, but now I need to add my company's proxy to it, so every SOAP http request would go through it (but not other http requests).
I've looked through multiple threads concerning the same issue and found these options:
system wide proxy by adding
System.getProperties().put("proxySet", "true");
System.getProperties().put("https.proxyHost", "10.10.10.10");
System.getProperties().put("https.proxyPort", "8080");
which doesn't work for me, since it affect the whole jvm.
adding the following code
HelloService hello = new HelloService();
HelloPortType helloPort = cliente.getHelloPort();
org.apache.cxf.endpoint.Client client = ClientProxy.getClient(helloPort);
HTTPConduit http = (HTTPConduit) client.getConduit();
http.getClient().setProxyServer("proxy");
http.getClient().setProxyServerPort(8080);
http.getProxyAuthorization().setUserName("user proxy");
http.getProxyAuthorization().setPassword("password proxy");
which I don't get how to use. My generated code doesn't have any traces of org.apache.cxf, only javax.xml.ws.
Adding this to my port configuration:
((BindingProvider) port).getRequestContext().put("http.proxyHost", "proxy#example.com");
((BindingProvider) port).getRequestContext().put("http.proxyPort", "80");
Here I use a random non-existing proxy and expect to get an error of any sort(timeout, invalid proxy, etc.), but instead it goes through without any errors.
Here is an example without using 3rd party libraries.
https://github.com/schuch/jaxws-proxy-example/blob/master/jaxws-client-with-proxy/src/main/java/ch/schu/example/helloworld/Client.java
package ch.schu.example.helloworld;
import java.net.ProxySelector;
import ch.schu.example.hello.HelloImpl;
import ch.schu.example.hello.HelloImplService;
public class Client {
public static void main(String[] args) {
ProxySelector.setDefault(new MyProxySelector());
HelloImplService service = new HelloImplService();
HelloImpl hello = service.getHelloImplPort();
System.out.println(hello.sayHello("Howard Wollowitz"));
}
}
https://github.com/schuch/jaxws-proxy-example/blob/master/jaxws-client-with-proxy/src/main/java/ch/schu/example/helloworld/MyProxySelector.java
package ch.schu.example.helloworld;
import java.io.IOException;
import java.net.InetSocketAddress;
import java.net.*;
import java.util.*;
public class MyProxySelector extends ProxySelector {
#Override
public List<Proxy> select(URI uri)
{
System.out.println("select for " + uri.toString());
Proxy proxy = new Proxy(Proxy.Type.HTTP, new InetSocketAddress("localhost", 9999));
ArrayList<Proxy> list = new ArrayList<Proxy>();
list.add(proxy);
return list;
}
#Override
public void connectFailed(URI uri, SocketAddress sa, IOException ioe) {
System.err.println("Connection to " + uri + " failed.");
}
}
I'm working on a tool which analyze some SSL Services, and right now I'm trying to test the client-initiated renegotiation.
I'm using BouncyCastle to do so, with a TlsClientProtocol with a custom function, because BC doesn't "handle" natively the renegotiation.
So, right now I'm using this class:
package org.bouncycastle.crypto.tls;
import java.io.ByteArrayOutputStream;
import java.io.IOException;
import java.io.InputStream;
import java.io.OutputStream;
import java.net.Socket;
import java.util.Hashtable;
import org.bouncycastle.crypto.tls.Certificate;
import org.bouncycastle.crypto.tls.CipherSuite;
import org.bouncycastle.crypto.tls.DefaultTlsClient;
import org.bouncycastle.crypto.tls.ExtensionType;
import org.bouncycastle.crypto.tls.ServerOnlyTlsAuthentication;
import org.bouncycastle.crypto.tls.TlsAuthentication;
import org.bouncycastle.crypto.tls.TlsClientProtocol;
import org.bouncycastle.util.encoders.Hex;
/**
*
* #version 1.0
*/
public class TestProtocol extends TlsClientProtocol {
private byte[] verifyData;
public TestProtocol(InputStream input, OutputStream output) {
super(input, output);
}
// I need to replace this method to have the verifyData,
// because we need to send it into the renegotiation_info ext
#Override
protected void sendFinishedMessage() throws IOException {
verifyData = createVerifyData(getContext().isServer());
ByteArrayOutputStream bos = new ByteArrayOutputStream();
TlsUtils.writeUint8(HandshakeType.finished, bos);
TlsUtils.writeUint24(verifyData.length, bos);
bos.write(verifyData);
byte[] message = bos.toByteArray();
safeWriteRecord(ContentType.handshake, message, 0, message.length);
}
public void renegotiate() throws IOException {
this.connection_state = CS_START;
sendClientHelloMessage();
this.connection_state = CS_CLIENT_HELLO;
completeHandshake();
}
public static void main(String[] args) throws IOException, InterruptedException {
Socket s = new Socket("10.0.0.101", 443);
final TestProtocol proto = new TestProtocol(s.getInputStream(), s.getOutputStream());
proto.connect(new DefaultTlsClient() {
public TlsAuthentication getAuthentication() throws IOException {
return new ServerOnlyTlsAuthentication() {
public void notifyServerCertificate(Certificate serverCertificate) throws IOException {}
};
}
#Override
public int[] getCipherSuites() {
return new int[]{CipherSuite.TLS_RSA_WITH_NULL_SHA, CipherSuite.TLS_RSA_WITH_NULL_MD5};
}
private boolean first = true;
#Override
public Hashtable getClientExtensions() throws IOException {
#SuppressWarnings("unchecked")
Hashtable<Integer, byte[]> clientExtensions = super.getClientExtensions();
if (clientExtensions == null) {
clientExtensions = new Hashtable<Integer, byte[]>();
}
// If this is the first ClientHello, we're not doing anything
if (first) {
first = false;
}
else {
// If this is the second, we add the renegotiation_info extension
byte[] ext = new byte[proto.verifyData.length + 1];
ext[0] = (byte) proto.verifyData.length;
System.arraycopy(proto.verifyData, 0, ext, 1, proto.verifyData.length);
clientExtensions.put(ExtensionType.renegotiation_info, ext);
}
clientExtensions.put(ExtensionType.session_ticket, new byte[] {});
return clientExtensions;
}
});
proto.renegotiate();
}
}
And it's working.. Almost..
When I call the renegotiate() method, it :
- sends the ClientHello
- receives the ServerHello
- receives the Certificate
- receives the ServerHelloDone
- sends the ClientKeyExchange
- sends the ChangeCipherSpec
- sends the Finish
- receives an alert: Fatal, Decrypt Error ; instead of NewSessionTicket,ChangeCipherSpec,Finish
And I just can't figure out why. I thought it could be the SeqNumber used to create the MAC which would need a refresh, but no. When I'm giving an obviously wrong value, I receive also a MAC Error Alert.
To do my testing, I'm using a server allowing CLEAR cipher suites and obviously allowing Client-initiated Renegotiation.
When I'm trying with OpenSSL, it works fine, and I can't see where is the difference, what I'm doing wrong.
The server is on a private VPN, so you can't use it to test things, but here are the .cap of the handshakes.
https://stuff.stooit.com/d/1/528b4a314e35d/openssl.cap
https://stuff.stooit.com/d/1/528b4a54a68cd/my.cap
The first one is the working one, using openSSL.
And the second one is mine, using BouncyCastle.
I'm aware that it won't be very easy to help me on this case, but hey, thanks to ppl who'll try :)
Ok, as always I found the answer a few time after posting my question -- (Even if I was on it for hours / days).
The problem comes with the "Finished" message the client sends. The verify_data is a hash containing all previous handshake messages of the current negotiation.
But in my case, it also contained the handshake messages of the first negotiation, so the verify_data doesn't have the good value.
So to make it works, I need to reset the RecordStream.hash, using RecordStream.hash.reset()
I'm trying to load map tiles from an internal SSL server. The SSL certificate's root of trust is not recognized by the Android system.
W/o*.o*.t*.m*.MapTileDow*(2837): IOException downloading MapTile: /8/37/4 :
javax.net.ssl.SSLPeerUnverifiedException: No peer certificate
I'm already familiar with the problem and have solved it in the rest of the application based on this excellent SO answer. Essentially, I extended my own SSLSocketFactory and X509TrustManager which load my SSL certificate's root of trust from a .bks file bundled with the app. To create a secure connection, I call ((HttpsURLConnection) connection).setSSLSocketFactory(mySSLSocketFactory) and the certificate is verified using my classes with my root of trust.
My question is how do I do the same thing for osmdroid? I'm creating my own XYTileSource where I set the URL, file extension, size, etc. of my map tiles. I see that osmdroid creates its connections to download map tile images in MapTileDownloader. I can write my own replacement class that will address the SSL issue in the same manner, but how do I tell osmdroid to use my custom downloader instead of the default?
It turns out this is possible without changing the source of osmdroid, due to the public MapView(Context context, int tileSizePixels, ResourceProxy resourceProxy, MapTileProviderBase aTileProvider) constrtuctor.
Assuming you already have a custom class like MySSLSocketFactory (which extends javax.net.ssl.SSLSocketFactory), the basic process looks like this:
Create a drop-in replacement class for MapTileDownloader to perform the download in a way that makes use of MySSLSocketFactory. Let's call this MyTileDownloader.
Create a drop-in replacement class for MapTileProviderBasic that instantiates your custom MyTileDownloader. Let's call this MyTileProvider.
Instantiate your tile source as a new XYTileSource (no need to write a custom class).
Instantiate MyTileProvider with your tile source instance.
Instantiate MapVew with your tile provider instance.
MySSLSocketFactory is left as an exercise for the reader. See this post.
MyTileDownloader looks something like this:
import java.io.BufferedInputStream;
import java.io.BufferedOutputStream;
import java.io.ByteArrayInputStream;
import java.io.ByteArrayOutputStream;
import java.io.FileNotFoundException;
import java.io.IOException;
import java.io.InputStream;
import java.io.OutputStream;
import java.net.HttpURLConnection;
import java.net.URL;
import java.net.UnknownHostException;
import javax.net.ssl.HttpsURLConnection;
import javax.net.ssl.SSLSocketFactory;
import org.osmdroid.tileprovider.MapTile;
import org.osmdroid.tileprovider.MapTileRequestState;
import org.osmdroid.tileprovider.modules.IFilesystemCache;
import org.osmdroid.tileprovider.modules.INetworkAvailablityCheck;
import org.osmdroid.tileprovider.modules.MapTileDownloader;
import org.osmdroid.tileprovider.modules.MapTileModuleProviderBase;
import org.osmdroid.tileprovider.tilesource.BitmapTileSourceBase.LowMemoryException;
import org.osmdroid.tileprovider.tilesource.ITileSource;
import org.osmdroid.tileprovider.tilesource.OnlineTileSourceBase;
import org.osmdroid.tileprovider.util.StreamUtils;
import android.graphics.drawable.Drawable;
import android.text.TextUtils;
import android.util.Log;
/**
* A drop-in replacement for {#link MapTileDownloader}. This loads tiles from an
* HTTP or HTTPS server, making use of a custom {#link SSLSocketFactory} for SSL
* peer verification.
*/
public class MyTileDownloader extends MapTileModuleProviderBase {
private static final String TAG = "MyMapTileDownloader";
protected OnlineTileSourceBase mTileSource;
protected final IFilesystemCache mFilesystemCache;
protected final INetworkAvailablityCheck mNetworkAvailablityCheck;
protected final SSLSocketFactory mSSLSocketFactory;
public MyTileDownloader(ITileSource pTileSource,
IFilesystemCache pFilesystemCache,
INetworkAvailablityCheck pNetworkAvailablityCheck,
SSLSocketFactory pSSLSocketFactory) {
super(4, TILE_DOWNLOAD_MAXIMUM_QUEUE_SIZE);
setTileSource(pTileSource);
mFilesystemCache = pFilesystemCache;
mNetworkAvailablityCheck = pNetworkAvailablityCheck;
mSSLSocketFactory = pSSLSocketFactory;
}
public ITileSource getTileSource() {
return mTileSource;
}
#Override
public void setTileSource(final ITileSource tileSource) {
// We are only interested in OnlineTileSourceBase tile sources
if (tileSource instanceof OnlineTileSourceBase)
mTileSource = (OnlineTileSourceBase) tileSource;
else
mTileSource = null;
}
#Override
public boolean getUsesDataConnection() {
return true;
}
#Override
protected String getName() {
return "Online Tile Download Provider";
}
#Override
protected String getThreadGroupName() {
return "downloader";
}
#Override
public int getMinimumZoomLevel() {
return (mTileSource != null ? mTileSource.getMinimumZoomLevel()
: MINIMUM_ZOOMLEVEL);
}
#Override
public int getMaximumZoomLevel() {
return (mTileSource != null ? mTileSource.getMaximumZoomLevel()
: MAXIMUM_ZOOMLEVEL);
}
#Override
protected Runnable getTileLoader() {
return new TileLoader();
};
private class TileLoader extends MapTileModuleProviderBase.TileLoader {
#Override
public Drawable loadTile(final MapTileRequestState aState)
throws CantContinueException {
if (mTileSource == null)
return null;
InputStream in = null;
OutputStream out = null;
final MapTile tile = aState.getMapTile();
try {
if (mNetworkAvailablityCheck != null
&& !mNetworkAvailablityCheck.getNetworkAvailable()) {
if (DEBUGMODE)
Log.d(TAG, "Skipping " + getName()
+ " due to NetworkAvailabliltyCheck.");
return null;
}
final String tileURLString = mTileSource.getTileURLString(tile);
if (DEBUGMODE)
Log.d(TAG, "Downloading Maptile from url: " + tileURLString);
if (TextUtils.isEmpty(tileURLString))
return null;
// Create an HttpURLConnection to download the tile
URL url = new URL(tileURLString);
HttpURLConnection connection = (HttpURLConnection) url
.openConnection();
connection.setConnectTimeout(30000);
connection.setReadTimeout(30000);
// Use our custom SSLSocketFactory for secure connections
if ("https".equalsIgnoreCase(url.getProtocol()))
((HttpsURLConnection) connection)
.setSSLSocketFactory(mSSLSocketFactory);
// Open the input stream
in = new BufferedInputStream(connection.getInputStream(),
StreamUtils.IO_BUFFER_SIZE);
// Check to see if we got success
if (connection.getResponseCode() != 200) {
Log.w(TAG, "Problem downloading MapTile: " + tile
+ " HTTP response: " + connection.getHeaderField(0));
return null;
}
// Read the tile into an in-memory byte array
final ByteArrayOutputStream dataStream = new ByteArrayOutputStream();
out = new BufferedOutputStream(dataStream,
StreamUtils.IO_BUFFER_SIZE);
StreamUtils.copy(in, out);
out.flush();
final byte[] data = dataStream.toByteArray();
final ByteArrayInputStream byteStream = new ByteArrayInputStream(
data);
// Save the data to the filesystem cache
if (mFilesystemCache != null) {
mFilesystemCache.saveFile(mTileSource, tile, byteStream);
byteStream.reset();
}
final Drawable result = mTileSource.getDrawable(byteStream);
return result;
} catch (final UnknownHostException e) {
Log.w(TAG, "UnknownHostException downloading MapTile: " + tile
+ " : " + e);
throw new CantContinueException(e);
} catch (final LowMemoryException e) {
Log.w(TAG, "LowMemoryException downloading MapTile: " + tile
+ " : " + e);
throw new CantContinueException(e);
} catch (final FileNotFoundException e) {
Log.w(TAG, "Tile not found: " + tile + " : " + e);
} catch (final IOException e) {
Log.w(TAG, "IOException downloading MapTile: " + tile + " : "
+ e);
} catch (final Throwable e) {
Log.e(TAG, "Error downloading MapTile: " + tile, e);
} finally {
StreamUtils.closeStream(in);
StreamUtils.closeStream(out);
}
return null;
}
#Override
protected void tileLoaded(final MapTileRequestState pState,
final Drawable pDrawable) {
// Don't return the tile Drawable because we'll wait for the fs
// provider to ask for it. This prevent flickering when a load
// of delayed downloads complete for tiles that we might not
// even be interested in any more.
super.tileLoaded(pState, null);
}
}
}
MyTileProvider looks something like this.
Note that you'll need a way to get access to your instance of MySSLSocketFactory inside this class. This is left as an exercise for the reader. I did this using app.getSSLSocketFactory(), where app is an instance of a custom class that extends Application, but your mileage may vary.
import javax.net.ssl.SSLSocketFactory;
import org.osmdroid.tileprovider.IMapTileProviderCallback;
import org.osmdroid.tileprovider.IRegisterReceiver;
import org.osmdroid.tileprovider.MapTileProviderArray;
import org.osmdroid.tileprovider.MapTileProviderBasic;
import org.osmdroid.tileprovider.modules.INetworkAvailablityCheck;
import org.osmdroid.tileprovider.modules.MapTileFileArchiveProvider;
import org.osmdroid.tileprovider.modules.MapTileFilesystemProvider;
import org.osmdroid.tileprovider.modules.NetworkAvailabliltyCheck;
import org.osmdroid.tileprovider.modules.TileWriter;
import org.osmdroid.tileprovider.tilesource.ITileSource;
import org.osmdroid.tileprovider.util.SimpleRegisterReceiver;
import android.content.Context;
/**
* A drop-in replacement for {#link MapTileProviderBasic}. This top-level tile
* provider implements a basic tile request chain which includes a
* {#link MapTileFilesystemProvider} (a file-system cache), a
* {#link MapTileFileArchiveProvider} (archive provider), and a
* {#link MyTileDownloader} (downloads map tiles via tile source).
*/
public class MyTileProvider extends MapTileProviderArray implements
IMapTileProviderCallback {
public MyTileProvider(final Context pContext, final ITileSource pTileSource) {
this(new SimpleRegisterReceiver(pContext),
new NetworkAvailabliltyCheck(pContext), pTileSource, app
.getSSLSocketFactory());
}
protected MyTileProvider(final IRegisterReceiver pRegisterReceiver,
final INetworkAvailablityCheck aNetworkAvailablityCheck,
final ITileSource pTileSource,
final SSLSocketFactory pSSLSocketFactory) {
super(pTileSource, pRegisterReceiver);
// Look for raw tiles on the file system
final MapTileFilesystemProvider fileSystemProvider = new MapTileFilesystemProvider(
pRegisterReceiver, pTileSource);
mTileProviderList.add(fileSystemProvider);
// Look for tile archives on the file system
final MapTileFileArchiveProvider archiveProvider = new MapTileFileArchiveProvider(
pRegisterReceiver, pTileSource);
mTileProviderList.add(archiveProvider);
// Look for raw tiles on the Internet
final TileWriter tileWriter = new TileWriter();
final MyTileDownloader downloaderProvider = new MyTileDownloader(
pTileSource, tileWriter, aNetworkAvailablityCheck,
pSSLSocketFactory);
mTileProviderList.add(downloaderProvider);
}
}
Finally, the instantiation looks something like this:
XYTileSource tileSource = new XYTileSource("MapQuest", null, 3, 8, 256, ".jpg",
"https://10.0.0.1/path/to/your/map/tiles/");
MapTileProviderBase tileProvider = new MyTileProvider(context, tileSource);
ResourceProxy resourceProxy = new DefaultResourceProxyImpl(context);
MapView mapView = new MapView(context, 256, resourceProxy, tileProvider);
I don't use osmdroid, but unless it has public interface to replace the downloader class(es), your best bet is to get the source and patch it to make it configurable or use your own downloader class. If MapTileDownloader implements some interface you could probably do some reflection voodoo to replace it at runtime, but that might have unknown side effects.
First to say I'm n00b in Java. I can understand most concepts but in my situation I want somebody to help me. I'm using JBoss Netty to handle simple http request and using MemCachedClient check existence of client ip in memcached.
import org.jboss.netty.channel.ChannelHandler;
import static org.jboss.netty.handler.codec.http.HttpHeaders.*;
import static org.jboss.netty.handler.codec.http.HttpHeaders.Names.*;
import static org.jboss.netty.handler.codec.http.HttpResponseStatus.*;
import static org.jboss.netty.handler.codec.http.HttpVersion.*;
import com.danga.MemCached.*;
import java.util.List;
import java.util.Map;
import java.util.Map.Entry;
import java.util.Set;
import org.jboss.netty.buffer.ChannelBuffer;
import org.jboss.netty.buffer.ChannelBuffers;
import org.jboss.netty.channel.ChannelFuture;
import org.jboss.netty.channel.ChannelFutureListener;
import org.jboss.netty.channel.ChannelHandlerContext;
import org.jboss.netty.channel.ExceptionEvent;
import org.jboss.netty.channel.MessageEvent;
import org.jboss.netty.channel.SimpleChannelUpstreamHandler;
import org.jboss.netty.handler.codec.http.Cookie;
import org.jboss.netty.handler.codec.http.CookieDecoder;
import org.jboss.netty.handler.codec.http.CookieEncoder;
import org.jboss.netty.handler.codec.http.DefaultHttpResponse;
import org.jboss.netty.handler.codec.http.HttpChunk;
import org.jboss.netty.handler.codec.http.HttpChunkTrailer;
import org.jboss.netty.handler.codec.http.HttpRequest;
import org.jboss.netty.handler.codec.http.HttpResponse;
import org.jboss.netty.handler.codec.http.HttpResponseStatus;
import org.jboss.netty.handler.codec.http.QueryStringDecoder;
import org.jboss.netty.util.CharsetUtil;
/**
* #author The Netty Project
* #author Andy Taylor (andy.taylor#jboss.org)
* #author Trustin Lee
*
* #version $Rev: 2368 $, $Date: 2010-10-18 17:19:03 +0900 (Mon, 18 Oct 2010) $
*/
#SuppressWarnings({"ALL"})
public class HttpRequestHandler extends SimpleChannelUpstreamHandler {
private HttpRequest request;
private boolean readingChunks;
/** Buffer that stores the response content */
private final StringBuilder buf = new StringBuilder();
protected MemCachedClient mcc = new MemCachedClient();
private static SockIOPool poolInstance = null;
static {
// server list and weights
String[] servers =
{
"lcalhost:11211"
};
//Integer[] weights = { 3, 3, 2 };
Integer[] weights = {1};
// grab an instance of our connection pool
SockIOPool pool = SockIOPool.getInstance();
// set the servers and the weights
pool.setServers(servers);
pool.setWeights(weights);
// set some basic pool settings
// 5 initial, 5 min, and 250 max conns
// and set the max idle time for a conn
// to 6 hours
pool.setInitConn(5);
pool.setMinConn(5);
pool.setMaxConn(250);
pool.setMaxIdle(21600000); //1000 * 60 * 60 * 6
// set the sleep for the maint thread
// it will wake up every x seconds and
// maintain the pool size
pool.setMaintSleep(30);
// set some TCP settings
// disable nagle
// set the read timeout to 3 secs
// and don't set a connect timeout
pool.setNagle(false);
pool.setSocketTO(3000);
pool.setSocketConnectTO(0);
// initialize the connection pool
pool.initialize();
// lets set some compression on for the client
// compress anything larger than 64k
//mcc.setCompressEnable(true);
//mcc.setCompressThreshold(64 * 1024);
}
#Override
public void messageReceived(ChannelHandlerContext ctx, MessageEvent e) throws Exception {
HttpRequest request = this.request = (HttpRequest) e.getMessage();
if(mcc.get(request.getHeader("X-Real-Ip")) != null)
{
HttpResponse response = new DefaultHttpResponse(HTTP_1_1, OK);
response.setHeader("X-Accel-Redirect", request.getUri());
ctx.getChannel().write(response).addListener(ChannelFutureListener.CLOSE);
}
else {
sendError(ctx, NOT_FOUND);
}
}
private void writeResponse(MessageEvent e) {
// Decide whether to close the connection or not.
boolean keepAlive = isKeepAlive(request);
// Build the response object.
HttpResponse response = new DefaultHttpResponse(HTTP_1_1, OK);
response.setContent(ChannelBuffers.copiedBuffer(buf.toString(), CharsetUtil.UTF_8));
response.setHeader(CONTENT_TYPE, "text/plain; charset=UTF-8");
if (keepAlive) {
// Add 'Content-Length' header only for a keep-alive connection.
response.setHeader(CONTENT_LENGTH, response.getContent().readableBytes());
}
// Encode the cookie.
String cookieString = request.getHeader(COOKIE);
if (cookieString != null) {
CookieDecoder cookieDecoder = new CookieDecoder();
Set<Cookie> cookies = cookieDecoder.decode(cookieString);
if(!cookies.isEmpty()) {
// Reset the cookies if necessary.
CookieEncoder cookieEncoder = new CookieEncoder(true);
for (Cookie cookie : cookies) {
cookieEncoder.addCookie(cookie);
}
response.addHeader(SET_COOKIE, cookieEncoder.encode());
}
}
// Write the response.
ChannelFuture future = e.getChannel().write(response);
// Close the non-keep-alive connection after the write operation is done.
if (!keepAlive) {
future.addListener(ChannelFutureListener.CLOSE);
}
}
#Override
public void exceptionCaught(ChannelHandlerContext ctx, ExceptionEvent e)
throws Exception {
e.getCause().printStackTrace();
e.getChannel().close();
}
private void sendError(ChannelHandlerContext ctx, HttpResponseStatus status) {
HttpResponse response = new DefaultHttpResponse(HTTP_1_1, status);
response.setHeader(CONTENT_TYPE, "text/plain; charset=UTF-8");
response.setContent(ChannelBuffers.copiedBuffer(
"Failure: " + status.toString() + "\r\n",
CharsetUtil.UTF_8));
// Close the connection as soon as the error message is sent.
ctx.getChannel().write(response).addListener(ChannelFutureListener.CLOSE);
}
}
When I try to send request like http://127.0.0.1:8090/1/2/3
I'm getting
java.lang.NoClassDefFoundError: com/danga/MemCached/MemCachedClient
at httpClientValidator.server.HttpRequestHandler.<clinit>(HttpRequestHandler.java:66)
I believe it's not related to classpath. May be it's related to context in which mcc doesn't exist.
Any help appreciated
EDIT:
Original code http://docs.jboss.org/netty/3.2/xref/org/jboss/netty/example/http/snoop/package-summary.html
I've modified some parts to fit my needs.
Why do you think this is not classpath related? That's the kind of error you get when the jar you need is not available. How do you start your app?
EDIT
Sorry - i loaded and tried the java_memcached-release_2.5.2 bundle in eclipse and found no issue so far. Debugging the class loading revealed nothing unusual. I can't help besides some more hints to double check:
make sure your download is correct. download and unpack again. (are the com.schooner.* classes available?)
make sure you use > java 1.5
make sure your classpath is correct and complete. The example you have shown does not include netty. Where is it.
I'm not familiar with interactions stemming from adding a classpath to the manifest. Maybe revert to plain style, add all jars needed (memcached, netty, yours) to the classpath and reference the main class to start, not a startable jar file
I'm accessing an internal database using MATLAB's urlread command, everything was working fine until the service was moved to a secure server (i.e. with an HTTPS address rather than an HTTP address). Now urlread no longer successfully retrieves results. It gives an error:
Error downloading URL. Your network connection may be down or your proxy settings improperly configured.
I believe the problem is that the service is using an invalid digital certificate since if I try to access the resource directly in a web browser I get "untrusted connection" warning which I am able to pass through by adding the site to an Exception list. urlread doesn't have an obvious way of handling this problem.
Under the hood urlread is using Java to access web resources, and the error is thrown at this line:
inputStream = urlConnection.getInputStream;
where urlConnection is a Java object: sun.net.www.protocol.https.HttpsURLConnectionImpl.
Anyone suggest a workaround for this problem?
Consider the following Java class. Borrowing from this code: Disabling Certificate Validation in an HTTPS Connection
C:\MATLAB\MyJavaClasses\com\stackoverflow\Downloader.java
package com.stackoverflow;
import java.io.BufferedReader;
import java.io.InputStream;
import java.io.InputStreamReader;
import java.net.HttpURLConnection;
import java.net.URL;
import java.security.cert.X509Certificate;
import javax.net.ssl.HttpsURLConnection;
import javax.net.ssl.SSLContext;
import javax.net.ssl.SSLSession;
import javax.net.ssl.TrustManager;
import javax.net.ssl.X509TrustManager;
import javax.net.ssl.HostnameVerifier;
public class Downloader {
public static String getData(String address) throws Exception {
// Create a trust manager that does not validate certificate chains
TrustManager[] trustAllCerts = new TrustManager[] {
new X509TrustManager() {
public java.security.cert.X509Certificate[] getAcceptedIssuers() {
return null;
}
public void checkClientTrusted(X509Certificate[] certs, String authType) {
}
public void checkServerTrusted(X509Certificate[] certs, String authType) {
}
}
};
// Create a host name verifier that always passes
HostnameVerifier allHostsValid = new HostnameVerifier() {
public boolean verify(String hostname, SSLSession session) {
return true;
}
};
// Install the all-trusting trust manager
SSLContext sc = SSLContext.getInstance("SSL");
sc.init(null, trustAllCerts, new java.security.SecureRandom());
HttpsURLConnection.setDefaultSSLSocketFactory(sc.getSocketFactory());
// Install the all-trusting host verifier
HttpsURLConnection.setDefaultHostnameVerifier(allHostsValid);
// open connection
URL page = new URL(address);
HttpURLConnection conn = (HttpURLConnection) page.openConnection();
BufferedReader buff = new BufferedReader(new InputStreamReader(conn.getInputStream()));
// read text
String line;
StringBuffer text = new StringBuffer();
while ( (line = buff.readLine()) != null ) {
//System.out.println(line);
text.append(line + "\n");
}
buff.close();
return text.toString();
}
public static void main(String[] argv) throws Exception {
String str = getData("https://expired.badssl.com/");
System.out.println(str);
}
}
MATLAB
First we compile the Java class (we must use a JDK version compatible with MATLAB):
>> version -java
>> system('javac C:\MATLAB\MyJavaClasses\com\stackoverflow\Downloader.java');
Next we instantiate and use it MATLAB as:
javaaddpath('C:\MATLAB\MyJavaClasses')
dl = com.stackoverflow.Downloader;
str = char(dl.getData('https://expired.badssl.com/'));
web(['text://' str], '-new')
Here are a few URLs with bad SSL certificates to test:
urls = {
'https://expired.badssl.com/' % expired
'https://wrong.host.badssl.com/' % wrong host
'https://self-signed.badssl.com/' % self-signed
'https://revoked.grc.com/' % revoked
};
UPDATE: I should mention that starting with R2014b, MATLAB has a new function webread that supersedes urlread.
thanks for the solution. It worked, however, sometimes, I had received the following exception "java.io.IOException: The issuer can not be found in the trusted CA list." and I was not able to get rid of this error.
Therefore, I tried an alternative solution that works well. You can use the following Java code in Matlab function:
function str = ReadUrl(url)
is = java.net.URL([], url, sun.net.www.protocol.https.Handler).openConnection().getInputStream();
br = java.io.BufferedReader(java.io.InputStreamReader(is));
str = char(br.readLine());
end
Best,
Jan
Note also that the "canonical" way to solve this issue is to import the certificate into MATLAB's keystore (i.e., not your JVM's keystore).
This is documented here: Mathworks on using untrusted SSL certificates.