I have an unbounded stream of complex objects that I want to load into BigQuery. The structure of these objects represents the schema of my destination table in BigQuery.
The problem is that since there are a lot of nested fields in the POJO, its an extremely tedious task to convert it to a TableSchema object and I'm looking for a quick/ automated way to convert my POJO to TableSchema object while writing to BigQuery.
I'm not very familiar with Apache Beam API, and any help will be appreciated.
In a pipeline, I load a list of schema from GCS. I keep them in string format because the TableSchema is not serializable. However, I load them to TableSchema for validate them.
Then I add them in string format to a map in the Option object.
String schema = new String(blob.getContent());
// Decorate list of fields for allowing a correct parsing
String targetSchema = "{\"fields\":" + schema + "}";
try {
//Preload schema to ensure validity, but then use string version
Transport.getJsonFactory().fromString(targetSchema, TableSchema.class);
String tableName = blob.getName().replace(SCHEMA_FILE_PREFIX, "").replace(SCHEMA_FILE_SUFFIX, "");
tableSchemaStringMap.put(tableName, targetSchema);
} catch (IOException e) {
logger.warn("impossible to read schema " + blob.getName() + " in bucket gs://" + options.getSchemaBucket());
}
I didn't find another solution when I developed this.
In my company I created kind of a ORM (we called OBQM) to do this. We are expecting to release it to the public. The code is quite big (specially because I created annotations and so on) but I can share with you some snippets for a quick schema generation:
public TableSchema generateTableSchema(#Nonnull final Class cls) {
final TableSchema tableSchema = new TableSchema();
tableSchema.setFields(generateFieldsSchema(cls));
return tableSchema;
}
public List<TableFieldSchema> generateFieldsSchema(#Nonnull final Class cls) {
final List<TableFieldSchema> schemaFields = new ArrayList<>();
final Field[] clsFields = cls.getFields();
for (final Field field : clsFields) {
schemaFields.add(fromFieldToSchemaField(field));
}
return schemaFields;
}
This code takes all the fields from the POJO class and creates a TableSchema object (the one that BigQueryIO uses in ApacheBeam). You can see a method that I created called fromFieldToSchemaField. This method identifies each field type and setup the field name, mode, description and type. In this case to keep it simple I'm going to focus on the type and name:
public static TableFieldSchema fromFieldToSchemaField(#Nonnull final Field field) {
return fromFieldToSchemaField(field, 0);
}
public static TableFieldSchema fromFieldToSchemaField(
#Nonnull final Field field,
final int iteration) {
final TableFieldSchema schemaField = new TableFieldSchema();
final Type customType = field.getGenericType().getTypeName()
schemaField.setName(field.getName());
schemaField.setMode("NULLABLE"); // You can add better logic here, we use annotations to override this value
schemaField.setType(getFieldTypeString(field));
schemaField.setDescription("Optional"); // Optional
if (iteration < MAX_RECURSION
&& (isStruct(schemaField.getType())
|| isRecord(schemaField.getType()))) {
final List<TableFieldSchema> schemaFields = new ArrayList<>();
final Field[] fields = getFieldsFromComplexObjectField(field);
for (final Field subField : fields) {
schemaFields.add(
fromFieldToSchemaField(
subField, iteration + 1));
}
schemaField.setFields(schemaFields.isEmpty() ? null : schemaFields);
}
return schemaField;
}
And now the method that returns the BigQuery field type.
public static String getFieldTypeString(#Nonnull final Field field) {
// On my side this code is much complex but this is a short version of that
final Class<?> cls = (Class<?>) field.getGenericType()
if (cls.isAssignableFrom(String.class)) {
return "STRING";
} else if (cls.isAssignableFrom(Integer.class) || cls.isAssignableFrom(Short.class)) {
return "INT64";
} else if (cls.isAssignableFrom(Double.class)) {
return "NUMERIC";
} else if (cls.isAssignableFrom(Float.class)) {
return "FLOAT64";
} else if (cls.isAssignableFrom(Boolean.class)) {
return "BOOLEAN";
} else if (cls.isAssignableFrom(Double.class)) {
return "BYTES";
} else if (cls.isAssignableFrom(Date.class)
|| cls.isAssignableFrom(DateTime.class)) {
return "TIMESTAMP";
} else {
return "STRUCT";
}
}
Keep in mind that I'm not showing how to identify primitive types or arrays. But this is a good start for your code :). Please let me know if you need any help.
If your using JSON for the message serialization in PubSub you can make use of one of the provided templates:
PubSub To BigQuery Template
The code for that template is here:
PubSubToBigQuery.java
sorry for duplicating the question, but my problem is other.
I have JSON parser method where I parse from json-string to map. But json-string has a value which is json-string too. Something like that:
{
"status_code":"255",
"data":"{\"user\":{\"idpolzovatel\":1,\"id_poluch_tip\":1,\"fio_polzovatel\":\"Andrew Artificial\",\"login\":\"imi\",\"parol\":\"698d51a19d8a121ce581499d7b701668\",\"key\":null,\"nachalnik\":1,\"buhgalter\":0,\"delopr\":1},\"token\":\"230047517dd122c8f8116a6fa591a704\"}",
"message":"Successfull!"
}
So, my parse-method:
public Map<String, String> convertToMapFromJSON(String res){
ObjectMapper objectMapper = new ObjectMapper();
Map<String, String> response = new HashMap<String, String>();
try {
response = objectMapper.readValue(res, new TypeReference<Map<String, String>>);
int t = 0;
} catch (IOException e) {
e.printStackTrace();
}
return response;
}
I get response in client:
ResponseEntity<String> responseEntity = restTemplate.postForEntity(REST_SERVICE_URI + "/auth/", data, String.class);
get body
String res = responseEntity.getBody();//получаем тело запроса в формате JSON
then use those method:
Map<String, String> response = convertToMapFromJSON(res);
Map<String, String> data1 = convertToMapFromJSON(response.get("data"));
Map<String, String> userDetailes = convertToMapFromJSON(data1.get("user"));
but, when I use last method data1.get("user"); I get exception:
java.lang.ClassCastException: java.util.LinkedHashMap cannot be cast to java.lang.String
ok, got it. So, data1.get("user") isn't a string, it's linkedHashMap. So, I could do this then:
Map<String, String> userDetailes = data1.get("user");
? But then I get the error, where IDE say me, that data1.get("user") is a string.
Screenshot from debugger:
So, how can I get this LinkedHashMap with my userdata? Sorry, for my english. Thank you.
Java apply type erasure for generics. It checks type correctness at compile time and then remove generic signature in compile code (ByteCode). Therefore, there's no check at runtime.
See this example which have same behaviour as your JSON library:
/** Returns a generic map which all keys are string but not values **/
T <T extends Map> raw(Class<T> clazz) {
Map object = new LinkedHashMap();
object.put("string", "This is a String");
object.put("map" , new LinkedHashMap());
return (T) object;
}
Here is your code:
/** codes you try to execute/write **/
void withStringValues() {
Map<String,String> object = raw(Map<String,String>.class);
String string = object.get("string"); // Ok
String map = object.get("map"); // ClassCastException
Map map = object.get("map"); // Doesn't compile
}
As you can see the call to raw is considered valid as compiled code don't check for generics. But it makes an invalid and implicit cast from Map to Map<String,String> which actually doesn't occured in compiled code.
Generics are remove and is the compiled version:
void withTypeErasure() {
Map object = raw(Map.class);
String string = (String) object.get("string");
String map = (String) object.get("map");
}
As you can see, Java compiler has removed all generic and adds necessary casts. You can see what's going wrong here.
Your real code must look like this:
void withRealValues() {
Map<String,Object> object = raw(Map<String,Object>.class);
String string = (String) object.get("string"); // Ok
Map<String,Object> map = (Map) object.get("map"); // Ok
}
Looks like ObjectMapper has decoded the string to be of JSON format and has parsed it for you. You could just add a new method to parse (data1.get("user")) which returns a Map.
Java EE has ServletRequest.getParameterValues().
On non-EE platforms, URL.getQuery() simply returns a string.
What's the normal way to properly parse the query string in a URL when not on Java EE?
It is popular in the answers to try and make your own parser. This is very interesting and exciting micro-coding project, but I cannot say that it is a good idea.
The code snippets below are generally flawed or broken. Breaking them is an interesting exercise for the reader. And to the hackers attacking the websites that use them.
Parsing query strings is a well defined problem but reading the spec and understanding the nuances is non-trivial. It is far better to let some platform library coder do the hard work, and do the fixing, for you!
On Android:
import android.net.Uri;
[...]
Uri uri=Uri.parse(url_string);
uri.getQueryParameter("para1");
Since Android M things have got more complicated. The answer of android.net.URI.getQueryParameter() has a bug which breaks spaces before JellyBean.
Apache URLEncodedUtils.parse() worked, but was deprecated in L, and removed in M.
So the best answer now is UrlQuerySanitizer. This has existed since API level 1 and still exists. It also makes you think about the tricky issues like how do you handle special characters, or repeated values.
The simplest code is
UrlQuerySanitizer.ValueSanitizer sanitizer = UrlQuerySanitizer.getAllButNullLegal();
// remember to decide if you want the first or last parameter with the same name
// If you want the first call setPreferFirstRepeatedParameter(true);
sanitizer.parseUrl(url);
String value = sanitizer.getValue("paramName");
If you are happy with the default parsing behavior you can do:
new UrlQuerySanitizer(url).getValue("paramName")
but you should make sure you understand what the default parsing behavor is, as it might not be what you want.
On Android, the Apache libraries provide a Query parser:
http://developer.android.com/reference/org/apache/http/client/utils/URLEncodedUtils.html and http://hc.apache.org/httpcomponents-client-ga/httpclient/apidocs/org/apache/http/client/utils/URLEncodedUtils.html
public static Map<String, List<String>> getUrlParameters(String url)
throws UnsupportedEncodingException {
Map<String, List<String>> params = new HashMap<String, List<String>>();
String[] urlParts = url.split("\\?");
if (urlParts.length > 1) {
String query = urlParts[1];
for (String param : query.split("&")) {
String pair[] = param.split("=", 2);
String key = URLDecoder.decode(pair[0], "UTF-8");
String value = "";
if (pair.length > 1) {
value = URLDecoder.decode(pair[1], "UTF-8");
}
List<String> values = params.get(key);
if (values == null) {
values = new ArrayList<String>();
params.put(key, values);
}
values.add(value);
}
}
return params;
}
If you have jetty (server or client) libs on your classpath you can use the jetty util classes (see javadoc), e.g.:
import org.eclipse.jetty.util.*;
URL url = new URL("www.example.com/index.php?foo=bar&bla=blub");
MultiMap<String> params = new MultiMap<String>();
UrlEncoded.decodeTo(url.getQuery(), params, "UTF-8");
assert params.getString("foo").equals("bar");
assert params.getString("bla").equals("blub");
If you're using Spring 3.1 or greater (yikes, was hoping that support went back further), you can use the UriComponents and UriComponentsBuilder:
UriComponents components = UriComponentsBuilder.fromUri(uri).build();
List<String> myParam = components.getQueryParams().get("myParam");
components.getQueryParams() returns a MultiValueMap<String, String>
Here's some more documentation.
I have methods to achieve this:
1):
public static String getQueryString(String url, String tag) {
String[] params = url.split("&");
Map<String, String> map = new HashMap<String, String>();
for (String param : params) {
String name = param.split("=")[0];
String value = param.split("=")[1];
map.put(name, value);
}
Set<String> keys = map.keySet();
for (String key : keys) {
if(key.equals(tag)){
return map.get(key);
}
System.out.println("Name=" + key);
System.out.println("Value=" + map.get(key));
}
return "";
}
2) and the easiest way to do this Using Uri class:
public static String getQueryString(String url, String tag) {
try {
Uri uri=Uri.parse(url);
return uri.getQueryParameter(tag);
}catch(Exception e){
Log.e(TAG,"getQueryString() " + e.getMessage());
}
return "";
}
and this is an example of how to use either of two methods:
String url = "http://www.jorgesys.com/advertisements/publicidadmobile.htm?position=x46&site=reform&awidth=800&aheight=120";
String tagValue = getQueryString(url,"awidth");
the value of tagValue is 800
For a servlet or a JSP page you can get querystring key/value pairs by using request.getParameter("paramname")
String name = request.getParameter("name");
There are other ways of doing it but that's the way I do it in all the servlets and jsp pages that I create.
On Android, I tried using #diyism answer but I encountered the space character issue raised by #rpetrich, for example:
I fill out a form where username = "us+us" and password = "pw pw" causing a URL string to look like:
http://somewhere?username=us%2Bus&password=pw+pw
However, #diyism code returns "us+us" and "pw+pw", i.e. it doesn't detect the space character. If the URL was rewritten with %20 the space character gets identified:
http://somewhere?username=us%2Bus&password=pw%20pw
This leads to the following fix:
Uri uri = Uri.parse(url_string.replace("+", "%20"));
uri.getQueryParameter("para1");
Parsing the query string is a bit more complicated than it seems, depending on how forgiving you want to be.
First, the query string is ascii bytes. You read in these bytes one at a time and convert them to characters. If the character is ? or & then it signals the start of a parameter name. If the character is = then it signals the start of a paramter value. If the character is % then it signals the start of an encoded byte. Here is where it gets tricky.
When you read in a % char you have to read the next two bytes and interpret them as hex digits. That means the next two bytes will be 0-9, a-f or A-F. Glue these two hex digits together to get your byte value. But remember, bytes are not characters. You have to know what encoding was used to encode the characters. The character é does not encode the same in UTF-8 as it does in ISO-8859-1. In general it's impossible to know what encoding was used for a given character set. I always use UTF-8 because my web site is configured to always serve everything using UTF-8 but in practice you can't be certain. Some user-agents will tell you the character encoding in the request; you can try to read that if you have a full HTTP request. If you just have a url in isolation, good luck.
Anyway, assuming you are using UTF-8 or some other multi-byte character encoding, now that you've decoded one encoded byte you have to set it aside until you capture the next byte. You need all the encoded bytes that are together because you can't url-decode properly one byte at a time. Set aside all the bytes that are together then decode them all at once to reconstruct your character.
Plus it gets more fun if you want to be lenient and account for user-agents that mangle urls. For example, some webmail clients double-encode things. Or double up the ?&= chars (for example: http://yoursite.com/blah??p1==v1&&p2==v2). If you want to try to gracefully deal with this, you will need to add more logic to your parser.
On Android its simple as the code below:
UrlQuerySanitizer sanitzer = new UrlQuerySanitizer(url);
String value = sanitzer.getValue("your_get_parameter");
Also if you don't want to register each expected query key use:
sanitzer.setAllowUnregisteredParamaters(true)
Before calling:
sanitzer.parseUrl(yourUrl)
On Android, you can use the Uri.parse static method of the android.net.Uri class to do the heavy lifting. If you're doing anything with URIs and Intents you'll want to use it anyways.
Just for reference, this is what I've ended up with (based on URLEncodedUtils, and returning a Map).
Features:
it accepts the query string part of the url (you can use request.getQueryString())
an empty query string will produce an empty Map
a parameter without a value (?test) will be mapped to an empty List<String>
Code:
public static Map<String, List<String>> getParameterMapOfLists(String queryString) {
Map<String, List<String>> mapOfLists = new HashMap<String, List<String>>();
if (queryString == null || queryString.length() == 0) {
return mapOfLists;
}
List<NameValuePair> list = URLEncodedUtils.parse(URI.create("http://localhost/?" + queryString), "UTF-8");
for (NameValuePair pair : list) {
List<String> values = mapOfLists.get(pair.getName());
if (values == null) {
values = new ArrayList<String>();
mapOfLists.put(pair.getName(), values);
}
if (pair.getValue() != null) {
values.add(pair.getValue());
}
}
return mapOfLists;
}
A compatibility helper (values are stored in a String array just as in ServletRequest.getParameterMap()):
public static Map<String, String[]> getParameterMap(String queryString) {
Map<String, List<String>> mapOfLists = getParameterMapOfLists(queryString);
Map<String, String[]> mapOfArrays = new HashMap<String, String[]>();
for (String key : mapOfLists.keySet()) {
mapOfArrays.put(key, mapOfLists.get(key).toArray(new String[] {}));
}
return mapOfArrays;
}
This works for me..
I'm not sure why every one was after a Map, List>
All I needed was a simple name value Map.
To keep things simple I used the build in URI.getQuery();
public static Map<String, String> getUrlParameters(URI uri)
throws UnsupportedEncodingException {
Map<String, String> params = new HashMap<String, String>();
for (String param : uri.getQuery().split("&")) {
String pair[] = param.split("=");
String key = URLDecoder.decode(pair[0], "UTF-8");
String value = "";
if (pair.length > 1) {
value = URLDecoder.decode(pair[1], "UTF-8");
}
params.put(new String(key), new String(value));
}
return params;
}
Guava's Multimap is better suited for this. Here is a short clean version:
Multimap<String, String> getUrlParameters(String url) {
try {
Multimap<String, String> ret = ArrayListMultimap.create();
for (NameValuePair param : URLEncodedUtils.parse(new URI(url), "UTF-8")) {
ret.put(param.getName(), param.getValue());
}
return ret;
} catch (URISyntaxException e) {
throw new RuntimeException(e);
}
}
Origanally answered here
On Android, there is Uri class in package android.net . Note that Uri is part of android.net, while URI is part of java.net .
Uri class has many functions to extract query key-value pairs.
Following function returns key-value pairs in the form of HashMap.
In Java:
Map<String, String> getQueryKeyValueMap(Uri uri){
HashMap<String, String> keyValueMap = new HashMap();
String key;
String value;
Set<String> keyNamesList = uri.getQueryParameterNames();
Iterator iterator = keyNamesList.iterator();
while (iterator.hasNext()){
key = (String) iterator.next();
value = uri.getQueryParameter(key);
keyValueMap.put(key, value);
}
return keyValueMap;
}
In Kotlin:
fun getQueryKeyValueMap(uri: Uri): HashMap<String, String> {
val keyValueMap = HashMap<String, String>()
var key: String
var value: String
val keyNamesList = uri.queryParameterNames
val iterator = keyNamesList.iterator()
while (iterator.hasNext()) {
key = iterator.next() as String
value = uri.getQueryParameter(key) as String
keyValueMap.put(key, value)
}
return keyValueMap
}
Apache AXIS2 has a self-contained implementation of QueryStringParser.java. If you are not using Axis2, just download the sourcecode and test case from here -
http://svn.apache.org/repos/asf/axis/axis2/java/core/trunk/modules/kernel/src/org/apache/axis2/transport/http/util/QueryStringParser.java
http://svn.apache.org/repos/asf/axis/axis2/java/core/trunk/modules/kernel/test/org/apache/axis2/transport/http/util/QueryStringParserTest.java
if (queryString != null)
{
final String[] arrParameters = queryString.split("&");
for (final String tempParameterString : arrParameters)
{
final String[] arrTempParameter = tempParameterString.split("=");
if (arrTempParameter.length >= 2)
{
final String parameterKey = arrTempParameter[0];
final String parameterValue = arrTempParameter[1];
//do something with the parameters
}
}
}
I don't think there is one in JRE. You can find similar functions in other packages like Apache HttpClient. If you don't use any other packages, you just have to write your own. It's not that hard. Here is what I use,
public class QueryString {
private Map<String, List<String>> parameters;
public QueryString(String qs) {
parameters = new TreeMap<String, List<String>>();
// Parse query string
String pairs[] = qs.split("&");
for (String pair : pairs) {
String name;
String value;
int pos = pair.indexOf('=');
// for "n=", the value is "", for "n", the value is null
if (pos == -1) {
name = pair;
value = null;
} else {
try {
name = URLDecoder.decode(pair.substring(0, pos), "UTF-8");
value = URLDecoder.decode(pair.substring(pos+1, pair.length()), "UTF-8");
} catch (UnsupportedEncodingException e) {
// Not really possible, throw unchecked
throw new IllegalStateException("No UTF-8");
}
}
List<String> list = parameters.get(name);
if (list == null) {
list = new ArrayList<String>();
parameters.put(name, list);
}
list.add(value);
}
}
public String getParameter(String name) {
List<String> values = parameters.get(name);
if (values == null)
return null;
if (values.size() == 0)
return "";
return values.get(0);
}
public String[] getParameterValues(String name) {
List<String> values = parameters.get(name);
if (values == null)
return null;
return (String[])values.toArray(new String[values.size()]);
}
public Enumeration<String> getParameterNames() {
return Collections.enumeration(parameters.keySet());
}
public Map<String, String[]> getParameterMap() {
Map<String, String[]> map = new TreeMap<String, String[]>();
for (Map.Entry<String, List<String>> entry : parameters.entrySet()) {
List<String> list = entry.getValue();
String[] values;
if (list == null)
values = null;
else
values = (String[]) list.toArray(new String[list.size()]);
map.put(entry.getKey(), values);
}
return map;
}
}
public static Map <String, String> parseQueryString (final URL url)
throws UnsupportedEncodingException
{
final Map <String, String> qps = new TreeMap <String, String> ();
final StringTokenizer pairs = new StringTokenizer (url.getQuery (), "&");
while (pairs.hasMoreTokens ())
{
final String pair = pairs.nextToken ();
final StringTokenizer parts = new StringTokenizer (pair, "=");
final String name = URLDecoder.decode (parts.nextToken (), "ISO-8859-1");
final String value = URLDecoder.decode (parts.nextToken (), "ISO-8859-1");
qps.put (name, value);
}
return qps;
}
Use Apache HttpComponents and wire it up with some collection code to access params by value: http://www.joelgerard.com/2012/09/14/parsing-query-strings-in-java-and-accessing-values-by-key/
using Guava:
Multimap<String,String> parseQueryString(String queryString, String encoding) {
LinkedListMultimap<String, String> result = LinkedListMultimap.create();
for(String entry : Splitter.on("&").omitEmptyStrings().split(queryString)) {
String pair [] = entry.split("=", 2);
try {
result.put(URLDecoder.decode(pair[0], encoding), pair.length == 2 ? URLDecoder.decode(pair[1], encoding) : null);
} catch (UnsupportedEncodingException e) {
throw new RuntimeException(e);
}
}
return result;
}
Answering here because this is a popular thread. This is a clean solution in Kotlin that uses the recommended UrlQuerySanitizer api. See the official documentation. I have added a string builder to concatenate and display the params.
var myURL: String? = null
// if the url is sent from a different activity where you set it to a value
if (intent.hasExtra("my_value")) {
myURL = intent.extras.getString("my_value")
} else {
myURL = intent.dataString
}
val sanitizer = UrlQuerySanitizer(myURL)
// We don't want to manually define every expected query *key*, so we set this to true
sanitizer.allowUnregisteredParamaters = true
val parameterNamesToValues: List<UrlQuerySanitizer.ParameterValuePair> = sanitizer.parameterList
val parameterIterator: Iterator<UrlQuerySanitizer.ParameterValuePair> = parameterNamesToValues.iterator()
// Helper simply so we can display all values on screen
val stringBuilder = StringBuilder()
while (parameterIterator.hasNext()) {
val parameterValuePair: UrlQuerySanitizer.ParameterValuePair = parameterIterator.next()
val parameterName: String = parameterValuePair.mParameter
val parameterValue: String = parameterValuePair.mValue
// Append string to display all key value pairs
stringBuilder.append("Key: $parameterName\nValue: $parameterValue\n\n")
}
// Set a textView's text to display the string
val paramListString = stringBuilder.toString()
val textView: TextView = findViewById(R.id.activity_title) as TextView
textView.text = "Paramlist is \n\n$paramListString"
// to check if the url has specific keys
if (sanitizer.hasParameter("type")) {
val type = sanitizer.getValue("type")
println("sanitizer has type param $type")
}
this method takes the uri and return map of par name and par value
public static Map<String, String> getQueryMap(String uri) {
String queryParms[] = uri.split("\\?");
Map<String, String> map = new HashMap<>();//
if (queryParms == null || queryParms.length == 0) return map;
String[] params = queryParms[1].split("&");
for (String param : params) {
String name = param.split("=")[0];
String value = param.split("=")[1];
map.put(name, value);
}
return map;
}
You say "Java" but "not Java EE". Do you mean you are using JSP and/or servlets but not a full Java EE stack? If that's the case, then you should still have request.getParameter() available to you.
If you mean you are writing Java but you are not writing JSPs nor servlets, or that you're just using Java as your reference point but you're on some other platform that doesn't have built-in parameter parsing ... Wow, that just sounds like an unlikely question, but if so, the principle would be:
xparm=0
word=""
loop
get next char
if no char
exit loop
if char=='='
param_name[xparm]=word
word=""
else if char=='&'
param_value[xparm]=word
word=""
xparm=xparm+1
else if char=='%'
read next two chars
word=word+interpret the chars as hex digits to make a byte
else
word=word+char
(I could write Java code but that would be pointless, because if you have Java available, you can just use request.getParameters.)