I write app in Spring.
This is my json: (it is an array of json objects)
[{"id" : 643419352,
"status" : "removed_by_user",
"url" : "https://www.olx.pl/d/oferta/opona-12-1-2-x-2-1-4-etrto-62-203-detka-CID767-IDHxILu.html",
"created_at" : "2020-11-27 10:46:07",
"activated_at" : "2020-12-11 12:41:12",
"valid_to" : "2020-12-17 15:38:10",
"title" : "opona 12 1/2 \" x 2 1/4 etrto 62-203 + dętka",
"description" : "opona w bardzo dobrym stanie + dętka, rozmiar 12 1/2 x 2 1/4 , dętka z zaworem samochodowym",
"category_id" : 1655,
"advertiser_type" : "private",
"external_id" : null,
"external_url" : null,
"contact" : {
"name" : "Damazy",
"phone" : "501474399"
},
"location" : {
"city_id" : 10609,
"district_id" : 301,
"latitude" : "51.80178",
"longitude" : "19.43928"
},
"images" : [ {
"url" : "https://ireland.apollo.olxcdn.com:443/v1/files/efa9any4ryrb-PL/image;s=1000x700"
} ],
"price" : {
"value" : "9",
"currency" : "PLN",
"negotiable" : false,
"budget" : false,
"trade" : false
},
"salary" : null,
"attributes" : [ {
"code" : "state",
"value" : "used",
"values" : null
} ],
"courier" : null
}, {
"id" : 643435839,
"status" : "removed_by_user",
"url" : "https://www.olx.pl/d/oferta/opona-4-80-4-00-8-do-taczki-nowa-CID628-IDHxN3p.html",
"created_at" : "2020-11-27 11:53:47",
"activated_at" : "2020-11-27 11:54:36",
"valid_to" : "2020-12-17 15:38:07",
"title" : "opona 4.80/4.00 - 8 do taczki nowa!!!",
"description" : "opona do taczki, nowa, nigdy nie używana, stan idealny.\r\nrozmiar 4.80/4.00-8. \r\nopona do taczki, nowa, nigdy nie używana, stan idealny.\r\nrozmiar 4.80/4.00-8.",
"category_id" : 1636,
"advertiser_type" : "private",
"external_id" : null,
"external_url" : null,
"contact" : {
"name" : "Damazy",
"phone" : "501474399"
},
"location" : {
"city_id" : 10609,
"district_id" : 301,
"latitude" : "51.80178",
"longitude" : "19.43928"
},
"images" : [ {
"url" : "https://ireland.apollo.olxcdn.com:443/v1/files/qmvssagjnq1r2-PL/image;s=1000x700"
} ],
"price" : {
"value" : "9",
"currency" : "PLN",
"negotiable" : false,
"budget" : false,
"trade" : false
},
"salary" : null,
"attributes" : [ {
"code" : "state",
"value" : "new",
"values" : null
} ],
"courier" : null
}]
and this is my entity class Advert :
#Setter
#Getter
#NoArgsConstructor
#AllArgsConstructor
#ToString
#Entity
public class Advert {
#Id
#GeneratedValue(strategy = GenerationType.AUTO)
private Long ident;
private int id;
private String status;
private String url;
private String created_at;
private String activated_at;
private String valid_to;
private String title;
#Lob
private String description;
private int category_id;
private String advertiser_type;
private Long external_id;
private String external_url;
private String salary;
private String attributes;
private String courier;
#Embedded
private Location location;
#Embedded
private Contact contact;
#Embedded
private Price price;
private String images;
and my saveAdverts method :
#RequestMapping("/saveadverts")
public String saveAdverts() throws IOException {
HttpEntity<String> requestEntity = entity.requestEntityProvider();
String url = "https://www.olx.pl/api/partner/adverts";
ResponseEntity<JsonNode> responseEntity = template.exchange(url, HttpMethod.GET, requestEntity, JsonNode.class);
String adverts = responseEntity.getBody().get("data").toString();
objectMapper.configure(DeserializationFeature.FAIL_ON_UNKNOWN_PROPERTIES, false);
try {
Advert[] array = objectMapper.readValue(adverts, Advert[].class);
for(Advert a : array) {
advertRepository.save(a);
}
} catch (Exception e) {
System.out.println(e);
}
return "index";
}
what I want to do is to parse json to entity objects and save all adverts objects to sql database in one table.
Method execution stops with exception on this line :
Advert[] array = objectMapper.readValue(adverts, Advert[].class);
I get this error message :
com.fasterxml.jackson.databind.exc.MismatchedInputException: Cannot deserialize value of type java.lang.String from Array value (token JsonToken.START_ARRAY)
at [Source: (StringReader); line: 24, column: 14] (through reference chain: java.lang.Object[][0]->pl.vida.model.Advert["images"])
Please notice that the field "images" realtes to nested array of json objects.
Please help, I spent one week on this and no result. Thanks
com.fasterxml.jackson.databind.exc.MismatchedInputException: Cannot deserialize value of type java.lang.String from Array value (token JsonToken.START_ARRAY) at [Source: (StringReader); line: 24, column: 14] (through reference chain: java.lang.Object[][0]->pl.vida.model.Advert["images"])
You are getting the above exception because you are trying to convert an array of objects into a String which is not possible. See in your JSON images & attributes are array of objects.
"images": [{ "url": "https://ireland.apollo.olxcdn.com:443/v1/files/qmvssagjnq1r2-PL/image;s=1000x700" }],
"attributes": [{ "code": "state", "value": "new", "values": null }]
and in your Advert class you have created images & attributes as String types.
private String attributes;
private String images;
Generally, for array kind of object, we make fields either List or Set and if the field is List/Set then we need to create separate classes for them and map as OneToMany relationship. So creating separate classes means separate tables will be created but you don't want to have multiple tables. You want to store all the data in a single table. In a normal case, it is not possible but if we write some additional configuration classes then we can achieve your requirement. These tweaks have been provided by Hibernate itself.
So basically, Hibernate has provided some built-in types like String, Integer, Float, Date, Timezone, etc. Here you can check the complete list of built-in types. But according to our requirements, we can create custom types as well. So to store the array kind of data Hibernate didn't provide any built-in type. Hence we shall create a custom type.
Solution:
So we want to store an array of object data and we can easily store it in com.fasterxml.jackson.databind.JsonNode object. But Hibernate doesn't support this class as a field type. Hence to make supportable for this class we need to write 2 extra classes i.e. JsonNodeStringType.java & JsonNodeStringDescriptor.
JsonNodeStringType.java
public class JsonNodeStringType extends AbstractSingleColumnStandardBasicType<JsonNode> implements DiscriminatorType<JsonNode> {
public static final JsonNodeStringType INSTANCE = new JsonNodeStringType();
public JsonNodeStringType() {
super(VarcharTypeDescriptor.INSTANCE, JsonNodeStringDescriptor.INSTANCE);
}
#Override
public String getName() {
return "JsonNode";
}
#Override
public JsonNode stringToObject(String xml) {
return fromString(xml);
}
#Override
public String objectToSQLString(JsonNode value, Dialect dialect) {
return '\'' + toString(value) + '\'';
}
}
JsonNodeStringDescriptor.java
public class JsonNodeStringDescriptor extends AbstractTypeDescriptor<JsonNode> {
public static final ObjectMapper mapper = new ObjectMapper();
public static final JsonNodeStringDescriptor INSTANCE = new JsonNodeStringDescriptor();
public JsonNodeStringDescriptor() {
super(JsonNode.class, ImmutableMutabilityPlan.INSTANCE);
}
#Override
public String toString(JsonNode value) {
try {
return mapper.writeValueAsString(value);
} catch (JsonProcessingException e) {
e.printStackTrace();
}
return null;
}
#Override
public JsonNode fromString(String string) {
try {
return mapper.readTree(string);
} catch (JsonProcessingException e) {
e.printStackTrace();
}
return null;
}
#Override
public <X> X unwrap(JsonNode value, Class<X> type, WrapperOptions options) {
if (value == null) {
return null;
}
if (String.class.isAssignableFrom(type)) {
return (X) toString(value);
}
throw unknownUnwrap(type);
}
#Override
public <X> JsonNode wrap(X value, WrapperOptions options) {
if (value == null) {
return null;
}
if (String.class.isInstance(value)) {
return fromString(value.toString());
}
throw unknownWrap(value.getClass());
}
}
Now our Advert class will look like as
import org.hibernate.annotations.Type;
#Setter
#Getter
#ToString
#Entity
#Table(name = "advert")
#TypeDef(name = "JsonNode", typeClass = JsonNodeStringType.class)
public class Advert {
#Id
#GeneratedValue(strategy = GenerationType.AUTO)
#Column(name = "ident", unique = true, nullable = false)
private Long ident;
private int id;
private String status;
private String url;
private String created_at;
private String activated_at;
private String valid_to;
private String title;
#Lob
private String description;
private int category_id;
private String advertiser_type;
private Long external_id;
private String external_url;
private String salary;
private String courier;
#Embedded
private Location location;
#Embedded
private Contact contact;
#Embedded
private Price price;
#Type(type = "JsonNode")
private JsonNode images;
#Type(type = "JsonNode")
private JsonNode attributes;
}
Here we go. If you execute the below code it works perfectly.
String advertsString = "[ { \"id\": 643419352, \"status\": \"removed_by_user\", \"url\": \"https://www.olx.pl/d/oferta/opona-12-1-2-x-2-1-4-etrto-62-203-detka-CID767-IDHxILu.html\", \"created_at\": \"2020-11-27 10:46:07\", \"activated_at\": \"2020-12-11 12:41:12\", \"valid_to\": \"2020-12-17 15:38:10\", \"title\": \"opona 12 1/2 \\\" x 2 1/4 etrto 62-203 + dętka\", \"description\": \"opona w bardzo dobrym stanie + dętka, rozmiar 12 1/2 x 2 1/4 , dętka z zaworem samochodowym\", \"category_id\": 1655, \"advertiser_type\": \"private\", \"external_id\": null, \"external_url\": null, \"contact\": { \"name\": \"Damazy\", \"phone\": \"501474399\" }, \"location\": { \"city_id\": 10609, \"district_id\": 301, \"latitude\": \"51.80178\", \"longitude\": \"19.43928\" }, \"images\": [ { \"url\": \"https://ireland.apollo.olxcdn.com:443/v1/files/efa9any4ryrb-PL/image;s=1000x700\" } ], \"price\": { \"value\": \"9\", \"currency\": \"PLN\", \"negotiable\": false, \"budget\": false, \"trade\": false }, \"salary\": null, \"attributes\": [ { \"code\": \"state\", \"value\": \"used\", \"values\": null } ], \"courier\": null }, { \"id\": 643435839, \"status\": \"removed_by_user\", \"url\": \"https://www.olx.pl/d/oferta/opona-4-80-4-00-8-do-taczki-nowa-CID628-IDHxN3p.html\", \"created_at\": \"2020-11-27 11:53:47\", \"activated_at\": \"2020-11-27 11:54:36\", \"valid_to\": \"2020-12-17 15:38:07\", \"title\": \"opona 4.80/4.00 - 8 do taczki nowa!!!\", \"description\": \"opona do taczki, nowa, nigdy nie używana, stan idealny.\\r\\nrozmiar 4.80/4.00-8. \\r\\nopona do taczki, nowa, nigdy nie używana, stan idealny.\\r\\nrozmiar 4.80/4.00-8.\", \"category_id\": 1636, \"advertiser_type\": \"private\", \"external_id\": null, \"external_url\": null, \"contact\": { \"name\": \"Damazy\", \"phone\": \"501474399\" }, \"location\": { \"city_id\": 10609, \"district_id\": 301, \"latitude\": \"51.80178\", \"longitude\": \"19.43928\" }, \"images\": [ { \"url\": \"https://ireland.apollo.olxcdn.com:443/v1/files/qmvssagjnq1r2-PL/image;s=1000x700\" } ], \"price\": { \"value\": \"9\", \"currency\": \"PLN\", \"negotiable\": false, \"budget\": false, \"trade\": false }, \"salary\": null, \"attributes\": [ { \"code\": \"state\", \"value\": \"new\", \"values\": null } ], \"courier\": null } ]";
ObjectMapper objectMapper = new ObjectMapper();
objectMapper.configure(DeserializationFeature.FAIL_ON_UNKNOWN_PROPERTIES, false);
Advert[] adverts = objectMapper.readValue(advertsString, Advert[].class);
for (Advert advert : adverts) {
Advert saved = advertRepository.save(advert);
System.out.println("saved " + saved.getIdent());
}
I hope your problem gets resolved which you have been stuck with for one week. If you don't want to manually create these types of descriptors you can follow this article to use as an external dependency.
I want to aggregate a collection of documents that match certain conditions, group them and map the output to a different class object. The aggregation works fine and I get the expected total but the _id field is always NULL.
I'm using spring-data-mongodb 2.1.11 and MongoDB 3.6.
This is the class to be aggregated:
#Document
public class LegOrder {
public static class Key {
#Indexed
long itemId;
long transactionId;
...
}
#Id
private Key id;
#Indexed
private long brandId;
private int units;
...
}
This is the aggregation output class:
#Document
public class ItemAggregation {
public static class Key {
#Indexed
long itemId;
#Indexed
long brandId;
}
#Id
private Key id;
private long total;
...
}
My aggregation method:
public ItemAggregation aggregate(long itemId, long brandId) {
MatchOperation matchStage = Aggregation.match(new Criteria().andOperator(
Criteria.where("id.itemId").is(itemId),
Criteria.where("brandId").is(brandId)
));
GroupOperation groupStage = Aggregation.group("id.itemId", "brandId")
.sum("units").as("total")
...
;
Aggregation aggregation = Aggregation.newAggregation(matchStage, groupStage);
return mongoTemplate.aggregate(aggregation, LegOrder.class, ItemAggregation.class).getUniqueMappedResult();
}
The executed query in MongoDB:
[
{
"$match": {
"$and": [
{ "_id.itemId": 1 },
{ "brandId": 2}
]
}
},
{
"$group": {
"_id": {
"itemId": "$_id.itemId",
"brandId": "$brandId"
},
"total": { "$sum": "$units" }
}
}
]
If I run this query in the mongo shell the _id field is properly populated.
Any idea how to achieve it?
Thank you
Sorry for the late response. I faced this issue now and found this solution.
My aggregation output in console is
{
"_id" : {
"ownerId" : BinData(3,"xkB0S9Wsktm+tSKBruv6og=="),
"groupbyF" : "height"
},
"docs" : [
{
"id" : ObjectId("5fe75026e211c50ef5741b31"),
"aDate" : ISODate("2020-12-26T15:00:51.056Z"),
"value" : "rrr"
}
]
}
{
"_id" : {
"ownerId" : BinData(3,"AAAAAAAAAAAAAAAAAAAAAA=="),
"groupbyF" : "weight"
},
"docs" : [
{
"id" : ObjectId("5fe6f702e211c50ef5741b2f"),
"aDate" : ISODate("2020-12-26T08:40:28.742Z"),
"value" : "55"
},
{
"id" : ObjectId("5fe6f6ade211c50ef5741b2e"),
"aDate" : ISODate("2020-12-26T08:38:58.098Z"),
"value" : "22"
}
]
}
The mapping that worked for me
import lombok.Data;
#Data
public class AggregationLatest2Type{
private String ownerId;
private String key;
private List<Doc> docs;
#Data
public class Doc{
private String _id;
private Date aDate;
private String value;
}
}
I need help in 2 things.
Creating a Spring mongoquery from a mongo query using spring mongo template.
Filter the result and get only lower level objects from a document which has a json tree structure.
I only want to get a list of matching merchant_shops by passing
community_id,list of package_type_id and category_id.
I am using mongodb version 4.
Thank you in advance.
Document:
{
"_id" : ObjectId("5bdafff206ba681e30895e50"),
"community_id" : "09a5c059-33b3-4e47-9905-709f5c244580",
"package_types" : [
{
"package_type_id" : 1,
"categories" : [
{
"category_id" : "5bd0433d8ac2ce275082ff1f",
"subcategories" : [
{
"subcategory_id" : "5bd0436d8ac2ce275082ff20",
"merchant_shops" : [
{
"shop_id" : "5bd19a704ab492427d7326a0",
"distance" : 14.2841995007249
},
{
"shop_id" : "5bd84aed4ab4926fb2060e76",
"distance" : 14.283313487973
},
{
"shop_id" : "5bdc14ad4ab492123f29c6a7",
"distance" : 14.2829648551977
},
{
"shop_id" : "5be9555e4ab4923a01c7b7f8",
"distance" : 11.8215058435006
},
{
"shop_id" : "5be95a974ab4923a01c7b7fc",
"distance" : 11.8081739265758
}
]
}
]
}
]
},
{
"package_type_id" : 3,
"categories" : [
{
"category_id" : "5bd0433d8ac2ce275082ff1f",
"subcategories" : [
{
"subcategory_id" : "5bd0436d8ac2ce275082ff20",
"merchant_shops" : [
{
"shop_id" : "5bd84aed4ab4926fb2060e76",
"distance" : 14.283313487973
}
]
}
]
}
]
}
]
}
Query that works fine in mongodb:
db.getCollection('community_shop').aggregate([
{ $match: {"community_id": "09a5c059-33b3-4e47-9905-709f5c244580"} },
{ $project:
{
package_types:
{$filter: {
input: '$package_types',
as: 'item',
cond: {
$or: [
{
$in: ['$$item.package_type_id', [1,3]]
},{
$eq: ['$$item.categories.category_id', "5bd0433d8ac2ce275082ff1f"]
}
]
}
}}
}
}
])
Java class is:
#Document(collection = "community_shop")
public class CommunityShopDOB {
#Id
private String id;
#Field("community_id")
private String communityId;
#Field("package_types")
private List<PackageType> packageTypes;
public static class PackageType {
#Field("package_type_id")
private Integer packageTypeId;
#Field("categories")
private List<Category> categories;
public static class Category {
#Field("category_id")
private String categoryId;
#Field("subcategories")
private List<Subcategory> subcategories;
public static class Subcategory {
#Field("subcategory_id")
private String subcategoryId;
#Field("merchant_shops")
private List<MerchantShop> merchantShops;
public static class MerchantShop {
#Field("shop_id")
private String shopId;
#Field("distance")
private Double distance;
#Field("filter_value_ids")
private List<String> filterValueIds;
}
}
}
}
My solution so far is and it is incorrect:
Criteria cr = Criteria.where("communityId").is(society_id);
MatchOperation filterStates = Aggregation.match(cr);
Aggregation aggregation = newAggregation(filterStates, project("packageTypes")
.and(filter("packageTypes").as("item").by(valueOf("item.packageTypeId").equalToValue(1))
).as("packageTypes"));
AggregationResults<CommunityShopDOB> output = mongoTemplate.aggregate(aggregation, CommunityShopDOB.class,
CommunityShopDOB.class);
return output.getUniqueMappedResult();
I have read several tutorial about one-to-one relation mapping forexample:
https://docs.oracle.com/javaee/6/api/javax/persistence/OneToOne.html,
http://websystique.com/hibernate/hibernate-one-to-one-unidirectional-with-foreign-key-associations-annotation-example/
https://en.wikibooks.org/wiki/Java_Persistence/OneToOne
I beleive I follow these tutorials, however my relational mapping still not works as expected. I have the following classes:
#Entity(name = "lesson")
public class Lesson {
#Id
#Type(type = "pg-uuid")
private UUID uid;
private String start_date_time;
private String end_date_time;
private String location;
#OneToOne
#JoinColumn(name="uid") //uid is the name of the Id i want to reference to in the subject class
private Subject subject_uid; // subject_uid is the name of the column in my subject table
public Lesson(UUID uid, String start_date_time, String end_date_time, String location, Subject subject_uid) {
this.uid = uid;
this.start_date_time = start_date_time;
this.end_date_time = end_date_time;
this.location = location;
this.subject_uid = subject_uid;
}
//getters setters
#Entity(name = "subject")
public class Subject {
#Id
#Type(type = "pg-uuid")
private UUID uid;
private String name;
private String start_date;
private String end_date;
private boolean is_lesson_created;
public Subject(UUID uid, String name, String start_date, String end_date, boolean is_lesson_created) {
this.uid = uid;
this.name = name;
this.start_date = start_date;
this.end_date = end_date;
this.is_lesson_created = is_lesson_created;
}
The response what the Spring Data Rest creates on /lessons endpoint looks the following:
{
"_embedded" : {
"lessons" : [ {
"start_date_time" : "2017-01-08 08:30:00",
"end_date_time" : "2017-01-08 10:15:00",
"location" : "A101 ",
"_links" : {
"self" : {
"href" : "http://localhost:3400/lessons/78038aeb-cdc9-4673-990e-36b8c1105500"
},
"lesson" : {
"href" : "http://localhost:3400/lessons/78038aeb-cdc9-4673-990e-36b8c1105500"
},
"subject_uid" : {
"href" : "http://localhost:3400/lessons/78038aeb-cdc9-4673-990e-36b8c1105500/subject_uid"
}
}
} ]
},
"_links" : {
"self" : {
"href" : "http://localhost:3400/lessons{?page,size,sort}",
"templated" : true
},
"profile" : {
"href" : "http://localhost:3400/profile/lessons"
}
},
"page" : {
"size" : 20,
"totalElements" : 1,
"totalPages" : 1,
"number" : 0
}
}
When I want access the http://localhost:3400/lessons/78038aeb-cdc9-4673-990e-36b8c1105500/subject_uidlink I get a 404.
Is the UUID type effects my mapping? What should I change to be able to access my student_uid?
Finally I found out the problem, which is something I haven't read anywhere. When a one-to-one join has to be done, JPA provides the default name of the join as the table name underscore id(subject_id). In my case, I have a tablename called "subject" in the database and the PK called simply "uid". So what you have to do is append the table name with the name of the attribute, which to join to:
#OneToOne
#JoinColumn(name="subject_uid")//the pattern is: "tablename_joined attribute"
private Subject subject_uid;
Given Person.java:
public class Person implements Serializable {
private static final long serialVersionUID = 1L;
private String lastName;
private String firstName;
private String middleName;
// ...
public String getLastName() {
return lastName;
}
public String getFirstName() {
return firstName;
}
public String getMiddleName() {
return middleName;
}
}
and Persons.java:
public class Persons implements Serializable {
private static final long serialVersionUID = 1L;
private final List<Person> persons;
public Persons(List<Person> persons) {
this.persons = persons;
}
public List<Person> getPersons() {
return persons;
}
}
When returning a JSON response for Persons, the "persons" element is repeated:
{
"persons" : {
"persons" : [ {
"lastName" : "McCartney",
"firstName" : "James",
"middleName" : "Paul"
}, {
"lastName" : "Lennon",
"firstName" : "John",
"middleName" : "Winston"
}, {
"lastName" : "Starkey",
"firstName" : "Richard",
"middleName" : null
}, {
"lastName" : "Harrison",
"firstName" : "George",
"middleName" : null
} ]
}
}
How do I remove the extra element?
The reason is probably because you have an model attribute named persons which you are returning for MappingJackson2JsonView to convert to json:
model.addAttribute("persons", personsType);
There are two good fixes that I can think of:
To use #ResponseBody annotated controller methods instead, this way you can return persons and MappingJackson2HttpMessageConverter would convert your type cleanly to json
If you want to continue with your approach, you can customize MappingJackson2JsonViewwith an additional flag to indicate that it has to extract value from the model before serializing to json - see here