AppEngine remote API keeps returning 302 - java

I need to download a datastore from my appengine application. The application itself is written in JAVA and I've already activated remote API according to this instruction. Then I run appcfg.py and it asks me for log in details and I get the following result:
06:18 PM Downloading data records.
[INFO ] Logging to bulkloader-log-20131101.181851
[INFO ] Throttling transfers:
[INFO ] Bandwidth: 250000 bytes/second
[INFO ] HTTP connections: 8/second
[INFO ] Entities inserted/fetched/modified: 20/second
[INFO ] Batch Size: 10
Please enter login credentials for [_app_address_]
Email: [_My_Google_Mail_]
Password for [_My_Google_Mail_]:
Error 302: --- begin server output ---
--- end server output ---
Obviously I've hidden app and my mail but they are correct (I use the same credentials when use appcfg.sh to deploy new version).
If it helps the app configuration is:
Authentication Type: (Experimental) Federated Login
Datastore Replication Options: High Replication
What can I do about it?? I really need to get local copy of the production data...

Huh. The problem was in Note: Bulk upload and download is not supported for apps that use Federated (OpenID) authentication
I made a workaround - maybe someone finds it useful. To download data I've created a second app with google account authentication type and uploaded simple python app from this instruction. This way I was able to move entities from my source app to target and then download it using simple appcfg.py call...

The parameter `--url` needs to start with `https`
e.g.
`appcfg.py upload_data --config_file=bulkloader.yaml --url=https://<APP_ID>.appspot.com/remote_api --kind=<ENTITY> --filename=<FILE_NAME>.csv`

Related

How to set up OIDC connection to Keycloak in Quarkus on Kubernetes

Has somebody succeeded in setting up OIDC connection to Keycloack in a Quarkus app deployed in a Kubernetes cluster ?
Could you clarify how does the connection-delay (and other related parameters) work ?
(Here is the documentation I tried to follow)
In our env (Quarkus 1.13.3.Final, Keycloak 12.0.4) we have such config:
quarkus.oidc.connection-delay: 6M
quarkus.oidc.connection-timeout: 30S
quarkus.oidc.tenant-id: testTenant-01
And these msgs appear in pod's log when it's being started:
2021-07-26 14:44:22,523 INFO [main] [OidcRecorder.java:264] -
Connecting to IDP for up to 180 times every 2 seconds
2021-07-26
14:44:24,142 DEBUG [vert.x-eventloop-thread-1]
[OidcRecorder.java:115] - 'testTenant-01' tenant initialization has failed:
'OpenId Connect Provider configuration metadata is not configured and
can not be discovered'. Access to resources protected by this tenant
will fail with HTTP 401.
(... following log comes later as the pod is running ...)
2021-07-27 06:11:54,261 DEBUG [vert.x-eventloop-thread-0]
[DefaultTenantConfigResolver.java:112] - Tenant 'null' is not
initialized
2021-07-27 06:11:54,262 ERROR
[vert.x-eventloop-thread-0] [QuarkusErrorHandler.java:101] - HTTP
Request to /q/health/live failed, error id:
89f83d1d-894c-4fed-9995-0d42d60cec17-2: io.quarkus.oidc.OIDCException:
Tenant configuration has not been resolved at
io.quarkus.oidc.runtime.OidcAuthenticationMechanism.resolve(OidcAuthenticationMechanism.java:61)
at
io.quarkus.oidc.runtime.OidcAuthenticationMechanism.authenticate(OidcAuthenticationMechanism.java:40)
at
io.quarkus.oidc.runtime.OidcAuthenticationMechanism_ClientProxy.authenticate(OidcAuthenticationMechanism_ClientProxy.zig:189)
at
io.quarkus.vertx.http.runtime.security.HttpAuthenticator.attemptAuthentication(HttpAuthenticator.java:100)
at
io.quarkus.vertx.http.runtime.security.HttpAuthenticator_ClientProxy.attemptAuthentication(HttpAuthenticator_ClientProxy.zig:157)
at
io.quarkus.vertx.http.runtime.security.HttpSecurityRecorder$2.handle(HttpSecurityRecorder.java:101)
at
io.quarkus.vertx.http.runtime.security.HttpSecurityRecorder$2.handle(HttpSecurityRecorder.java:51)
at
io.vertx.ext.web.impl.RouteState.handleContext(RouteState.java:1038)
Questions:
Any way how to find out what metadata are missing ?
Can I somehow change the 2s period between connection attempts ?
Any relation between connection-delay and connection-timeout ?
It failed after cca 2s - does it mean that it fails immediately in the 1st attempt, or has it finished 180 attempts so fast ?
Does DefaultTenantConfigResolver get tenant from different resource than OidcRecorder in initialization, i.e. should tenant be configured at multiple places ?
Finally made it work. Caused by incorrect auth-server-url which is not clear at all from the log messages.
quarkus.oidc.client-id: my-app
quarkus.oidc.enabled: true
quarkus.oidc.connection-delay: 6M
quarkus.oidc.connection-timeout: 30S
quarkus.oidc.tenant-id: testTenant-01
quarkus.oidc.auth-server-url: ${keycloak.url}/auth/realms/${quarkus.oidc.tenant-id}
The URL format is emphasized in Quarkus doc: Note if you work with Keycloak OIDC server, make sure the base URL is in the following format: https://host:port/auth/realms/{realm} where {realm} has to be replaced by the name of the Keycloak realm

Accessing Neo4j/GrapheneDB (Dev free plan) on Heroku from Micronaut Java app fails: Connection to database terminated

currently I'm struggling with Neo4j/GrapheneDB (Dev free plan) on Heroku platform.
Launching my app locally via "heroku local" works fine, it connects (Neo4j Java Driver 4) to a Neo4j 3.5.18 (running from Docker image "neo4j:3.5").
My app is built using Micronaut framework, using its Neo4j support. Launching my app on Heroku platform succeeds, I'm using Gradle Heroku plugin for this task.
But accessing the database with business operations (and health checks) fails with exception like this:
INFO Driver - Direct driver instance 1523082263 created for server address hobby-[...]ldel.dbs.graphenedb.com:24787
WARN RetryLogic - Transaction failed and will be retried in 1032ms
org.neo4j.driver.exceptions.ServiceUnavailableException: Connection to the database terminated. Please ensure that your database is listening on the correct host and port and that you have compatible encryption settings both on Neo4j server and driver. Note that the default encryption setting has changed in Neo4j 4.0.
at org.neo4j.driver.internal.util.Futures.blockingGet(Futures.java:143)
at org.neo4j.driver.internal.InternalSession.beginTransaction(InternalSession.java:163)
at org.neo4j.driver.internal.InternalSession.lambda$transaction$4(InternalSession.java:147)
at org.neo4j.driver.internal.retry.ExponentialBackoffRetryLogic.retry(ExponentialBackoffRetryLogic.java:101)
at org.neo4j.driver.internal.InternalSession.transaction(InternalSession.java:146)
at org.neo4j.driver.internal.InternalSession.readTransaction(InternalSession.java:112)
at org.neo4j.driver.internal.InternalSession.readTransaction(InternalSession.java:106)
at PersonController.logInfoOf(PersonController.java:57)
at PersonController.<init>(PersonController.java:50)
at $PersonControllerDefinition.build(Unknown Source)
at io.micronaut.context.DefaultBeanContext.doCreateBean(DefaultBeanContext.java:1814)
[...]
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
at java.base/java.lang.Thread.run(Thread.java:832)
Suppressed: org.neo4j.driver.internal.util.ErrorUtil$InternalExceptionCause: null
at org.neo4j.driver.internal.util.ErrorUtil.newConnectionTerminatedError(ErrorUtil.java:52)
at org.neo4j.driver.internal.async.connection.HandshakeHandler.channelInactive(HandshakeHandler.java:81)
[...]
at org.neo4j.driver.internal.shaded.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
at org.neo4j.driver.internal.shaded.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
... 1 common frames omitted
I'm sure to get login credentials from OS environment variables GRAPHENEDB_BOLT_URL, GRAPHENEDB_BOLT_USER, and GRAPHENEDB_BOLT_PASSWORD injected to the app correctly; I've verified it with some debug log statements:
State changed from starting to up
INFO io.micronaut.runtime.Micronaut - Startup completed in 2360ms. Server Running: http://localhost:7382
INFO Application - Neo4j Bolt URIs: [bolt://hobby-[...]ldel.dbs.graphenedb.com:24787]
INFO Application - Neo4j Bolt encrypted? false
INFO Application - Neo4j Bolt trust strategy: TRUST_SYSTEM_CA_SIGNED_CERTIFICATES
INFO Application - Changed trust strategy to: TRUST_ALL_CERTIFICATES
INFO Application - Env.: GRAPHENEDB_BOLT_URL='bolt://hobby-[...]ldel.dbs.graphenedb.com:24787'
INFO Application - Env.: GRAPHENEDB_BOLT_USER='app1[...]hdai'
INFO Application - Env.: GRAPHENEDB_BOLT_PASSWORD of length 31
I've also tried restarting GrapheneDB instance via Heroku plugin website, but with same negative results.
What's going wrong here? Are there any ways to further nail down the root cause?
Thanks
Christian
I had a closer look at this and it seems that you need the driver encryption turned on for the Graphene db instances. This can be configured in application.yml as below:
neo4j:
encryption: true
For reference, here is a sample project https://github.com/aldrinm/micronaut-neo4j-heroku

Get the service IPs ports running under mesospehere dcos through program or command

I want to connect to Cassandra which is running as a service in Mesosphere DC/OS. Is there any programmatic way(any api exposed) to get the Cassandra port and ip details ?
From Command I can connect like below... same thing I want to do it programmatically may be with REST api... or Java client to connect to these services.
As per this doc https://github.com/mesosphere/dcos-cassandra-service/blob/master/docs/connecting-clients.md
$ dcos cassandra --name=<service-name> connection
{
"address": [
"10.0.0.47:9042",
"10.0.0.50:9042",
"10.0.0.49:9042"
],
"dns": [
"node-0.cassandra.mesos:9042",
"node-1.cassandra.mesos:9042",
"node-2.cassandra.mesos:9042"
]
}
Note : The reason for doing this is every time cassandra IP and port changing. I have to manually adjust in my property file to get the latest details. If its through program its very easy to set the property with out manual interaction.
If you use the entries from the DNS section (shown below) they would not change even if the task relocated to another node
"node-0.cassandra.mesos:9042",
"node-1.cassandra.mesos:9042",
"node-2.cassandra.mesos:9042"

How to define jwt configuration in jhipster?

I have two microservices, The first one is a web API, the another is a gateway.
I've already started both, but I cannot authenticate with no user. Neither admin or user.
When I try to log in, I receive this message on API server.
InsufficientAuthenticationException: Full authentication is required
to access this resource 2018-02-23 13:05:31.373 WARN 7144 --- [
XNIO-2 task-3] o.z.p.spring.web.advice.AdviceTrait :
Unauthorized: Full authentication is required to access this resource
Both microservices are with the default configuration of jhipster .
The gateway was created with --skip-server option.
I believe the problem is the server_api_url because all the calls to api is directed to port 9000 but the api is running on 8081.
I got this from Chrome Dev tools:
Request URL:http://localhost:9000/api/profile-info?cacheBuster=1519408025933
The :9000 is wrong the port must be 8081.
Where can I change that ?.
I have already change the webpack.common.js
SERVER_API_URL: 'http://localhost:8081/'
But nothing work.

SpringBoot app running on AWS-EC2 unable to connect to MySQL AWS-RDS database

I am having problems running an app I have developed in an EC2 instance. When I execute the .jar (java -jar app.jar), the SpringBoot app starts but it fails when trying to connect to my MySQL RDS database. The thing is when I run the app locally on my machine, It has no issues with the DB connection.
I have opened the port where the app is running (8090) and MySql port as well (3306) for inbound and outbound traffic:
This is the error I get:
2016-09-23 17:46:38.132 INFO 10161 --- [main] .t.TomcatEmbeddedServletContainerFactory : Server initialized with port: 8090
2016-09-23 17:46:38.604 INFO 10161 --- [main] o.apache.catalina.core.StandardService : Starting service Tomcat
2016-09-23 17:46:38.605 INFO 10161 --- [main] org.apache.catalina.core.StandardEngine : Starting Servlet Engine: Apache Tomcat/7.0.54
2016-09-23 17:46:38.724 INFO 10161 --- [ost startStop 1] o.a.c.c.C.[Tomcat].[localhost].[/] : Initializing Spring embedded WebApplicationContext
2016-09-23 17:46:38.725 INFO 10161 --- [ost startStop 1] o.s.web.context.ContextLoader: Root WebApplicationContext: initialization completed in 5028 ms
2016-09-23 17:48:48.476 ERROR 10161 --- [ost startStop 1] o.a.tomcat.jdbc.pool.ConnectionPool: Unable to create initial connections of pool.
com.mysql.jdbc.exceptions.jdbc4.CommunicationsException: Communications link failure
The last packet sent successfully to the server was 0 milliseconds ago. The driver has not received any packets from the server.
Any ideas how can i solve this problem?
Thank you very much for your help
Regards
Andres
From your description and log file, it's likely that network configuration is the cause here.
You might want to draw the network topology of your instances (region/availability zone, VPC, subnet, network acl, security group). This will be very helpful when you do more complex development work.
There are good references: VPC Introduction and Security in your VPC and Scenarios for Accessing a DB Instance in a VPC
I suggest the following actions for your troubleshooting:
Check security group (SG) configuration of your EC2 instance and RDS instance.
You can check this by going to EC2 Dashboard/RDS Dashboard -> Click on an instance and look at "Security Group" description, or you can click on the Setting icon (Show/Hide columns) and tick "Security Groups".
In RDS's SG configuration: make sure you have enable access from EC2 instance's SG to port 3306. You can do this by putting EC2 instance's SG ID into Source field of the config, as a "Custom IP" value. See the 1st scenario in the above reference for more detail.
Use mysql command line to test the connection between EC2 instance and RDS.
Hope it helps.
You need to perform following steps :
1) Go to EC2 instance and find security group you want access in RDS
2) Now go to your RDS security group and select inbound rules
Select ALL TCP and add your sg-xxx(security group)
https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/USER_VPC.Scenarios.html

Categories