Java Spring AWS: 06/15/21

Tuesday, 15 June 2021

Prometheus & Grafana dashboard setup

Resorce to setup Prometheus:

--------------------

https://itnext.io/using-prometheus-in-azure-kubernetes-service-aks-ae22cada8dd9

To open Grafana dashboard: (userid/password : admin/admin)

---------------------------

kubectl --namespace monitoring port-forward $(kubectl get pod --namespace monitoring -l app=kube-prometheus-grafana -o template --template "{{(index .items 0).metadata.name}}") 3000:3000

(OR)

kubectl get pod --namespace monitoring -l app=kube-prometheus-grafana

kubectl port-forward -n monitoring kube-prometheus-grafana-8669b7f999-xnq7q 3000:3000

http://localhost:3000/

To open Prometheus dashboard: (Prometheus a datasource, Grafana is used to show it in UI)

------------------------------

kubectl --namespace monitoring port-forward $(kubectl get pod --namespace monitoring -l prometheus=kube-prometheus -l app=prometheus -o template --template "{{(index.items 0).metadata.name}}") 9090:9090

Create keys - Public and Private - OpenSSL

kubectl create secret tls ambassador-certs --cert=emea_com.pem --key=emea_com.key

Private key:

------------

openssl pkcs12 -in emea_com.pfx -nocerts -nodes -out emea_com1.key

Public key:

------------

openssl pkcs12 -in emea_com.pfx -nokeys -out emea_com1.pem

chmod 664 emea_com.key emea_com.pem

Terms - words

superficial

transient

GIT force commit

git clone <repo url>

git status

if any changes u see do - git stash

git fetch --all

git branch -a (in this u should see qa2.0 branch)

git check out qa2.0

git reset --hard origin/dev2.0

git status

git gui - do force commit here

Kafka Connect basic - including setup and run

C:\kafka_2.13-2.8.0\bin\windows>zookeeper-server-start.bat C:\kafka_2.13-2.8.0\config\zookeeper.properties

C:\kafka_2.13-2.8.0\bin\windows>kafka-server-start.bat C:\kafka_2.13-2.8.0\config\server.properties

C:\mysql-8.0.25-winx64\bin>mysqld --console

C:\mysql-8.0.25-winx64\bin>mysql --user=root --password=t3r6e*OjnQRa

C:\kafka_2.13-2.8.0\bin\windows>kafka-topics.bat --create --zookeeper localhost:2181 --replication-factor 1 --partitions 1 --topic connect-test

ALTER USER 'root'@'localhost' IDENTIFIED BY 'password123';

to find port : netstat -ano | findstr :8080

to kill port : taskkill /PID <PROCESSID> /F

Supported Jar files

====================

Create a folder kafka-connect-jdbc in C:\kafka_2.13-2.8.0\confluent-6.1.1\share\java, and copy the below 2 jars into kafka-connect-jdbc

https://dev.mysql.com/downloads/connector/j/ ===> select 'Platform Independent' option and download the jar

https://www.confluent.io/hub/confluentinc/kafka-connect-jdbc ==> Download this jar

To know the number of messages exists in a topic:

==================================================

C:\kafka_2.13-2.8.0\bin\windows>kafka-run-class kafka.tools.GetOffsetShell --broker-list localhost:9092 --topic myTopic

To view the Kafka cluster details

==================================

download kafdrop github project and build the project.

java -jar kafdrop-2.1.0.jar --zookeeper.connect=localhost:2181 --kafka.brokers=localhost:9092

Open the browser and launc the URL as mentioned on kafdrop github

connect-standalone.properties

================================

bootstrap.servers=localhost:9092

key.converter=org.apache.kafka.connect.json.JsonConverter

value.converter=org.apache.kafka.connect.json.JsonConverter

key.converter.schemas.enable=true

value.converter.schemas.enable=true

offset.storage.file.filename=C:/kafka_2.13-2.8.0/kafkaconnect-Standalone-offset/connect.offsets

offset.flush.interval.ms=10000

plugin.path=C:/kafka_2.13-2.8.0/confluent-6.1.1/share/java

connect-file-source.properties

===============================

name=local-file-source

connector.class=FileStreamSource

tasks.max=3

file=C:/kafka_2.13-2.8.0/testFileStreamData/test.txt

topic=connect-test

connect-jdbc-source.properties

===============================

name=test-source-mysql-jdbc-kafka-connect

connector.class=JdbcSourceConnector

tasks.max=1

connection.url=jdbc:mysql://localhost:3306/kafka_database?user=root&password=password123

mode=incrementing

incrementing.column.name=id

catalog.pattern=kafka_database

table.whitelist=user_entity

topic.prefix=connect-test

C:\kafka_2.13-2.8.0\bin\windows>connect-standalone.bat C:\kafka_2.13-2.8.0\confluent-6.1.1\etc\kafka\connect-standalone.properties C:\kafka_2.13-2.8.0\confluent-6.1.1\etc\kafka\connect-file-source.properties