Prometheus & Grafana dashboard setup

 Resorce to setup Prometheus:

--------------------

https://itnext.io/using-prometheus-in-azure-kubernetes-service-aks-ae22cada8dd9

To open Grafana dashboard: (userid/password : admin/admin)

---------------------------

kubectl --namespace monitoring port-forward $(kubectl get pod --namespace monitoring -l app=kube-prometheus-grafana -o template --template "{{(index .items 0).metadata.name}}") 3000:3000

(OR)

kubectl get pod --namespace monitoring -l app=kube-prometheus-grafana

kubectl port-forward -n monitoring kube-prometheus-grafana-8669b7f999-xnq7q  3000:3000

http://localhost:3000/

To open Prometheus dashboard: (Prometheus a datasource, Grafana is used to show it in UI)

------------------------------

kubectl --namespace monitoring port-forward $(kubectl get pod --namespace monitoring -l prometheus=kube-prometheus -l app=prometheus -o template --template "{{(index.items 0).metadata.name}}") 9090:9090

Create keys - Public and Private - OpenSSL

 kubectl create secret tls ambassador-certs --cert=emea_com.pem --key=emea_com.key

Private key:

------------

openssl pkcs12 -in emea_com.pfx -nocerts -nodes -out emea_com1.key

Public key:

------------

openssl pkcs12 -in emea_com.pfx -nokeys -out emea_com1.pem

chmod 664 emea_com.key emea_com.pem

Terms - words

superficial

transient

GIT force commit

 git clone <repo url>

git status

if any changes u see do - git stash

git fetch --all

git branch -a  (in this u should see qa2.0 branch)

git check out qa2.0

git reset --hard origin/dev2.0

git status

git gui - do force commit here

Kafka Connect basic - including setup and run

 C:\kafka_2.13-2.8.0\bin\windows>zookeeper-server-start.bat C:\kafka_2.13-2.8.0\config\zookeeper.properties

C:\kafka_2.13-2.8.0\bin\windows>kafka-server-start.bat C:\kafka_2.13-2.8.0\config\server.properties

C:\mysql-8.0.25-winx64\bin>mysqld --console

C:\mysql-8.0.25-winx64\bin>mysql --user=root --password=t3r6e*OjnQRa

C:\kafka_2.13-2.8.0\bin\windows>kafka-topics.bat --create --zookeeper localhost:2181 --replication-factor 1 --partitions 1 --topic connect-test

ALTER USER 'root'@'localhost' IDENTIFIED BY 'password123';

to find port : netstat -ano | findstr :8080

to kill port  : taskkill /PID <PROCESSID> /F

Supported Jar files

====================

Create a folder kafka-connect-jdbc in C:\kafka_2.13-2.8.0\confluent-6.1.1\share\java, and copy the below 2 jars into  kafka-connect-jdbc

https://dev.mysql.com/downloads/connector/j/     ===> select 'Platform Independent' option and download the jar

https://www.confluent.io/hub/confluentinc/kafka-connect-jdbc ==> Download this jar

To know the number of messages exists in a topic:

==================================================

C:\kafka_2.13-2.8.0\bin\windows>kafka-run-class kafka.tools.GetOffsetShell --broker-list localhost:9092 --topic myTopic

To view the Kafka cluster details

==================================

download kafdrop github project and build the project.

java -jar kafdrop-2.1.0.jar --zookeeper.connect=localhost:2181 --kafka.brokers=localhost:9092

Open the browser and launc the URL as mentioned on kafdrop github

connect-standalone.properties

================================

bootstrap.servers=localhost:9092

key.converter=org.apache.kafka.connect.json.JsonConverter

value.converter=org.apache.kafka.connect.json.JsonConverter

key.converter.schemas.enable=true

value.converter.schemas.enable=true

offset.storage.file.filename=C:/kafka_2.13-2.8.0/kafkaconnect-Standalone-offset/connect.offsets

offset.flush.interval.ms=10000

plugin.path=C:/kafka_2.13-2.8.0/confluent-6.1.1/share/java

connect-file-source.properties

===============================

name=local-file-source

connector.class=FileStreamSource

tasks.max=3

file=C:/kafka_2.13-2.8.0/testFileStreamData/test.txt

topic=connect-test

connect-jdbc-source.properties

===============================

name=test-source-mysql-jdbc-kafka-connect

connector.class=JdbcSourceConnector

tasks.max=1

connection.url=jdbc:mysql://localhost:3306/kafka_database?user=root&password=password123

mode=incrementing

incrementing.column.name=id

catalog.pattern=kafka_database

table.whitelist=user_entity

topic.prefix=connect-test

C:\kafka_2.13-2.8.0\bin\windows>connect-standalone.bat C:\kafka_2.13-2.8.0\confluent-6.1.1\etc\kafka\connect-standalone.properties C:\kafka_2.13-2.8.0\confluent-6.1.1\etc\kafka\connect-file-source.properties

C:\kafka_2.13-2.8.0\bin\windows>connect-standalone.bat C:\kafka_2.13-2.8.0\confluent-6.1.1\etc\kafka\connect-standalone.properties C:\kafka_2.13-2.8.0\confluent-6.1.1\etc\kafka-connect-jdbc\connect-jdbc-source.properties

Basic estimation factors

Efforts includes : 

-------------------

TRQ Evaluate

LLD Preparation

CUT (Code and Unit Testing)

CodeReview

Rework

Run SonarQube & Fix

External Code Review

External Rework

Integration Dev Test Rework

Integration Testing  with external systems

SAST/DAST (Critical/High/Medium)

DevOps (Release)

To export jar file to an Artifacts repo (maven command/settings.xml)

 mvn deploy:deploy-file -DgroupId=com.pom.group.id -DartifactId=POMArtifactid -Dversion=2.1.9 -Dpackaging="jar" -Durl=https://artifacts-url/maven/v1 -Dfile="JarfileToBeExported-2.1.9.jar" -DrepositoryId=reposerver-id

To use this jar file:

<dependency>
  <groupId>com.pom.group.id</groupId>
  <artifactId>POMArtifactid</artifactId>
  <version>2.1.9</version>
</dependency>

Settings.xml:

<?xml version="1.0" encoding="UTF-8"?>

<settings xmlns="http://maven.apache.org/SETTINGS/1.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://maven.apache.org/SETTINGS/1.0.0                           https://maven.apache.org/xsd/settings-1.0.0.xsd">
    <localRepository/>
   <interactiveMode />
   <usePluginRegistry />
   <offline />
   <pluginGroups />
   <servers>
      <server>
         <id>reposerver-id</id>
         <username>username</username>
         <password>password</password>
      </server>
   </servers>
   <profiles>
     
    <profile>
      <id>distributionManagement</id>
      <repositories>
        <repository>
          <id>reposerver-id</id>
          <url>https://artifacts-url/maven/v1</url>
        </repository>
      </repositories>
    </profile>
  </profiles>
	 
   
   <activeProfiles>
      <activeProfile>distributionManagement</activeProfile>
   </activeProfiles>
   <mirrors />
   <proxies />  
   
</settings>