Kafka Connect basic - including setup and run

 C:\kafka_2.13-2.8.0\bin\windows>zookeeper-server-start.bat C:\kafka_2.13-2.8.0\config\zookeeper.properties

C:\kafka_2.13-2.8.0\bin\windows>kafka-server-start.bat C:\kafka_2.13-2.8.0\config\server.properties

C:\mysql-8.0.25-winx64\bin>mysqld --console

C:\mysql-8.0.25-winx64\bin>mysql --user=root --password=t3r6e*OjnQRa

C:\kafka_2.13-2.8.0\bin\windows>kafka-topics.bat --create --zookeeper localhost:2181 --replication-factor 1 --partitions 1 --topic connect-test

ALTER USER 'root'@'localhost' IDENTIFIED BY 'password123';

to find port : netstat -ano | findstr :8080

to kill port  : taskkill /PID <PROCESSID> /F

Supported Jar files

====================

Create a folder kafka-connect-jdbc in C:\kafka_2.13-2.8.0\confluent-6.1.1\share\java, and copy the below 2 jars into  kafka-connect-jdbc

https://dev.mysql.com/downloads/connector/j/     ===> select 'Platform Independent' option and download the jar

https://www.confluent.io/hub/confluentinc/kafka-connect-jdbc ==> Download this jar

To know the number of messages exists in a topic:

==================================================

C:\kafka_2.13-2.8.0\bin\windows>kafka-run-class kafka.tools.GetOffsetShell --broker-list localhost:9092 --topic myTopic

To view the Kafka cluster details

==================================

download kafdrop github project and build the project.

java -jar kafdrop-2.1.0.jar --zookeeper.connect=localhost:2181 --kafka.brokers=localhost:9092

Open the browser and launc the URL as mentioned on kafdrop github

connect-standalone.properties

================================

bootstrap.servers=localhost:9092

key.converter=org.apache.kafka.connect.json.JsonConverter

value.converter=org.apache.kafka.connect.json.JsonConverter

key.converter.schemas.enable=true

value.converter.schemas.enable=true

offset.storage.file.filename=C:/kafka_2.13-2.8.0/kafkaconnect-Standalone-offset/connect.offsets

offset.flush.interval.ms=10000

plugin.path=C:/kafka_2.13-2.8.0/confluent-6.1.1/share/java

connect-file-source.properties

===============================

name=local-file-source

connector.class=FileStreamSource

tasks.max=3

file=C:/kafka_2.13-2.8.0/testFileStreamData/test.txt

topic=connect-test

connect-jdbc-source.properties

===============================

name=test-source-mysql-jdbc-kafka-connect

connector.class=JdbcSourceConnector

tasks.max=1

connection.url=jdbc:mysql://localhost:3306/kafka_database?user=root&password=password123

mode=incrementing

incrementing.column.name=id

catalog.pattern=kafka_database

table.whitelist=user_entity

topic.prefix=connect-test

C:\kafka_2.13-2.8.0\bin\windows>connect-standalone.bat C:\kafka_2.13-2.8.0\confluent-6.1.1\etc\kafka\connect-standalone.properties C:\kafka_2.13-2.8.0\confluent-6.1.1\etc\kafka\connect-file-source.properties

C:\kafka_2.13-2.8.0\bin\windows>connect-standalone.bat C:\kafka_2.13-2.8.0\confluent-6.1.1\etc\kafka\connect-standalone.properties C:\kafka_2.13-2.8.0\confluent-6.1.1\etc\kafka-connect-jdbc\connect-jdbc-source.properties