본문 바로가기
공부/Trouble Shooting

Kafka 오류 - 미해결

by son_i 2023. 10. 17.
728x90
consumer로 접속시 밀리초 단위로 마구 나오는 오류

bash-5.1# kafka-console-consumer.sh --bootstrap-server localhost:9092 --topic loan_request
이렇게 접속하면 이것들이 마구 나옴

[2023-10-16 10:01:46,818] WARN [Consumer clientId=consumer-console-consumer-53651-1, groupId=console-consumer-53651] 1 partitions have leader brokers without a matching listener, including [loan_request-0] (org.apache.kafka.clients.NetworkClient)
[2023-10-16 10:01:46,918] WARN [Consumer clientId=consumer-console-consumer-53651-1, groupId=console-consumer-53651] 1 partitions have leader brokers without a matching listener, including [loan_request-0] (org.apache.kafka.clients.NetworkClient)
[2023-10-16 10:01:47,020] WARN [Consumer clientId=consumer-console-consumer-53651-1, groupId=console-consumer-53651] 1 partitions have leader brokers without a matching listener, including [loan_request-0] (org.apache.kafka.clients.NetworkClient)
[2023-10-16 10:01:47,120] WARN [Consumer clientId=consumer-console-consumer-53651-1, groupId=console-consumer-53651] 1 partitions have leader brokers without a matching listener, including [loan_request-0] (org.apache.kafka.clients.NetworkClient)
[2023-10-16 10:01:47,221] WARN [Consumer clientId=consumer-console-consumer-53651-1, groupId=console-consumer-53651] 1 partitions have leader brokers without a matching listener, including [loan_request-0] (org.apache.kafka.clients.NetworkClient)
[2023-10-16 10:01:47,322] WARN [Consumer clientId=consumer-console-consumer-53651-1, groupId=console-consumer-53651] 1 partitions have leader brokers without a matching listener, including [loan_request-0] (org.apache.kafka.clients.NetworkClient)
[2023-10-16 10:01:47,426] WARN [Consumer clientId=consumer-console-consumer-53651-1, groupId=console-consumer-53651] 1 partitions have leader brokers without a matching listener, including [loan_request-0] (org.apache.kafka.clients.NetworkClient)

consumer main application을 실행시키면

2023-10-16 18:58:50.618  WARN 6572 --- [ntainer#0-0-C-1] org.apache.kafka.clients.NetworkClient   : [Consumer clientId=consumer-fintech-1, groupId=fintech] 1 partitions have leader brokers without a matching listener, including [loan_request-0]
2023-10-16 18:58:50.726  WARN 6572 --- [ntainer#0-0-C-1] org.apache.kafka.clients.NetworkClient   : [Consumer clientId=consumer-fintech-1, groupId=fintech] 1 partitions have leader brokers without a matching listener, including [loan_request-0]
2023-10-16 18:58:50.834  WARN 6572 --- [ntainer#0-0-C-1] org.apache.kafka.clients.NetworkClient   : [Consumer clientId=consumer-fintech-1, groupId=fintech] 1 partitions have leader brokers without a matching listener, including [loan_request-0]

이런오류가 마구 난다.

 

처음에는 kafka 알려진 호스트가 없습니다 이런 오류가 나서 docker-compose.yml 의

KAFKA_ADVERTISED_HOST_NAME: 127.0.0.1

이 부분을 kafka -> 127.0.0.1로 바꿔주었다.

 

그랬더니 일단 이렇게 된 것 !

https://stackoverflow.com/questions/53571823/why-could-kafka-warn-partitions-have-leader-brokers-without-a-matching-listener

 

Why could Kafka warn "partitions have leader brokers without a matching listener"?

I'm trying to get Kafka to work on docker-compose for the first time. The application runs fine without docker. But on docker, I get the error as described below. Any reason why Kafka would throw t...

stackoverflow.com

여기에서 KAFKA_ADVERTISED_HOST_NAME 대신 KAFKA_ADVERTISED_LISTENERS이걸 쓰라고 해서 바꿔줬다.

KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://kafka:29092,PLAINTEXT_HOST://localhost:9092

앞에 kafka는 hostname이고 localhost는 호스트 장비 IP의미 

Docker 네트워크 내부에서 접근할 주소와 외부에서 접근할 주소를 기입해줌.

이것만 넣으면 kafka가 켜졌다 바로 종료됨

 

고쳐주고 consumer 실행시 이번엔 아래 코드가 반복되다

2023-10-16 22:05:20.756  WARN 2168 --- [ntainer#0-0-C-1] org.apache.kafka.clients.NetworkClient   : [Consumer clientId=consumer-fintech-1, groupId=fintech] Connection to node -1 (localhost/127.0.0.1:9092) could not be established. Broker may not be available.
2023-10-16 22:05:20.756  WARN 2168 --- [ntainer#0-0-C-1] org.apache.kafka.clients.NetworkClient   : [Consumer clientId=consumer-fintech-1, groupId=fintech] Bootstrap broker localhost:9092 (id: -1 rack: null) disconnected
2023-10-16 22:05:21.735  INFO 2168 --- [ntainer#0-0-C-1] org.apache.kafka.clients.NetworkClient   : [Consumer clientId=consumer-fintech-1, groupId=fintech] Node -1 disconnected.

이 부분이 나오고 종료돼버림 

2023-10-16 22:05:22.077  INFO 2168 --- [ntainer#0-0-C-1] o.a.k.c.c.internals.ConsumerCoordinator  : [Consumer clientId=consumer-fintech-1, groupId=fintech] Resetting generation due to: consumer pro-actively leaving the group
2023-10-16 22:05:22.078  INFO 2168 --- [ntainer#0-0-C-1] o.a.k.c.c.internals.ConsumerCoordinator  : [Consumer clientId=consumer-fintech-1, groupId=fintech] Request joining group due to: consumer pro-actively leaving the group
2023-10-16 22:05:22.078  INFO 2168 --- [ntainer#0-0-C-1] o.a.k.clients.consumer.KafkaConsumer     : [Consumer clientId=consumer-fintech-1, groupId=fintech] Unsubscribed all topics or patterns and assigned partitions
2023-10-16 22:05:22.079  INFO 2168 --- [ntainer#0-0-C-1] o.a.k.c.c.internals.ConsumerCoordinator  : [Consumer clientId=consumer-fintech-1, groupId=fintech] Resetting generation due to: consumer pro-actively leaving the group
2023-10-16 22:05:22.079  INFO 2168 --- [ntainer#0-0-C-1] o.a.k.c.c.internals.ConsumerCoordinator  : [Consumer clientId=consumer-fintech-1, groupId=fintech] Request joining group due to: consumer pro-actively leaving the group

 

 

UnknownHostException 
kafka 모듈 작성 후 Test 시 발생한 오류
$ curl --location --request POST 'http://localhost:8080/fintech/api/v1/request' 
--header 'Content-Type: application/json' 
--data-raw 
'{
"userRegistrationNumber" : "1234568090",
"userName" : "쏘은",
"userIncomeAmount" : 100000
}'

userKey 값이 리턴되어야 하는데 PostMan에서는 잘 되는데 

터미널에서 하면 안 된다 대체 왜 ????????????? 그리고 ApiApplication에서는 아래와 같은 오류가 계속 반복해서 뜬다.

2023-10-16 23:29:42.414  WARN 24068 --- [ad | producer-1] org.apache.kafka.clients.NetworkClient   : [Producer clientId=producer-1] Error connecting to node kafka:9092 (id: 1001 rack: null)

java.net.UnknownHostException: 알려진 호스트가 없습니다 (kafka)
	at java.base/java.net.Inet6AddressImpl.lookupAllHostAddr(Native Method) ~[na:na]
	at java.base/java.net.InetAddress$PlatformNameService.lookupAllHostAddr(InetAddress.java:930) ~[na:na]
	at java.base/java.net.InetAddress.getAddressesFromNameService(InetAddress.java:1543) ~[na:na]
	at java.base/java.net.InetAddress$NameServiceAddresses.get(InetAddress.java:848) ~[na:na]
	at java.base/java.net.InetAddress.getAllByName0(InetAddress.java:1533) ~[na:na]
	at java.base/java.net.InetAddress.getAllByName(InetAddress.java:1386) ~[na:na]
	at java.base/java.net.InetAddress.getAllByName(InetAddress.java:1307) ~[na:na]
	at org.apache.kafka.clients.DefaultHostResolver.resolve(DefaultHostResolver.java:27) ~[kafka-clients-3.1.2.jar:na]
	at org.apache.kafka.clients.ClientUtils.resolve(ClientUtils.java:110) ~[kafka-clients-3.1.2.jar:na]
	at org.apache.kafka.clients.ClusterConnectionStates$NodeConnectionState.currentAddress(ClusterConnectionStates.java:511) ~[kafka-clients-3.1.2.jar:na]
	at org.apache.kafka.clients.ClusterConnectionStates$NodeConnectionState.access$200(ClusterConnectionStates.java:468) ~[kafka-clients-3.1.2.jar:na]
	at org.apache.kafka.clients.ClusterConnectionStates.currentAddress(ClusterConnectionStates.java:173) ~[kafka-clients-3.1.2.jar:na]
	at org.apache.kafka.clients.NetworkClient.initiateConnect(NetworkClient.java:988) ~[kafka-clients-3.1.2.jar:na]
	at org.apache.kafka.clients.NetworkClient.access$600(NetworkClient.java:73) ~[kafka-clients-3.1.2.jar:na]
	at org.apache.kafka.clients.NetworkClient$DefaultMetadataUpdater.maybeUpdate(NetworkClient.java:1161) ~[kafka-clients-3.1.2.jar:na]
	at org.apache.kafka.clients.NetworkClient$DefaultMetadataUpdater.maybeUpdate(NetworkClient.java:1049) ~[kafka-clients-3.1.2.jar:na]
	at org.apache.kafka.clients.NetworkClient.poll(NetworkClient.java:558) ~[kafka-clients-3.1.2.jar:na]
	at org.apache.kafka.clients.producer.internals.Sender.runOnce(Sender.java:328) ~[kafka-clients-3.1.2.jar:na]
	at org.apache.kafka.clients.producer.internals.Sender.run(Sender.java:243) ~[kafka-clients-3.1.2.jar:na]
	at java.base/java.lang.Thread.run(Thread.java:829) ~[na:na]

2023-10-16 23:29:42.475  WARN 24068 --- [ad | producer-1] org.apache.kafka.clients.NetworkClient   : [Producer clientId=producer-1] Error connecting to node kafka:9092 (id: 1001 rack: null)

java.net.UnknownHostException: kafka
	at java.base/java.net.InetAddress$CachedAddresses.get(InetAddress.java:797) ~[na:na]
	at java.base/java.net.InetAddress.getAllByName0(InetAddress.java:1533) ~[na:na]
	at java.base/java.net.InetAddress.getAllByName(InetAddress.java:1386) ~[na:na]
	at java.base/java.net.InetAddress.getAllByName(InetAddress.java:1307) ~[na:na]
	at org.apache.kafka.clients.DefaultHostResolver.resolve(DefaultHostResolver.java:27) ~[kafka-clients-3.1.2.jar:na]
	at org.apache.kafka.clients.ClientUtils.resolve(ClientUtils.java:110) ~[kafka-clients-3.1.2.jar:na]
	at org.apache.kafka.clients.ClusterConnectionStates$NodeConnectionState.currentAddress(ClusterConnectionStates.java:511) ~[kafka-clients-3.1.2.jar:na]
	at org.apache.kafka.clients.ClusterConnectionStates$NodeConnectionState.access$200(ClusterConnectionStates.java:468) ~[kafka-clients-3.1.2.jar:na]
	at org.apache.kafka.clients.ClusterConnectionStates.currentAddress(ClusterConnectionStates.java:173) ~[kafka-clients-3.1.2.jar:na]
	at org.apache.kafka.clients.NetworkClient.initiateConnect(NetworkClient.java:988) ~[kafka-clients-3.1.2.jar:na]
	at org.apache.kafka.clients.NetworkClient.access$600(NetworkClient.java:73) ~[kafka-clients-3.1.2.jar:na]
	at org.apache.kafka.clients.NetworkClient$DefaultMetadataUpdater.maybeUpdate(NetworkClient.java:1161) ~[kafka-clients-3.1.2.jar:na]
	at org.apache.kafka.clients.NetworkClient$DefaultMetadataUpdater.maybeUpdate(NetworkClient.java:1049) ~[kafka-clients-3.1.2.jar:na]
	at org.apache.kafka.clients.NetworkClient.poll(NetworkClient.java:558) ~[kafka-clients-3.1.2.jar:na]
	at org.apache.kafka.clients.producer.internals.Sender.runOnce(Sender.java:328) ~[kafka-clients-3.1.2.jar:na]
	at org.apache.kafka.clients.producer.internals.Sender.run(Sender.java:243) ~[kafka-clients-3.1.2.jar:na]
	at java.base/java.lang.Thread.run(Thread.java:829) ~[na:na]

0.0.0.0을 카프카가 설치된 서버의 주소로 대체하기 위해 작성

https://jyoondev.tistory.com/184

 

카프카 - (2) 설치

카프카 설치 카프카는 자바기반의 언어로 작성되었기 때문에 구동하기 위해서는 자바가 필요하다. 따라서 자바를 설치해 주어야 한다. 해당 포스팅에서는 자바 1.8이 사용되었고, 리눅스 환경에

jyoondev.tistory.com

soens@soni MINGW64 /c/spring_prac/fintech/docker (main)
$ docker exec -it kafka bash
bash-5.1# kafka-broker-api-versions.sh --bootstrap-server 0.0.0.0:9092
[2023-10-16 14:34:57,341] WARN [LegacyAdminClient clientId=admin-1] 1 partitions have leader brokers without a matching listener, including [loan_request-0] (org.apache.kafka.clients.NetworkClient)
127.0.0.1:9092 (id: 1002 rack: null) -> (
        Produce(0): 0 to 8 [usable: 8],
        Fetch(1): 0 to 11 [usable: 11],
        ListOffsets(2): 0 to 5 [usable: 5],
        Metadata(3): 0 to 9 [usable: 9],
        LeaderAndIsr(4): 0 to 4 [usable: 4],
        StopReplica(5): 0 to 2 [usable: 2],
        UpdateMetadata(6): 0 to 6 [usable: 6],
        ControlledShutdown(7): 0 to 3 [usable: 3],
        OffsetCommit(8): 0 to 8 [usable: 8],
        OffsetFetch(9): 0 to 7 [usable: 7],
        FindCoordinator(10): 0 to 3 [usable: 3],
        JoinGroup(11): 0 to 7 [usable: 7],
        Heartbeat(12): 0 to 4 [usable: 4],
        LeaveGroup(13): 0 to 4 [usable: 4],
        SyncGroup(14): 0 to 5 [usable: 5],
        DescribeGroups(15): 0 to 5 [usable: 5],
        ListGroups(16): 0 to 3 [usable: 3],
        SaslHandshake(17): 0 to 1 [usable: 1],
        ApiVersions(18): 0 to 3 [usable: 3],
        CreateTopics(19): 0 to 5 [usable: 5],
        DeleteTopics(20): 0 to 4 [usable: 4],
        DeleteRecords(21): 0 to 1 [usable: 1],
        InitProducerId(22): 0 to 3 [usable: 3],
        OffsetForLeaderEpoch(23): 0 to 3 [usable: 3],
        AddPartitionsToTxn(24): 0 to 1 [usable: 1],
        AddOffsetsToTxn(25): 0 to 1 [usable: 1],
        EndTxn(26): 0 to 1 [usable: 1],
        WriteTxnMarkers(27): 0 [usable: 0],
        TxnOffsetCommit(28): 0 to 3 [usable: 3],
        DescribeAcls(29): 0 to 2 [usable: 2],
        CreateAcls(30): 0 to 2 [usable: 2],
        DeleteAcls(31): 0 to 2 [usable: 2],
        DescribeConfigs(32): 0 to 2 [usable: 2],
        AlterConfigs(33): 0 to 1 [usable: 1],
        AlterReplicaLogDirs(34): 0 to 1 [usable: 1],
        DescribeLogDirs(35): 0 to 1 [usable: 1],
        SaslAuthenticate(36): 0 to 2 [usable: 2],
        CreatePartitions(37): 0 to 2 [usable: 2],
        CreateDelegationToken(38): 0 to 2 [usable: 2],
        RenewDelegationToken(39): 0 to 2 [usable: 2],
        ExpireDelegationToken(40): 0 to 2 [usable: 2],
        IncrementalAlterConfigs(44): 0 to 1 [usable: 1],
        AlterPartitionReassignments(45): 0 [usable: 0],
        ListPartitionReassignments(46): 0 [usable: 0],
        OffsetDelete(47): 0 [usable: 0]
)

이젠 이 오류가 ms 단위로 계속 뜬다.

2023-10-16 23:39:37.112  WARN 19056 --- [ad | producer-1] org.apache.kafka.clients.NetworkClient   : [Producer clientId=producer-1] 1 partitions have leader brokers without a matching listener, including [loan_request-0]
2023-10-16 23:39:37.220  WARN 19056 --- [ad | producer-1] org.apache.kafka.clients.NetworkClient   : [Producer clientId=producer-1] 1 partitions have leader brokers without a matching listener, including [loan_request-0]
2023-10-16 23:39:37.330  WARN 19056 --- [ad | producer-1] org.apache.kafka.clients.NetworkClient   : [Producer clientId=producer-1] 1 partitions have leader brokers without a matching listener, including [loan_request-0]

docker desktop - kafka - files - opt - kafka_2.12-2.5.0 - config - server.properties에

31번째 줄 주석을 해제해주라는데 나는 이미 해제가 되어있었음

 

# kafka-console-consumer.sh --bootstrap-server localhost:9092 --topic loan_request

이렇게 consume을 해봐도 마찬가지 ... 

[2023-10-16 16:15:18,136] WARN [Consumer clientId=consumer-console-consumer-31017-1, groupId=console-consumer-31017] 1 partitions have leader brokers without a matching listener, including [loan_request-0] (org.apache.kafka.clients.NetworkClient)

저 server.properties에 0.0.0.0을 localhost로 바꿔주었음

그래도 동일한 오류...

 

아직 미해결 ㅠ

저작자표시 비영리 변경금지

'공부 > Trouble Shooting' 카테고리의 다른 글

UserDetailService 2개 구현 시 발생하는 에러 처리  (1) 2023.10.25
빈 순환참조 문제 The dependencies of some of the beans in the application context form a cycle:  (0) 2023.10.21
docker에 kafka, zookeeper 설치 오류  (2) 2023.10.16
(Kotlin)Multi-Module-Gradle 프로젝트 첫 실행 시 Parameter 0 of constructor in com.zerobase.api.test.TestService required a bean of type 오류  (0) 2023.10.13
[docker mysql 연결] docker에서 mysql연결 시 Connection 에러  (0) 2023.10.13

관련글

티스토리툴바