[Spring Boot] SpringBoot 2.x의 HikariCp 에러 정리

[ 오늘의  장애 정리 ] 

어제 새벽부터 갑자기 서버가 죽음 ㅜ

두대 돌리고 있었는데 두대 다 죽어버림..

로그는 아래와 같다 (주요 로그들만)

2019:01:17 01:02:35.401 WARN  --- [http-nio-8080-exec-1] o.h.e.jdbc.spi.SqlExceptionHelper : SQL Error: 0, SQLState: null  2019:01:17 01:02:36.032 ERROR --- [http-nio-8080-exec-1] o.h.e.jdbc.spi.SqlExceptionHelper : HikariPool-1 - Connection is not available, request timed out after 30000ms.  2019:01:17 01:02:39.641 INFO  --- [http-nio-8080-exec-91] c.s.s.s.impl.ResultServiceImpl : org.springframework.transaction.CannotCreateTransactionException: Could not open JPA EntityManager for transaction; nested exception is org.hibernate.exception.JDBCConnectionException: Unable to acquire JDBC Connection  2019:01:17 01:02:44.224 WARN  --- [http-nio-8080-exec-19] o.h.e.jdbc.spi.SqlExceptionHelper : SQL Error: 0, SQLState: null  2019:01:17 01:02:46.098 ERROR --- [http-nio-8080-exec-19] o.h.e.jdbc.spi.SqlExceptionHelper : HikariPool-1 - Connection is not available, request timed out after 38882ms.  2019:01:17 01:04:49.721 INFO  --- [http-nio-8080-exec-192] c.s.s.s.impl.ResultServiceImpl : org.springframework.transaction.CannotCreateTransactionException: Could not open JPA EntityManager for transaction; nested exception is org.hibernate.exception.JDBCConnectionException: Unable to acquire JDBC Connection  2019:01:17 01:12:02.982 INFO  --- [http-nio-8080-exec-1] c.s.s.s.impl.ResultServiceImpl : org.springframework.dao.DataAccessResourceFailureException: unable to obtain isolated JDBC connection; nested exception is org.hibernate.exception.JDBCConnectionException: unable to obtain isolated JDBC connection  2019:01:17 01:12:03.304 ERROR --- [http-nio-8080-exec-6] o.h.e.jdbc.spi.SqlExceptionHelper : HikariPool-1 - Connection is not available, request timed out after 30125ms.  2019:01:17 01:12:03.312 INFO  --- [http-nio-8080-exec-4] c.s.s.s.impl.ResultServiceImpl : org.springframework.dao.DataAccessResourceFailureException: unable to obtain isolated JDBC connection; nested exception is org.hibernate.exception.JDBCConnectionException: unable to obtain isolated JDBC connection  2019:01:17 01:12:03.589 ERROR --- [http-nio-8080-exec-91] o.a.c.c.C.[.[.[.[dispatcherServlet] : Servlet.service() for servlet [dispatcherServlet] in context with path [] threw exception [Handler dispatch failed; nested exception is java.lang.OutOfMemoryError: Java heap space] with root cause  java.lang.OutOfMemoryError: Java heap space

// 로그에서 Hikari 검색 

2019:01:17 13:00:08.962 WARN  --- [http-nio-8080-exec-6] com.zaxxer.hikari.pool.PoolBase : HikariPool-1 - Failed to validate connection com.mysql.jdbc.JDBC4Connection@32e56bc7 (No operations allowed after connection closed.). Possibly consider using a shorter maxLifetime value.

저런게 엄청 나옴...

아니 근데 찾아도 찾아도 Hikari라는 건 안쓴단 말이지 

찾아보니

Spring Boot 2.x 이상 부터 Hikari가 강제적으로 기본 제공된다고 함 -ㅂ- 

참고 : https://github.com/spring-projects/spring-boot/wiki/Spring-Boot-2.0-Migration-Guide#configuring-a-datasource

요 Hikari 기본 셋팅이 사용하는 DBMS 셋팅 시간 보다 길어 쓰레드 풀이 일어남 

왜 그런지는 아래 블로그 참고

참고 : https://d2.naver.com/helloworld/5102792

먼저 DB설정 값들을 확인 해보자

show variables like '%time%'

현재 DB에 설정 된 값이..

wait_timeout 600

net_read_timeout 30

connect_timeout 10

[ 해결방법 ]

MsSql 설정 값들 보다 wait타임 시간 적게함

설정 변경

spring.datasource.hikari.idle-timeout=10000
spring.datasource.hikari.max-lifetime=420000
spring.datasource.hikari.connection-timeout=10000
spring.datasource.hikari.validation-timeout=10000

dependency 추가

<dependency>
    <groupId>org.springframework.boot</groupId>
    <artifactId>spring-boot-starter-jdbc</artifactId>
    <exclusions>
        <exclusion>
            <groupId>org.apache.tomcat</groupId>
            <artifactId>tomcat-jdbc</artifactId>
        </exclusion>
    </exclusions>
</dependency>

변경 후 적용했음

힙 메모리도 1024m 에서 2048m로 올림

아직 까진 잘 돌아가는 중...

참고 했던 블로그들 첨부 

https://eclipse4j.tistory.com/264

http://pstree.tistory.com/133

https://jojoldu.tistory.com/296?fbclid=IwAR1YgPWw3WOGGkdbEe0V2iZwPRbEsGNkQWzKjLeqwO2nq8PDXX8m5ovQfLI

오늘은 별일 없길.. ㅠ