EFK 로그수집 (ElasticSearch+Fluendt+Kibana)

//------------------------------------------------------------------------------
// IAM 설정
// Elasticsearch 에서 사용할 유저를 생성
//------------------------------------------------------------------------------
생성시 Access type 을 Programmatic access 선택. 나머지는 그냥 next
Access key ID : -
Secret access key : -

//------------------------------------------------------------------------------
// Elasticsearch 설정
// (http://mykumi.tistory.com/entry/AWS-ElasticSearchKibanaFluentd-1-AWS-ElasticSearch-생성?category=503302)
//------------------------------------------------------------------------------
 Configure cluster
기본값으로
 Set up access
Network configuration : Public access
Kibana authentication : 체크안함
Access policy
- Allow or deny access to one or more AWS accounts or IAM users 선택
위에서 생성한 유저의 arn 입력 (fluentd에서 접근하기 위해서)
- Allow access to the domain from specific IPs 선택
접근할 ip 입력 (kibana 접근을 위해서)
- 위의 2개를 합셔처 정책을 edit 해야 함.

//------------------------------------------------------------------------------
// Fluentd 설치 (오픈소스)
// // http://mykumi.tistory.com/entry/AWS-ElasticSearchKibanaFluentd-통계-2-Fluentd?category=503302
//------------------------------------------------------------------------------
Fluentd 설치전 확인사항
-ntp 설치 확인 (우리 서버에는 이미 설치되어 있음)
  ps -ef | grep ntpd
-프로세스의 자원한도 증가 (우리 서버에는 이미 설정되어 있음)
 ulimit -n
 65535 이상 이어야 함
-네트웍커널 파라메터 최적화 (이건 잘 몰라서 적용안함)
-루비설치여부 확인
 ruby -v
ruby 1.8.7 (2013-06-27 patchlevel 374) [x86_64-linux]

http://www.fluentd.org/ 다운로드.
(우리 52.194.115.16 서버에 설치함. 이게 로그들을 모아놓은 서버임)
(td-agent 는 Fluentd를 쉽게 설치하고 사용하기 위한 래퍼프로그램)
(centOS 설치 가이드 : https://docs.fluentd.org/v1.0/articles/install-by-rpm)

centOS에 설치
curl -L https://toolbelt.treasuredata.com/sh/install-redhat-td-agent3.sh | sh
Complete!
Installation completed. Happy Logging!

elasticsearch plugin 설치
sudo /usr/sbin/td-agent-gem install fluent-plugin-aws-elasticsearch-service
149 gems installed
뭔가 많이 설치된다.
 
td-agent 설정파일
/etc/td-agent/td-agent.conf

td-agent 로그파일
/var/log/td-agent/td-agent.log
(이 파일에는 아무 내용이 없어야 정상이다)

 

보내는서버 td-agent 설정 (로드밸런싱에 속한 ec2들)

#------------------------------------------------------------------------------- # 입력 : 라라벨로그파일 # path : 검사할 파일 # pos_file : 검사할 파일의 위치를 기록하는거 같음. #------------------------------------------------------------------------------- <source> @type tail path /home/www/xxx/current/storage/logs/laravel-%Y-%m-%d.log pos_file /var/log/td-agent/laravel.log.pos format none tag 태그이름1 </source>

#------------------------------------------------------------------------------- # 입력 : nginx로그파일 # nginx로그파일은 root유저만 읽을수 있으므로, # /etc/init.d/td-agent 파일을 TD_AGENT_USER=root 로 수정해야 함. #------------------------------------------------------------------------------- <source> @type tail path /var/log/nginx/xxx.access.log pos_file /var/log/td-agent/xxx.access.log.pos tag 태그이름2 <parse> @type nginx </parse> </source>

#------------------------------------------------------------------------------- # 출력: 다른 서버의 Fluentd #------------------------------------------------------------------------------- <match 태그이름1> @type forward <server> host 받을서버의ip주소 port 24224 </server> <buffer> @type file path /var/log/td-agent/buffer flush_mode interval flush_interval 10s #flush_mode immediate flush_at_shutdown true </buffer> </match>

#------------------------------------------------------------------------------- # 출력: aws elasticsearch # logstash_format true : 출력형식을 logstash 포맷으로 (키바나를 사용하기 위해) #------------------------------------------------------------------------------- <match 태그이름2> @type "aws-elasticsearch-service" logstash_format true <endpoint> url https://--- region ap-northeast-1 access_key_id --- secret_access_key --- </endpoint> </match>

받는서버 td-agent 설정 (라라벨 로그를 받아서 한곳에 저장함)

#------------------------------------------------------------------------------- # 입력 : 다른 서버에 설치된 fluentd 에서 입력을 받음 #------------------------------------------------------------------------------- <source> @type forward port 24224 </source>

#------------------------------------------------------------------------------- # 출력 : elb에 속한 api 에서 들어온 데이타를 출력함 #------------------------------------------------------------------------------- <match 태그이름1> @type file path /var/log/td-agent/laravel.elb.api/laravel.elb.api append true <buffer> @type file path /var/log/td-agent/buffer #flush_mode interval flush_mode immediate flush_at_shutdown true </buffer> <format> @type single_value add_newline true message_key message </format> </match>

 

//------------------------------------------------------------------------------

// td-agent 명령어

//------------------------------------------------------------------------------
sudo service td-agent start
sudo service td-agent stop
sudo service td-agent restart
sudo service td-agent status

sudo /etc/init.d/td-agent start
sudo /etc/init.d/td-agent stop
sudo /etc/init.d/td-agent restart
sudo /etc/init.d/td-agent status



//------------------------------------------------------------------------------
// nginx 로그를 모아놓은 es에서 Kibana 활용을 잘 하면 됨
//------------------------------------------------------------------------------