{"id":3911,"date":"2019-04-26T10:54:26","date_gmt":"2019-04-26T02:54:26","guid":{"rendered":"http:\/\/switch.linesno.com\/?p=3911"},"modified":"2019-04-26T10:54:26","modified_gmt":"2019-04-26T02:54:26","slug":"spring-boot-with-elkelasticsearch-logstash-kibana","status":"publish","type":"post","link":"http:\/\/switch.linesno.com\/?p=3911","title":{"rendered":"Spring boot with ELK(Elasticsearch + Logstash + Kibana)"},"content":{"rendered":"\n

5.31. Spring boot with ELK(Elasticsearch + Logstash + Kibana)<\/h2>\n\n\n\n

\u5c06 Spring boot \u65e5\u5fd7\u5199\u5165 ELK \u6709\u591a\u79cd\u5b9e\u73b0\u65b9\u5f0f\uff0c\u8fd9\u91cc\u4ec5\u63d0\u4f9b\u4e09\u79cd\u65b9\u6848\uff1a<\/p>\n\n\n\n

  1. Spring boot -> logback -> Tcp\/IP -> logstash -> elasticsearch\n\u8fd9\u79cd\u65b9\u5f0f\u5b9e\u73b0\u975e\u5e38\u65b9\u4fbf\u4e0d\u9700\u8981\u800c\u5916\u5305\u6216\u8005\u8f6f\u4ef6<\/li>
  2. Spring boot -> logback -> Redis -> logstash -> elasticsearch\n\u5229\u7528 Redis \u63d0\u4f9b\u7684\u53d1\u5e03\u8ba2\u9605\u529f\u80fd\u5c06\u65e5\u5fd7\u6295\u9012\u5230 elasticsearch<\/li>
  3. Spring boot -> logback -> Kafka -> logstash -> elasticsearch\nKafka \u65b9\u6cd5\u9002\u5408\u5927\u6570\u636e\u7684\u60c5\u51b5\u3002<\/li><\/ol>\n\n\n\n

    5.31.1. TCP \u65b9\u6848<\/h3>\n\n\n\n

    logstash \u914d\u7f6e<\/p>\n\n\n\n

    \t\t\tinput {\n  tcp {\n    host => \"172.16.1.16\" \n    port => 9250\n    mode => \"server\"\n    tags => [\"tags\"]\n    codec => json_lines  \/\/\u53ef\u80fd\u9700\u8981\u66f4\u65b0logstash\u63d2\u4ef6\n  }\n}\n\noutput {\n stdout{codec =>rubydebug}\n  elasticsearch {\n   hosts => [\"localhost:9200\"]  \/\/\u8fd9\u5757\u914d\u7f6e\u9700\u8981\u5e26\u7aef\u53e3\u53f7\n    flush_size => 1000\n\n  }\n}\t\t\t<\/pre>\n\n\n\n
    Spring boot logback.xml \u914d\u7f6e<\/p>\n\n\n\n
    \t\t\t<?xml version=\"1.0\" encoding=\"UTF-8\"?>\n<configuration>\n    <property resource=\"properties\/logback-variables.properties\" \/> \n\n    <appender name=\"STDOUT\" class=\"ch.qos.logback.core.ConsoleAppender\">\n        <encoder charset=\"UTF-8\">\n            <pattern>%d{HH:mm:ss.SSS} [%thread] %-5level %logger - %msg%n\n            <\/pattern>\n        <\/encoder>\n    <\/appender>\n    <appender name=\"LOGSTASH\" class=\"net.logstash.logback.appender.LogstashTcpSocketAppender\">\n        <destination>172.16.1.16:9250<\/destination>\n        <encoder charset=\"UTF-8\" class=\"net.logstash.logback.encoder.LogstashEncoder\" \/>\n    <\/appender>\n\n    <!--<appender name=\"async\" class=\"ch.qos.logback.classic.AsyncAppender\">-->\n        <!--<appender-ref ref=\"stash\" \/>-->\n    <!--<\/appender>-->\n    \n    <root level=\"info\">                    <!-- \u8bbe\u7f6e\u65e5\u5fd7\u7ea7\u522b -->\n        <appender-ref ref=\"STDOUT\" \/>\n        <appender-ref ref=\"LOGSTASH\" \/>\n    <\/root>\n<\/configuration>\t\t\t<\/pre>\n\n\n\n
    5.31.2. Redis \u65b9\u6848<\/h3>\n\n\n\n
    Maven pom.xml \u589e\u52a0 Logback Redis \u4f9d\u8d56<\/p>\n\n\n\n
    \t\t\t<!-- https:\/\/mvnrepository.com\/artifact\/com.cwbase\/logback-redis-appender -->\n<dependency>\n    <groupId>com.cwbase<\/groupId>\n    <artifactId>logback-redis-appender<\/artifactId>\n    <version>1.1.5<\/version>\n<\/dependency>\t\t\t<\/pre>\n\n\n\n
    Spring boot logback.xml \u914d\u7f6e<\/p>\n\n\n\n
    \t\t\t<?xml version=\"1.0\" encoding=\"UTF-8\"?>\n<configuration>\n\t<include resource=\"org\/springframework\/boot\/logging\/logback\/defaults.xml\" \/>\n\t<include resource=\"org\/springframework\/boot\/logging\/logback\/file-appender.xml\" \/>\n\t<property name=\"type.name\" value=\"test\" \/>\n\t<appender name=\"LOGSTASH\" class=\"com.cwbase.logback.RedisAppender\">\n\t\t<source>spring-application<\/source>\n\t\t<type>${type.name}<\/type>\n\t\t<host>localhost<\/host>\n\t\t<key>logstash:redis<\/key>\n\t\t<tags>test-2<\/tags>\n\t\t<mdc>true<\/mdc>\n\t\t<location>true<\/location>\n\t\t<callerStackIndex>0<\/callerStackIndex>\n\t\t<!--additionalField\u6dfb\u52a0\u9644\u52a0\u5b57\u6bb5 \u7528\u4e8ehead\u63d2\u4ef6\u663e\u793a -->\n\t\t<additionalField>\n\t\t\t<key>MyKey<\/key>\n\t\t\t<value>MyValue<\/value>\n\t\t<\/additionalField>\n\t\t<additionalField>\n\t\t\t<key>MySecondKey<\/key>\n\t\t\t<value>MyOtherValue<\/value>\n\t\t<\/additionalField>\n\t<\/appender>\n\t<root level=\"INFO\">\n\t\t<appender-ref ref=\"FILE\" \/>\n\t\t<appender-ref ref=\"LOGSTASH\" \/>\n\t<\/root>\n<\/configuration>\t\t\t<\/pre>\n\n\n\n
    logstash \u914d\u7f6e<\/p>\n\n\n\n
    \t\t\tinput {\n    redis {\n        host => 'localhost'\n        data_type => 'list'\n        port => \"6379\"\n        key => 'logstash:redis' #\u81ea\u5b9a\u4e49\n        type => 'redis-input'   #\u81ea\u5b9a\u4e49\n    }\n}\noutput {\n    elasticsearch {\n        host => \"localhost\" \n        codec => \"json\"\n        protocol => \"http\"\n    }\n}\t\t\t<\/pre>\n\n\n\n
    5.31.3. Kafka \u65b9\u6848<\/h3>\n\n\n\n
    \u6682\u65f6\u6ca1\u6709\u673a\u5668\u8d44\u6e90\uff0c\u672c\u7ae0\u8282\u5f85\u7eed<\/p>\n","protected":false},"excerpt":{"rendered":"
    5.31. Spring boot with ELK(Elasticsearch + Logstash + K […]<\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"closed","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[7],"tags":[],"class_list":["post-3911","post","type-post","status-publish","format-standard","hentry","category-day"],"_links":{"self":[{"href":"http:\/\/switch.linesno.com\/index.php?rest_route=\/wp\/v2\/posts\/3911","targetHints":{"allow":["GET"]}}],"collection":[{"href":"http:\/\/switch.linesno.com\/index.php?rest_route=\/wp\/v2\/posts"}],"about":[{"href":"http:\/\/switch.linesno.com\/index.php?rest_route=\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"http:\/\/switch.linesno.com\/index.php?rest_route=\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"http:\/\/switch.linesno.com\/index.php?rest_route=%2Fwp%2Fv2%2Fcomments&post=3911"}],"version-history":[{"count":1,"href":"http:\/\/switch.linesno.com\/index.php?rest_route=\/wp\/v2\/posts\/3911\/revisions"}],"predecessor-version":[{"id":3912,"href":"http:\/\/switch.linesno.com\/index.php?rest_route=\/wp\/v2\/posts\/3911\/revisions\/3912"}],"wp:attachment":[{"href":"http:\/\/switch.linesno.com\/index.php?rest_route=%2Fwp%2Fv2%2Fmedia&parent=3911"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"http:\/\/switch.linesno.com\/index.php?rest_route=%2Fwp%2Fv2%2Fcategories&post=3911"},{"taxonomy":"post_tag","embeddable":true,"href":"http:\/\/switch.linesno.com\/index.php?rest_route=%2Fwp%2Fv2%2Ftags&post=3911"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}