一、EFK日志收集系统介绍

1.EFK简介

EFK：分别表示：Elasticsearch , Fluentd, Kibana , 其中ELasticsearch负责日志保存和搜索，Fluentd负责收集日志，Kibana 负责界面，三者配合起来，形成一个非常完美的解决方案。

2.Elasticsearch简介

Elasticsearch是一个基于Apache Lucene的开源搜索和数据分析引擎引擎，Elasticsearch使用Java进行开发，并使用Lucene作为其核心实现所有索引和搜索的功能。

(相关资料图)

3.Elasticsearch的特点

Elasticsearch是一个实时的，分布式的，可扩展的搜索引擎。Elasticsearch允许进行全文本和结构化搜索以及对日志进行分析。Elasticsearch 是一个搜索引擎，负责存储日志并提供查询接口。Elasticsearch通常用于索引和搜索大量日志数据，也可以用于搜索许多不同种类的文档。

4.、Filebeat与Kibana介绍

Filebeat是用于转发和集中日志数据的轻量级传送工具。Filebeat监视您指定的日志文件或位置，收集日志事件，并将它们转发到Elasticsearch或 Logstash进行索引。Kibana是一个开源的分析与可视化平台，被设计用于和Elasticsearch一起使用的。通过kibana可以搜索、查看和交互存放在Elasticsearch中的数据，利用各种不同的图表、表格和地图等，Kibana能够对数据进行分析与可视化。

5.EFK架构图

image.png

二、实践环境介绍

计算机名	系统版本	IP地址	Docker版本
jeven	centos 7.6	192.168.3.166	20.10.17

三、检查本地Docker环境

1.检查本地Docker版本

检查当前系统的docker版本

[root@jeven ~]# docker versionClient: Docker Engine - Community Version:           20.10.17 API version:       1.41 Go version:        go1.17.11 Git commit:        100c701 Built:             Mon Jun  6 23:05:12 2022 OS/Arch:           linux/amd64 Context:           default Experimental:      trueServer: Docker Engine - Community Engine:  Version:          20.10.17  API version:      1.41 (minimum version 1.12)  Go version:       go1.17.11  Git commit:       a89b842  Built:            Mon Jun  6 23:03:33 2022  OS/Arch:          linux/amd64  Experimental:     false containerd:  Version:          1.6.6  GitCommit:        10c12954828e7c7c9b6e0ea9b0c02b01407d3ae1 runc:  Version:          1.1.2  GitCommit:        v1.1.2-0-ga916309 docker-init:  Version:          0.19.0  GitCommit:        de40ad0

2.检查Docker服务状态

检查本地Docker状态是否正常

[root@jeven ~]# systemctl status docker● docker.service - Docker Application Container Engine   Loaded: loaded (/usr/lib/systemd/system/docker.service; enabled; vendor preset: disabled)   Active: active (running) since Mon 2023-01-02 23:06:06 CST; 2 weeks 2 days ago     Docs: https://docs.docker.com Main PID: 100874 (dockerd)    Tasks: 58   Memory: 3.6G   CGroup: /system.slice/docker.service

3.检查docker compose版本

检查本机docker compose版本

[root@jeven efk]# docker compose versionDocker Compose version v2.6.0

四、下载elasticsearch等相关镜像

下载elasticsearch镜像

[root@jeven efk]# docker pull docker.elastic.co/elasticsearch/elasticsearch:7.17.57.17.5: Pulling from elasticsearch/elasticsearch5486d18d7ee8: Pull complete 059ab60189a6: Pull complete f68717dc7875: Pull complete 543411f2e134: Pull complete db298b0bce73: Pull complete 841c800fd413: Pull complete 9401277c6728: Pull complete d677f77adbd8: Pull complete f0aaff8ec792: Pull complete Digest: sha256:76344d5f89b13147743db0487eb76b03a7f9f0cd55abe8ab887069711f2ee27dStatus: Downloaded newer image for docker.elastic.co/elasticsearch/elasticsearch:7.17.5docker.elastic.co/elasticsearch/elasticsearch:7.17.5

下载dkibana:7.17.5镜像root@jeven efk# docker pull docker.elastic.co/kibana/kibana:7.17.57.17.5: Pulling from kibana/kibana5486d18d7ee8: Already exists 16dd14f60afd: Pull complete d3f31853b425: Pull complete b189fa69d6ae: Pull complete 84f30eda8712: Pull complete 89732bc75041: Pull complete 2f3ab907567d: Pull complete eee1465a6f1a: Pull complete dee78761106f: Pull complete 77a7057d9e64: Pull complete 4f944b4f8458: Pull complete 0ee7e54154c9: Pull complete 3d21668734f2: Pull complete 2c4f2f7870ad: Pull complete Digest: sha256:07038507d29f21e96f3af081e4ae059661c8e16a4307776ef00d75a692cf99c7Status: Downloaded newer image for docker.elastic.co/kibana/kibana:7.17.5docker.elastic.co/kibana/kibana:7.17.5下载filebeat:7.17.5镜像root@jeven efk# docker pull docker.elastic.co/beats/filebeat:7.17.57.17.5: Pulling from beats/filebeat5486d18d7ee8: Already exists 64e71d8ac435: Pull complete b32f71c16e16: Pull complete ec1697632735: Pull complete 296368aad1f3: Pull complete ed91d5b07939: Pull complete 3c3383e08710: Pull complete 855a09516cb7: Pull complete bf85c6107785: Pull complete f2317aa670f5: Pull complete 14c66766e716: Pull complete Digest: sha256:072f7c62c0d684189e9af8fc632b3235b2aceffa62a4657a1a4e0201c5976310Status: Downloaded newer image for docker.elastic.co/beats/filebeat:7.17.5docker.elastic.co/beats/filebeat:7.17.5

五、编辑filebeat.yaml文件

编辑filebeat.yaml配置文件

[root@jeven efk]# cat filebeat.yaml [root@jeven efk]# cat filebeat.yaml filebeat.inputs:- type: log  paths:     - "/usr/share/filebeat/logs/*"processors:- decode_json_fields:    fields: ["message"]    target: ""    overwrite_keys: trueoutput.elasticsearch:  hosts: ["http://192.168.3.166:9200"]  indices:    - index: "filebeat-%{+yyyy.MM.dd}"#    - index: "filebeat-%{[agent.version]}-%{+yyyy.MM.dd}"#  username: "..."#  password: "..."setup.kibana:  host: "http://192.168.3.166:5601"#output.console:#  enabled: true#  codec.json:#    pretty: true#    #escape_html: falselogging.json: truelogging.metrics.enabled: false

六、部署EFK系统

1.创建数据目录

创建部署及数据目录

mkdir -p /data/efk/es/data

目录授权

chmod -R 666 /data/efk

2.编辑docker-compose.yaml文件

编辑efk系统的docker-compose.yaml部署文件

[root@jeven efk]# cat docker-compose.yaml version: "3.3"services:  elasticsearch:    image: "docker.elastic.co/elasticsearch/elasticsearch:7.17.5"    container_name: elasticsearch    restart: always    environment:    - "ES_JAVA_OPTS=-Xms512m -Xmx512m"    - "discovery.type=single-node"    - "cluster.name=myes"    - "node.name=jeven"    # - xpack.security.enabled: "false"    ulimits:      memlock:        soft: -1        hard: -1     networks:      myefk:        ipv4_address: 172.29.120.10        aliases:        - es        - jeven    ports:    - "9200:9200"    - "9300:9300"    volumes:    - /data/efk/es/data/:/usr/share/elasticsearch/data  kibana:    image: "docker.elastic.co/kibana/kibana:7.17.5"    restart: always    environment:      ELASTICSEARCH_URL: http://192.168.3.166:9200      ELASTICSEARCH_HOSTS: "["http:/192.168.3.166:9200"]"      I18N_LOCALE: zh-CN    networks:      myefk:        ipv4_address: 172.29.120.20        aliases:          - kibana          - kib    ports:    - "5601:5601"    links:    - "elasticsearch"  filebeat:    image: "docker.elastic.co/beats/filebeat:7.17.5"    restart: always    networks:      myefk:        ipv4_address: 172.29.120.30        aliases:          - filebeat          - fb    user: root    command: ["--strict.perms=false"]    volumes:    - ./filebeat.yaml:/usr/share/filebeat/filebeat.yml    - /var/lib/docker:/var/lib/docker:ro    - /var/run/docker.sock:/var/run/docker.sock    links:    - "elasticsearch"    - "kibana"networks:  myefk:    driver: bridge    ipam:      config:        - subnet: 172.29.120.0/24

3.部署EFK

执行docker compose up -d ，开始部署efk

[root@jeven efk]# docker compose up -d[+] Running 4/4 ⠿ Network efk_myefk         Created                                                                                           0.0s ⠿ Container elasticsearch   Started                                                                                           0.4s ⠿ Container efk-kibana-1    Started                                                                                           0.8s ⠿ Container efk-filebeat-1  Started                                                                                           0.8s

4.检查efk相关容器状态

检查efk的相关容器启动状态

[root@jeven efk]# docker compose psNAME                COMMAND                  SERVICE             STATUS              PORTSefk-filebeat-1      "/usr/bin/tini -- /u…"   filebeat            running             efk-kibana-1        "/bin/tini -- /usr/l…"   kibana              running             0.0.0.0:5601->5601/tcp, :::5601->5601/tcpelasticsearch       "/bin/tini -- /usr/l…"   elasticsearch       running             0.0.0.0:9200->9200/tcp, 0.0.0.0:9300->9300/tcp, :::9200->9200/tcp, :::9300->9300/tcp

5.查看efk相关容器日志信息

通过docker compose logs 来查看容器运行日志信息。

[root@jeven efk]# docker compose logs |headefk-kibana-1  | {"type":"log","@timestamp":"2023-01-20T01:43:47+00:00","tags":["info","plugins-service"],"pid":7,"message":"Plugin \"metricsEntities\" is disabled."}efk-kibana-1  | {"type":"log","@timestamp":"2023-01-20T01:43:47+00:00","tags":["info","http","server","Preboot"],"pid":7,"message":"http server running at http://0.0.0.0:5601"}efk-kibana-1  | {"type":"log","@timestamp":"2023-01-20T01:43:47+00:00","tags":["warning","config","deprecation"],"pid":7,"message":"Starting in 8.0, the Kibana logging format will be changing. This may affect you if you are doing any special handling of your Kibana logs, such as ingesting logs into Elasticsearch for further analysis. If you are using the new logging configuration, you are already receiving logs in both old and new formats, and the old format will simply be going away. If you are not yet using the new logging configuration, the log format will change upon upgrade to 8.0. Beginning in 8.0, the format of JSON logs will be ECS-compatible JSON, and the default pattern log format will be configurable with our new logging system. Please refer to the documentation for more information about the new logging format."}efk-kibana-1  | {"type":"log","@timestamp":"2023-01-20T01:43:47+00:00","tags":["warning","config","deprecation"],"pid":7,"message":"The default mechanism for Reporting privileges will work differently in future versions, which will affect the behavior of this cluster. Set \"xpack.reporting.roles.enabled\" to \"false\" to adopt the future behavior before upgrading."}efk-kibana-1  | {"type":"log","@timestamp":"2023-01-20T01:43:47+00:00","tags":["warning","config","deprecation"],"pid":7,"message":"从 8.0 开始，用户会话将在处于非活动状态 8 小时后自动超时。覆盖此值以更改超时。"}efk-kibana-1  | {"type":"log","@timestamp":"2023-01-20T01:43:47+00:00","tags":["warning","config","deprecation"],"pid":7,"message":"从 8.0 开始，将在 30 天后自动要求用户重新登录。覆盖此值以更改超时。"}efk-kibana-1  | {"type":"log","@timestamp":"2023-01-20T01:43:47+00:00","tags":["info","plugins-system","standard"],"pid":7,"message":"Setting up [113] plugins: [translations,licensing,globalSearch,globalSearchProviders,features,licenseApiGuard,code,usageCollection,xpackLegacy,taskManager,telemetryCollectionManager,telemetryCollectionXpack,kibanaUsageCollection,share,embeddable,uiActionsEnhanced,screenshotMode,banners,telemetry,newsfeed,mapsEms,mapsLegacy,kibanaLegacy,fieldFormats,expressions,dataViews,charts,esUiShared,bfetch,data,savedObjects,presentationUtil,expressionShape,expressionRevealImage,expressionRepeatImage,expressionMetric,expressionImage,customIntegrations,home,searchprofiler,painlessLab,grokdebugger,management,watcher,licenseManagement,advancedSettings,spaces,security,savedObjectsTagging,reporting,canvas,lists,ingestPipelines,fileUpload,encryptedSavedObjects,dataEnhanced,cloud,snapshotRestore,eventLog,actions,alerting,triggersActionsUi,transform,stackAlerts,ruleRegistry,visualizations,visTypeXy,visTypeVislib,visTypeVega,visTypeTimelion,visTypeTagcloud,visTypeTable,visTypePie,visTypeMetric,visTypeMarkdown,tileMap,regionMap,expressionTagcloud,expressionMetricVis,console,graph,fleet,indexManagement,remoteClusters,crossClusterReplication,indexLifecycleManagement,dashboard,maps,dashboardMode,dashboardEnhanced,visualize,visTypeTimeseries,rollup,indexPatternFieldEditor,lens,cases,timelines,discover,osquery,observability,discoverEnhanced,dataVisualizer,ml,uptime,securitySolution,infra,upgradeAssistant,monitoring,logstash,enterpriseSearch,apm,savedObjectsManagement,indexPatternManagement]"}efk-kibana-1  | {"type":"log","@timestamp":"2023-01-20T01:43:47+00:00","tags":["info","plugins","taskManager"],"pid":7,"message":"TaskManager is identified by the Kibana UUID: 5ef7fd18-4086-4ce8-8d43-91e99b733fcb"}efk-kibana-1  | {"type":"log","@timestamp":"2023-01-20T01:43:48+00:00","tags":["warning","plugins","security","config"],"pid":7,"message":"Generating a random key for xpack.security.encryptionKey. To prevent sessions from being invalidated on restart, please set xpack.security.encryptionKey in the kibana.yml or use the bin/kibana-encryption-keys command."}efk-kibana-1  | {"type":"log","@timestamp":"2023-01-20T01:43:48+00:00","tags":["warning","plugins","security","config"],"pid":7,"message":"Session cookies will be transmitted over insecure connections. This is not recommended."}

6.测试访问elasticsearch

测试本地访问elasticsearch

[root@jeven efk]# curl 192.168.3.166:9200{  "name" : "jeven",  "cluster_name" : "myes",  "cluster_uuid" : "BfacKp5xRBqNKvus7q3tIA",  "version" : {    "number" : "7.17.5",    "build_flavor" : "default",    "build_type" : "docker",    "build_hash" : "8d61b4f7ddf931f219e3745f295ed2bbc50c8e84",    "build_date" : "2022-06-23T21:57:28.736740635Z",    "build_snapshot" : false,    "lucene_version" : "8.11.1",    "minimum_wire_compatibility_version" : "6.8.0",    "minimum_index_compatibility_version" : "6.0.0-beta1"  },  "tagline" : "You Know, for Search"}

image.png