当前位置： 首页 > news >正文

后台风格网站百度官方网站

news 2025/8/15 12:09:04
后台风格网站,百度官方网站,门户网站开发注意哪些,跨境电商平台有哪些知乎1.LogStash性能不稳定（某天关闭后，再次启动就非常慢），所以后面我们用Filebeat。2.先禁用 # geoip { # source > "clientip" # }3.在生产中要是用nignx服务或tomcat服务我们用EFK架构就可以排查技巧观察点 LogS…
1.LogStash性能不稳定（某天关闭后，再次启动就非常慢），所以后面我们用Filebeat。2.先禁用
#  geoip {
#    source => "clientip"
#  }3.在生产中要是用nignx服务或tomcat服务我们用EFK架构就可以

排查技巧观察点

LogStash报错分析

在这里插入图片描述

在这里插入图片描述

在这里插入图片描述

java日志查找

在这里插入图片描述

LogStash

数据映射之带宽统计案例
见昨日内容
多实例
#补充命令：
nc（netcat）命令是一个功能强大的网络工具，用于在网络连接上进行读写数据。它可以在 TCP 或 UDP 协议下工作，被广泛用于网络调试、端口扫描、数据传输等多种任务

1.启动第一个实例
[11:29:55 root@elk2:/etc/logstash/conf.d]#logstash -f 01-stdin-to-stdout.conf2.启动第二个实例
[10:36:25 root@elk2:/etc/logstash/conf.d]#logstash -f 02-file-to-stdout.conf --path.data /tmp/logstash-data#注意实验事项
1.如果不指定数据路径，则Logstash默认的数据路径在安装目录的data目录下。比如: "/usr/share/logstash/data/"
2.如果同一个节点启动了多个Logstash实例，不指定数据路径，则会报错：Logstash could not be started because there is already another instance using the configured data directory.  If you wish to run multiple instances, you must change the "path.data" setting.`------------------------------实验开始位置-----------------------------------------`	1.启动Logstash实例1
[11:31:36 root@elk2:/etc/logstash/conf.d]#vim 09-multiple-tcp-to-es.conf
input { # 监听本地的tcp端口，用于接受数据tcp {port => 7777}}  filter {mutate {remove_field => ["@version"]}}output { stdout { codec => rubydebug } elasticsearch {hosts => ["10.0.0.91:9200","10.0.0.92:9200","10.0.0.93:9200"]index => "linux-elfk-multiple-tcp-%{+yyyy.MM.dd}"}}[11:35:07 root@elk2:/etc/logstash/conf.d]#logstash -rf 09-multiple-tcp-to-es.conf 2.启动Logstash实例2 
[11:35:00 root@elk2:/etc/logstash/conf.d]#vim 10-multiple-beats-to-es.conf
input { beats {port => 8888}}  filter {mutate {remove_field => [ "@version","log","tags","agent","ecs","input" ]}}output { stdout { codec => rubydebug } elasticsearch {hosts => ["10.0.0.91:9200","10.0.0.92:9200","10.0.0.93:9200"]index => "linux-elfk-multiple-beats-%{+yyyy.MM.dd}"}}[11:36:43 root@elk2:/etc/logstash/conf.d]#logstash -rf 10-multiple-beats-to-es.conf --path.data /tmp/logstash-data3.启动filebeat 
[11:28:53 root@elk3:/etc/filebeat]#vim 10-tcp-to-logstash.yaml
filebeat.inputs:
- type: tcphost: "0.0.0.0:9000"output.logstash:hosts: ["10.0.0.92:8888"][11:50:12 root@elk3:/etc/filebeat]#rm -rf /var/lib/filebeat/
[11:50:15 root@elk3:/etc/filebeat]#filebeat -e -c 10-tcp-to-logstash.yaml4.发送测试数据到实例1
[11:28:44 root@elk1:~]#echo www.baidu.com | nc 10.0.0.92 77775.发送测试数据到filebeat 
[11:38:17 root@elk1:~]#echo www.jingdong.com | nc 10.0.0.93 9000

在这里插入图片描述

在这里插入图片描述

多分支
1.启动Logstash实例 
[12:00:26 root@elk2:/etc/logstash/conf.d]#vim 11-if-logstash.conf
input { tcp {type => "tcp"port => 7777}beats {port => 8888type => "beats"}}  filter {if [type] == "tcp" {mutate {remove_field => ["@version"]}} else {mutate {remove_field => [ "@version","log","tags","agent","ecs","input" ]}}}output { stdout { codec => rubydebug } if [type] == "tcp" {elasticsearch {hosts => ["10.0.0.91:9200","10.0.0.92:9200","10.0.0.93:9200"]index => "linux-elfk-if-tcp-%{+yyyy.MM.dd}"}}else {elasticsearch {hosts => ["10.0.0.91:9200","10.0.0.92:9200","10.0.0.93:9200"]index => "linux-elfk-if-beats-%{+yyyy.MM.dd}"}}}[12:04:48 root@elk2:/etc/logstash/conf.d]#logstash -rf 11-if-logstash.conf 2.启动filebeat实例
[11:50:12 root@elk3:/etc/filebeat]#rm -rf /var/lib/filebeat/
[11:50:15 root@elk3:/etc/filebeat]#filebeat -e -c 10-tcp-to-logstash.yaml3.发送测试数据到Filebeat
[11:28:44 root@elk1:~]#echo 11 | nc 10.0.0.92 7777
[11:38:17 root@elk1:~]#echo 22 | nc 10.0.0.93 9000echo 好好学习 | nc 10.0.0.92 7777
echo 天天k8s | nc 10.0.0.93 9000

在这里插入图片描述

在这里插入图片描述

pipeline
1.修改Logstash的配置文件[14:59:26 root@elk2:/etc/logstash/conf.d]#vim 12-pipeline-tcp-to-es.conf
input { tcp {port => 7777}}  filter {mutate {remove_field => ["@version"]}}output { stdout { codec => rubydebug } elasticsearch {hosts => ["10.0.0.91:9200","10.0.0.92:9200","10.0.0.93:9200"]index => "linux-elfk-pipeline-tcp-%{+yyyy.MM.dd}"}}[14:59:48 root@elk2:/etc/logstash/conf.d]#vim 13-pipeline-beats-to-es.conf 
input { beats {port => 8888}}  filter {mutate {remove_field => [ "@version","log","tags","agent","ecs","input" ]}}output { stdout { codec => rubydebug } elasticsearch {hosts => ["10.0.0.91:9200","10.0.0.92:9200","10.0.0.93:9200"]index => "linux-elfk-pipeline-beats-%{+yyyy.MM.dd}"}}2.修改pipeline配置文件
[14:44:49 root@elk2:/etc/logstash]#vim pipelines.yml 
- pipeline.id: tcppath.config: /etc/logstash/conf.d/12-pipeline-tcp-to-es.conf- pipeline.id: beatspath.config: /etc/logstash/conf.d/13-pipeline-beats-to-es.conf3.创建pipeline的软连接
[15:01:26 root@elk2:/etc/logstash]#mkdir /usr/share/logstash/config
[15:01:33 root@elk2:/etc/logstash]#ln -svf /etc/logstash/pipelines.yml /usr/share/logstash/config/4.启动Logstash 
[15:01:35 root@elk2:/etc/logstash]#logstash -r5.启动filebeat 
[15:03:06 root@elk3:/etc/filebeat]#rm -rf /var/lib/filebeat/
[15:04:04 root@elk3:/etc/filebeat]#filebeat -e -c 10-tcp-to-logstash.yaml6.发送测试数据
[12:14:09 root@elk1:~]#echo 好好学习 | nc 10.0.0.92 7777
[14:59:20 root@elk1:~]#echo 天天k8s | nc 10.0.0.93 9000

在这里插入图片描述

在这里插入图片描述

自定义正则
1.准备自定义的patterns
[16:09:58 root@elk2:/etc/logstash/conf.d]#mkdir patterns[16:10:01 root@elk2:/etc/logstash/conf.d]#vim patterns/linux
YEAR \d{4}
SCHOOL [a-z]{9}
CLASS [a-z0-9]{7}2.编写配置文件 
[root@elk92 conf.d]# cat 14-tcp-grok_custom_patterns-es.conf 
input { tcp {port => 7777}}  filter {grok {# 指定自定义patterns目录，会自动加载该目录下的所有文件内容相关的patterns模式patterns_dir => ["/etc/logstash/conf.d/patterns"]match => { "message" => "%{YEAR:year}北京欢迎您: https://www.%{SCHOOL:School}.com 班级: %{CLASS:cLaSs}" }}}output { stdout { codec => rubydebug } # elasticsearch {
#      hosts => ["10.0.0.91:9200","10.0.0.92:9200","10.0.0.93:9200"]
#      index => "linux-elfk-grok-custom-%{+yyyy.MM.dd}"
#  }}[16:12:10 root@elk2:/etc/logstash/conf.d]#logstash -rf 14-tcp-grok_custom_patterns-es.conf 3.发送测试数据 
[root@elk93 ~]# echo "北京欢迎您: https://www.beijing.com 班级: linux" | nc 10.0.0.92 77774.接收的数据效果如下：
{"port" => 48110,"@timestamp" => 2024-10-30T08:13:44.725Z,"host" => "10.0.0.93","@version" => "1","tags" => [[0] "_grokparsefailure"],"message" => "北京欢迎您: https://www.beijing.com 班级: linux"
}
JVM优化
1.查看Logstash实例启动时占用的heap堆内存大小
[16:15:05 root@elk2:~]#ps -ef | grep logstash
root        2026    1053 99 16:15 pts/0    00:00:36 /usr/share/elasticsearch/jdk/bin/java -Xms1g -Xmx1g2.修改Logstash的堆内存大小
[16:15:02 root@elk2:~]#vim /etc/logstash/jvm.options 
## JVM configuration# Xms represents the initial size of total heap space
# Xmx represents the maximum size of total heap space#-Xms1g
#-Xmx1g-Xms256m
-Xmx256m3.启动Logstash实例测试 
[16:14:53 root@elk2:/etc/logstash/conf.d]#logstash -rf 14-tcp-grok_custom_patter4.查看JVM占用内存情况
[16:16:57 root@elk2:~]#ps -ef | grep logstash
root        2123    1053 99 16:17 pts/0    00:00:05 /usr/share/elasticsearch/jdk/bin/java -Xms128m -Xmx128m#注意：出现java.lang.OutOfMemoryError: Java heap space
表示内存不足，仅需要添加内存即可，工作中如何不设置Logstash的堆内存大小，默认为1GB。
自定义Nginx的json日志格式
1.清空nginx访问日志
[16:38:12 root@elk3:~]# > /var/log/nginx/access.log2.修改nginx的配置文件
[16:38:13 root@elk3:~]#vim /etc/nginx/nginx.conf ...# 注释原有的配置行# access_log /var/log/nginx/access.log;log_format nginx_json '{"@timestamp":"$time_iso8601",''"host":"$server_addr",''"clientip":"$remote_addr",''"SendBytes":$body_bytes_sent,''"responsetime":$request_time,''"upstreamtime":"$upstream_response_time",''"upstreamhost":"$upstream_addr",''"http_host":"$host",''"uri":"$uri",''"domain":"$host",''"xff":"$http_x_forwarded_for",''"referer":"$http_referer",''"tcp_xff":"$proxy_protocol_addr",''"http_user_agent":"$http_user_agent",''"status":"$status"}';access_log  /var/log/nginx/access.log  nginx_json;
...[16:39:49 root@elk3:~]#nginx -t
nginx: the configuration file /etc/nginx/nginx.conf syntax is ok
nginx: configuration file /etc/nginx/nginx.conf test is successful[16:39:56 root@elk3:~]#systemctl reload nginx3.访问测试 
[15:52:19 root@elk1:~]#curl 10.0.0.93
<!DOCTYPE html>
<html>
<head>
<title>Welcome to nginx!</title>
<style>body {width: 35em;margin: 0 auto;font-family: Tahoma, Verdana, Arial, sans-serif;}
</style>
</head>
<body>
<h1>Welcome to nginx!</h1>
<p>If you see this page, the nginx web server is successfully installed and
working. Further configuration is required.</p><p>For online documentation and support please refer to
<a href="http://nginx.org/">nginx.org</a>.<br/>
Commercial support is available at
<a href="http://nginx.com/">nginx.com</a>.</p><p><em>Thank you for using nginx.</em></p>
</body>
</html>4.查看日志格式
[16:40:12 root@elk3:~]#tail -f /var/log/nginx/access.log
{"@timestamp":"2024-10-30T16:40:26+08:00","host":"10.0.0.93","clientip":"10.0.0.91","SendBytes":612,"responsetime":0.000,"upstreamtime":"-","upstreamhost":"-","http_host":"10.0.0.93","uri":"/index.nginx-debian.html","domain":"10.0.0.93","xff":"-","referer":"-","tcp_xff":"-","http_user_agent":"curl/7.81.0","status":"200"}5.使用filebeat采集nginx访问日志
[16:42:49 root@elk3:/etc/filebeat]#vim 11-log-to-es.yamlfilebeat.inputs:
- type: logpaths:- /var/log/nginx/access.log# 解析message字段的JSON格式，并将键值对放在顶级字段中json.keys_under_root: true#output.console:
#  pretty: trueoutput.elasticsearch:hosts: ["http://10.0.0.91:9200","http://10.0.0.92:9200","http://10.0.0.93:9200"] index: "linux-nginx-json-%{+yyyy.MM.dd}" setup.ilm.enabled: false
setup.template.name: "linux-nginx"
setup.template.pattern: "linux-nginx*"
setup.template.overwrite: false
setup.template.settings:index.number_of_shards: 3index.number_of_replicas: 0[16:43:54 root@elk3:/etc/filebeat]#rm -rf /var/lib/filebeat/
[16:44:18 root@elk3:/etc/filebeat]#filebeat -e -c 11-log-to-es.yaml6.kibana查看数据
[16:40:26 root@elk1:~]#curl 10.0.0.93/beijing.html

在这里插入图片描述

在这里插入图片描述

程序名词

- SDK:英文全称为"Software development kit"，表示软件开发工具包，指的是程序员所需的开发环境工具包，比如Java，Go，C++，Python，Ruby，PHP等程序员依赖的开发环境。- JDK:英文全称为: Java Development Kit ，表示Java开发的工具包集合，针对Java开发员的软件开发工具包。JDK就是典型的Java程序员所使用的SDK。用于编写"*.java"文件，对应的编译程序是"javac"工具。- JRE:英文全称为： Java Runtime Environment，表示Java的运行环境，相比JDK而言更轻量级，软件包会更小，只能保证程序能够运行，缺少开发的工具包。- JVM:英文全称为： Java Virtual Machine，表示Java的虚拟机，一般用于执行Java程序编译后的字节码文件"*.class"，对应的运行程序是"java"工具。#JVM虚拟机又细分为多个存储区域：
- 对外内存 
- 方法区
- ...
- 堆内存- 伊甸区：表示对象产生的内存。- 幸存区：经过GC时在伊甸区没有结束生命周期的对象会经过幸存区过滤到年老区。- 年老区：长期存在的对象。

项目篇: ElasticStack的RBAC实战

2.配置ES集群加密
2.1 生成证书文件
[17:01:21 root@elk1:~]#/usr/share/elasticsearch/bin/elasticsearch-certutil cert -out /etc/elasticsearch/elastic-certificates.p12 -pass ""
........
Note: Generating certificates without providing a CA certificate is deprecated.A CA certificate will become mandatory in the next major release.Certificates written to /etc/elasticsearch/elastic-certificates.p12This file should be properly secured as it contains the private key for 
your instance.This file is a self contained file and can be copied and used 'as is'
For each Elastic product that you wish to configure, you should copy
this '.p12' file to the relevant configuration directory
and then follow the SSL configuration instructions in the product guide.[17:33:28 root@elk1:~]#ll /etc/elasticsearch/elastic-certificates.p12 
-rw------- 1 root elasticsearch 3596 Oct 30 17:33 /etc/elasticsearch/elastic-certificates.p122.2 同步证书文件到其他节点
[17:33:52 root@elk1:~]#scp /etc/elasticsearch/elastic-certificates.p12 10.0.0.92:/etc/elasticsearch/   
[17:34:10 root@elk1:~]#scp /etc/elasticsearch/elastic-certificates.p12 10.0.0.93:/etc/elasticsearch/2.3 修改ES集群的配置文件
[17:34:59 root@elk1:~]#vim /etc/elasticsearch/elasticsearch.yml 
........
# 在最后一行添加以下内容
xpack.security.enabled: true
xpack.security.transport.ssl.enabled: true
xpack.security.transport.ssl.verification_mode: certificate
xpack.security.transport.ssl.keystore.path: elastic-certificates.p12
xpack.security.transport.ssl.truststore.path: elastic-certificates.p122.4 同步ES配置文件到其他节点
[17:35:25 root@elk1:~]#scp /etc/elasticsearch/elasticsearch.yml 10.0.0.92:/etc/elasticsearch/[17:35:44 root@elk1:~]#scp /etc/elasticsearch/elasticsearch.yml 10.0.0.93:/etc/elasticsearch/2.5 "ES集群所有节点"修改证书文件的权限信息
chown elasticsearch:elasticsearch /etc/elasticsearch/elastic-certificates.p122.6 "ES集群所有节点"重启ES服务
systemctl restart elasticsearch.service 2.7 生成随机密码【请保留密码，下一个步骤要用！】
[17:37:53 root@elk1:~]#/usr/share/elasticsearch/bin/elasticsearch-setup-passwords auto
Initiating the setup of passwords for reserved users elastic,apm_system,kibana,kibana_system,logstash_system,beats_system,remote_monitoring_user.
The passwords will be randomly generated and printed to the console.
Please confirm that you would like to continue [y/N]yChanged password for user apm_system
PASSWORD apm_system = 4Am23FzpVOxMeKK4NVYdChanged password for user kibana_system
PASSWORD kibana_system = CJlMQpD1MxNNK6m4FVI9Changed password for user kibana
PASSWORD kibana = CJlMQpD1MxNNK6m4FVI9Changed password for user logstash_system
PASSWORD logstash_system = nvSvRzHYsQbhQJu2pi3JChanged password for user beats_system
PASSWORD beats_system = tGiTVdjelFHGnzo8e0ZfChanged password for user remote_monitoring_user
PASSWORD remote_monitoring_user = tSW5yxDEo1pnn3lLLJghChanged password for user elastic
PASSWORD elastic = OSUzxjrIEq9HRcbj4bKB2.8 验证集群是否正常【注意，elastic是ES集群的管理员用户，密码根据自己的系统生成使用即可】
[17:43:06 root@elk2:~]#curl -u elastic:OSUzxjrIEq9HRcbj4bKB 10.0.0.91:9200/_cat/nodes
10.0.0.92 36 83 10 0.15 0.24 0.16 cdfhilmrstw * elk2
10.0.0.93 27 82  9 0.04 0.13 0.09 cdfhilmrstw - elk3
10.0.0.91 13 96  5 0.00 0.08 0.13 cdfhilmrstw - elk1#配置kibana访问ES加密集群
1.修改kibana的配置文件
[17:45:03 root@elk1:~]#vim /etc/kibana/kibana.yml
...........
# 根据你自己环境生成的"kibana_system"的随机密码做相应的修改即可
elasticsearch.username: "kibana_system"
elasticsearch.password: "CJlMQpD1MxNNK6m4FVI9"2.重启kibana服务
[17:51:28 root@elk1:~]#systemctl restart kibana
[17:51:48 root@elk1:~]#ss -tnl|grep  5601
LISTEN 0      511          0.0.0.0:5601      0.0.0.0:*   3.访问kibana的WebUI
10.0.0.91:5601将"elastic"的用户名和密码进行修改！

在这里插入图片描述

修改用户密码

在这里插入图片描述

在这里插入图片描述

在这里插入图片描述

总结

- 数据映射ELFK架构关于带宽统计案例	* 
- Logstash的多实例 				*** 
- logstash的多分支语法 			** 
- logstash的pipeline技术 		*****
- grok自定义匹配模式patterns	* 
- logstash的JVM配置优化		*** 
- 自定义nginx的访问日志格式	** 
- ES集群加密配置			*****
- kibana访问ES加密集群并修改elastic管理员密码	*****
- 故障排查技巧	*****- 1.日志界别INFO WARNFATAL- 2.找关键字ERROR- 3.对于java日志查找方式从日志的最下网上看，找at上面没有at的行就是报错信息- 4.日志查找方式- 通过安装目录	/var/log/elasticsearch/ES集群名称.log - 启动脚本查看systemctl cat 启动脚本服务的名称
http://www.ds6.com.cn/news/111033.html

相关文章：

  • 萝岗网站建设竞价培训课程
  • 零食销售网站开发与设计长春网站开发公司
  • 山东高端网站建设方案百度移动端关键词优化
  • 建设造价信息网站点点站长工具
  • 宁蒗县政府门户网站建设吴江seo网站优化软件
  • java开源cms商品标题seo是什么意思
  • 网站认证是什么意思营销活动方案模板
  • 网站想建设子站24小时自助下单平台网站便宜
  • 个人兼职做建设网站网络营销的六大功能
  • 郑州建站模板源码软件开发公司简介
  • 哈尔滨网站制作专业百度指数十年
  • b2b网站建设排名大数据营销是什么
  • 如何用java做网站视频百度推广怎么才能效果好
  • 网站建设哪里好点郑州品牌网站建设
  • 网站建设目标是什么意思网络营销策划方案案例
  • 做网站应怎么缴税北京seo供应商
  • WordPress 先登录广州网站seo
  • 自己做的网站用别的电脑怎么访问查询网 网站查询
  • 短视频培训要多少学费电子商务seo是什么意思
  • 滁州网站设计收录优美图片app
  • 做网站挂谷歌广告赚钱吗网站如何优化排名
  • 用什么软件做公司网站如何推广引流
  • 任经理 徐州网站建设seo排名点击软件运营
  • 卡片式主题wordpress南京seo网站优化推广
  • 新乡网站建设多少钱百度最新秒收录方法2022
  • 网站以下内容未做缓存品牌广告语
  • 电子电工技术工程网搜索引擎优化自然排名的优点
  • 自贡市城市建设投资开发集团有限公司网站智慧软文网站
  • 做网站的应该怎么发广告西安优化排名推广
  • 营销策划公司职位seo排名工具