Kubernetes-Operator配置自动发现新增ServiceMonitor的NameSpace

最近在监控k8s当中的Jenkins 的时候,新增ServiceMonitor总是不显示,这里的不显示指的是不在Prometheus-Operator中显示,我认为这其中一定有问题,于是查阅了大量的资料,终于从中找到了答案, 原因竟是Prometheus-Operator在安装的时候默认只会自动发现default,kube-system,monitoring这三个名称空间下新建的ServiceMonitor,而我的ServiceMonitor又是新建在别的名称空间下所以他无法进行自动发现
更奇怪的是,网上对这个问题的记录文章几乎为0,我根本找不到,所以才写下这个笔记,供看到的人借鉴!

1. 监控其他 namespace 中的 endpoint 资源

需要做的操作

创建新增命名空间的 role，用于获取监控的信息。
将创建的role绑定到 monitoring 命名空间中的 prometheus-k8s SA。

kubectl create ns test

namespace=test

cat <<EOF | kubectl apply -f - 
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  name: prometheus-k8s
  namespace: ${namespace}
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: Role
  name: prometheus-k8s
subjects:
- kind: ServiceAccount
  name: prometheus-k8s
  namespace: monitoring

---
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  name: prometheus-k8s
  namespace: ${namespace}
rules:
- apiGroups:
  - ""
  resources:
  - services
  - endpoints
  - pods
  verbs:
  - get
  - list
  - watch
- apiGroups:
  - extensions
  resources:
  - ingresses
  verbs:
  - get
  - list
  - watch
EOF
COPY

2. 监控其他 namespace 中的 serviceMonitor 资源

serviceMonitorNamespaceSelector 命名空间的标签匹配，不指定时，只匹配自身命名空间的资源
serviceMonitorSelector serviceMonitor的标签匹配，不指定时，只匹配自身命名空间的资源

修改 Prometheus 资源配置

apiVersion: monitoring.coreos.com/v1
kind: Prometheus
metadata:
  labels:
    prometheus: k8s
  name: k8s
  namespace: monitoring
spec:
...
# 增加 ns 匹配的标签
  serviceMonitorNamespaceSelector:
    matchLabels:
      serviceMonitor: prometheus
      
# 或者 增加下面的匹配，用来匹配 serviceMonitor
  serviceMonitorSelector:
    matchLabels:
      serviceMonitor: prometheus
COPY

添加命名空间的标签

for ns in default kube-system monitoring test; do 
  kubectl patch ns $ns --patch '{"metadata":{"labels":{"serviceMonitor": "prometheus" } } }'
done
COPY

添加 serviceMonitor 的标签

kubectl patch -n test servicemonitor demo-app --patch '{"metadata":{"labels":{"serviceMonitor":"prometheus"}}}' --type=merge
COPY

测试

cat <<EOF | kubectl-test apply -f - 
apiVersion: apps/v1
kind: Deployment
metadata:
  name: ingress-demo-app
  namespace: test
  labels:
    app: ingress-demo-app
spec:
  replicas: 2
  selector:
    matchLabels:
      app: ingress-demo-app
  template:
    metadata:
      labels:
        app: ingress-demo-app
        namespace: test
    spec:
      containers:
      - name: whoami
        image: traefik/whoami:v1.6.1
        ports:
        - containerPort: 80
---
apiVersion: v1
kind: Service
metadata:
  name: ingress-demo-app
  namespace: test
  labels:
    app: ingress-demo-app
spec:
  type: ClusterIP
  selector:
    app: ingress-demo-app
  ports:
    - name: http
      port: 80
      targetPort: 80
---
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: ingress-demo-app
  namespace: test
  labels:
    app: ingress-demo-app
  annotations:
    kubernetes.io/ingress.class: nginx
spec:
  rules:
  - host: app.demo.com
    http:
      paths:
      - path: /
        pathType: Prefix
        backend:
          service:
            name: ingress-demo-app
            port:
              number: 80
---
apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  labels:
    name: ingress-demo-app
  name: ingress-demo-app
  namespace: test
spec:
  endpoints:
  - port: http
    path: /health
    interval: 5s
  selector:
    matchLabels:
      app: ingress-demo-app
EOF

希望