[k8s.io] Kubelet [Serial] [Slow] [k8s.io] regular resource usage tracking resource tracking for 100 pods per node {Kubernetes e2e suite}
Problem
https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/kubernetes-soak-continuous-e2e-gce/6512/ Failed: [k8s.io] Kubelet [Serial] [Slow] [k8s.io] regular resource usage tracking resource tracking for 100 pods per node {Kubernetes e2e suite} [code block] Previous issues for this test: #26982
Unverified for your environment
Select your OS to check compatibility.
1 Fix
Optimize Kubelet Resource Tracking for High Pod Density
The Kubelet is experiencing performance degradation when tracking resource usage for a high number of pods (100 pods per node). This can be attributed to inefficient resource tracking algorithms or insufficient resource allocation for the Kubelet itself, leading to slow response times and potential timeouts during resource updates.
Awaiting Verification
Be the first to verify this fix
- 1
Increase Kubelet Resource Limits
Adjust the resource limits for the Kubelet to ensure it has sufficient CPU and memory to handle the load of 100 pods per node. This can be done by modifying the Kubelet deployment configuration.
yamlapiVersion: v1 kind: Pod metadata: name: kubelet spec: containers: - name: kubelet resources: limits: cpu: "2000m" memory: "2Gi" - 2
Tune Kubelet Configuration Parameters
Modify the Kubelet configuration parameters to optimize performance for high pod counts. Specifically, adjust the '--max-pods' parameter to ensure it can handle the desired number of pods efficiently.
bash--max-pods=100 - 3
Enable Kubelet Caching
Enable caching for Kubelet resource usage tracking to reduce the overhead of frequent updates. This can be done by setting the '--kube-reserved' and '--system-reserved' flags appropriately.
bash--kube-reserved=cpu=500m,memory=1Gi --system-reserved=cpu=500m,memory=1Gi - 4
Monitor Kubelet Performance
After applying the changes, monitor the Kubelet's performance metrics to ensure that resource tracking is functioning as expected without significant delays.
bashkubectl top nodes - 5
Run E2E Tests
Execute the Kubernetes end-to-end tests again to validate that the changes have resolved the performance issues and that resource tracking is stable for 100 pods per node.
bashkubectl apply -f test/e2e/framework/test.yaml
Validation
Confirm that the Kubelet can successfully track resource usage for 100 pods without significant delays or timeouts. Check the logs for any errors and ensure that the end-to-end tests pass without failures related to resource tracking.
Sign in to verify this fix
Environment
Submitted by
Alex Chen
2450 rep