[k8s.io] Kubelet Container Manager [Serial] Validate OOM score adjustments once the node is setup pod infra containers oom-score-adj should be -998 and best effort container's should be 1000 {E2eNode Suite}
Problem
https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/kubelet-serial-gce-e2e-ci/782/ Failed: [k8s.io] Kubelet Container Manager [Serial] Validate OOM score adjustments once the node is setup pod infra containers oom-score-adj should be -998 and best effort container's should be 1000 {E2eNode Suite} [code block]
Unverified for your environment
Select your OS to check compatibility.
1 Fix
Fix OOM Score Adjustments for Kubelet Pod Infra Containers
The failure in the E2E test is due to incorrect OOM score adjustments for pod infra containers and best effort containers. Specifically, the pod infra containers should have an OOM score adjustment of -998 to ensure they are less likely to be killed by the OOM killer, while best effort containers should have an OOM score adjustment of 1000, making them more likely to be terminated under memory pressure. This discrepancy can occur due to misconfiguration in the Kubelet's container manager settings or issues in the container runtime integration.
Awaiting Verification
Be the first to verify this fix
- 1
Verify Kubelet Configuration
Check the Kubelet configuration file to ensure that the container manager is set up correctly to manage OOM scores. Look for the 'oomScoreAdj' settings.
bashcat /var/lib/kubelet/config.yaml | grep oomScoreAdj - 2
Update OOM Score Adjustments
If the OOM score adjustments are not set correctly, update the Kubelet configuration to enforce the correct values for pod infra containers and best effort containers. Modify the config.yaml file to include the following settings.
yamloomScoreAdj: infra: -998 bestEffort: 1000 - 3
Restart Kubelet Service
After modifying the Kubelet configuration, restart the Kubelet service to apply the changes. This will ensure that the new OOM score adjustments take effect.
bashsystemctl restart kubelet - 4
Deploy Test Pods
Deploy test pods with different resource requests and limits to validate the OOM score adjustments. Ensure that you have both pod infra containers and best effort containers in your test.
yamlkubectl apply -f test-pods.yaml - 5
Check OOM Scores
After deploying the test pods, check the OOM score adjustments for the containers to confirm they are set correctly. Use the following command to inspect the OOM scores.
bashkubectl exec <pod-name> -- cat /proc/<pid>/oom_score_adj
Validation
To confirm the fix worked, ensure that the OOM score adjustments for pod infra containers are -998 and for best effort containers are 1000. Additionally, run the E2E tests again to verify that they pass without errors related to OOM score adjustments.
Sign in to verify this fix
Environment
Submitted by
Alex Chen
2450 rep