Capturing Heap Dumps, Thread Dumps, and JFR Profiles in Kubernetes

March 6, 2025 · 2 minutes

· 0

Goal

This guide explains how to capture heap dumps, thread dumps, and Java Flight Recorder (JFR) profiles in a Kubernetes-based Appsmith deployment. These diagnostic files provide insights into memory usage, performance bottlenecks, and Out-of-Memory (OOM) errors.

Overview

If your Appsmith instance is running in Kubernetes and experiencing performance issues such as high CPU usage, memory leaks, or frequent crashes, capturing diagnostic files can help identify the root cause.

Key Diagnostic Files:

Heap Dump – Captures a snapshot of application memory, helping diagnose Out-of-Memory (OOM) errors and memory leaks.
Thread Dump – Provides a snapshot of all active threads, their states, and stack traces to analyze deadlocks and high CPU usage.
Java Flight Recorder (JFR) Profile – Records execution patterns, thread activity, and performance bottlenecks, useful for debugging unresponsive applications.

Capturing a Thread Profile (JFR) in Kubernetes

A JFR profile helps analyze thread activity, execution patterns, and potential bottlenecks. This is useful for debugging high CPU usage, unresponsive applications, or frequent OOM-related crashes.
1. Get all running pods:
```
kubectl get pods
```
2. Identify the problematic pod (e.g., experiencing frequent restarts) and access it:
```
kubectl exec -it <pod_name> bash
```
3. Inside the pod, check if the profiling scripts exist:
```
ls -lrta
```
  You should see:
```
thread-profile-start.sh
thread-profile-stop.sh
```
4. Start capturing the thread profile:
```
./thread-profile-start.sh
```
5. Stop profiling after reproducing the issue:
```
./thread-profile-stop.sh
```
  Note: If the pod restarts, profiling will automatically stop.
6. Locate the generated JFR file inside the pod at:
```
/appsmith-stacks/heap_dumps/ad-hoc/<pod_name>/thread-profile/
```
  The file will be named in the format profile-<pod_name>.jfr.
7. Copy the JFR file from the pod to your local system:
```
kubectl cp --retries -1 <pod_name>:/appsmith-stacks/heap_dumps/ad-hoc/<pod_name>/thread-profile/<profile-filename> destination_path/<intended_file_name>
```
8. Share the file with the support team for further analysis.
Capturing a Heap Dump in Kubernetes

A heap dump captures a snapshot of the application’s memory, helping diagnose OOM errors and memory leaks.
1. Get all running pods:
```
kubectl get pods
```
2. Identify and access the problematic pod:
```
kubectl exec -it <pod_name> bash
```
3. Inside the pod, check if the heap dump script exists:
```
ls -lrta
```
  You should see:
```
record-heap-dump.sh
```
4. Capture the heap dump:
```
./record-heap-dump.sh
```
  You should see output similar to:
```
Dumping heap to filename ...
Heap dump file created [1285426410 bytes in 9.988 secs]
```
5. Locate the generated heap dump file inside the pod at:
```
/appsmith-stacks/heap_dumps/ad-hoc/<pod_name>/heap-profile/
```
  The file will be named in the format <file_name>.log.
6. Copy the heap dump file from the pod to your local system:
```
kubectl cp <pod_name>:/appsmith-stacks/heap_dumps/ad-hoc/<pod_name>/heap-profile/<file_name>.log destination_path/<intended_file_name>
```
7. Share the file with the support team for further analysis.
Capturing a Thread Dump in Kubernetes

A thread dump captures all active threads, their states, and stack traces, helping diagnose high CPU usage, deadlocks, and performance issues.
1. Get all running pods:
```
kubectl get pods
```
2. Identify and access the problematic pod:
```
kubectl exec -it <pod_name> bash
```
3. Inside the pod, check if the thread dump script exists:
```
ls -lrta
```
  You should see:
```
record-thread-dump.sh
```
4. Capture the thread dump:
```
./record-thread-dump.sh
```
5. Locate the generated thread dump file inside the pod at:
```
/appsmith-stacks/heap_dumps/ad-hoc/<pod_name>/thread-profile/
```
6. Copy the thread dump file from the pod to your local system:
```
kubectl cp <pod_name>:/appsmith-stacks/heap_dumps/ad-hoc/<pod_name>/thread-profile/<file_name> destination_path/<intended_file_name>
```
7. Share the file with the support team for further analysis.

Conclusion

Capturing thread profiles (JFR), heap dumps, and thread dumps in Kubernetes helps diagnose OOM errors, high CPU usage, deadlocks, and performance issues in an Appsmith deployment. Use tools like Eclipse Memory Analyzer (MAT) for heap dumps and JDK Mission Control for JFR files to analyze the collected data.

For further assistance, share the logs with the Appsmith support team along with details of the issue. Our team will help analyze the data and provide recommendations for resolution.