Steering the Backtracking Vector

Oct 13, 2025 by windailys.com 43 views

Experiment: How Redundant is the Thought Process in R1-Distilled Models?

Introduction

R1-distilled models have gained significant attention in recent years due to their ability to provide accurate and efficient results. However, one of the key aspects of these models is their thought process, which can be redundant and time-consuming. In this experiment, we aim to investigate the redundancy of the thought process in R1-distilled models and explore ways to optimize it.

Experiment Setting

For this experiment, we used the DeepSeek-R1-Distill-Llama-8B model and the GSM8K dataset. We selected samples with backtracking counts between 4 and 10, as these samples exhibited a high degree of redundancy in their thought process. Our goal was to analyze the confidence of the model at each backtracking instance and determine whether the model's thought process was faithful or if it was using the process as a sketchpad to derive the answer.

Results

Our results showed that the confidence of the answer increased as the model backtracked more. However, when prompted again at the 1st instance, the accuracy was 0.9 and did not increase after further backtracking and arriving at the same solution. This suggests that the model's thought process is not always necessary to arrive at the correct answer.

Confidence of Answer Increases with Backtracking

We observed that the confidence of the answer increased as the model backtracked more. This is evident from the logits, which are normalized across the backtracking within each sample. The increase in confidence suggests that the model is becoming more confident in its answer as it continues to backtrack.

Importance of Thought Process

Judging from the x-axis at 0, where the model immediately generates the answer without thinking, it fails to get the answer correct most of the time. However, it is not clear if the thought process is faithful or if the model uses the thought process as a sketchpad to derive the answer. This highlights the importance of understanding the thought process of R1-distilled models and how it can be optimized.

TODO

Based on our results, we have identified several areas for further investigation:

Steering the Backtracking Vector: We found a backtracking vector which, if steered, can cause the model to either backtrack or wrap up its thoughts. On samples where the model still gets the answer wrong, we can steer it to backtrack and see if it can get the answer correct.
Understanding the Attention Pattern: When the model backtracks, which tokens is it looking at? Does it look at the most recent tokens and think that "oh the steps are not concrete, I should think some more"? We can use attribution methods such as finding important attention heads and looking at the attention pattern to answer this question.
Early Exit: We can use the backtracking vector to early exit the thought process when the model starts backtracking. We can compare the cosine sim of the steering vec with the residual at the effective steering layer to determine when the model is about to backtrack.

Conclusion

In conclusion, our experiment has shown that the thought process of R1-distilled models can be redundant and time-consuming. However, by analyzing the confidence of the answer and the attention pattern, we can identify areas for optimization and improve the efficiency of the model. We hope that this experiment will contribute to a better understanding of the thought process of R1-distilled models and provide insights for future research.

Future Work

Based on our results, we propose the following future work:

Investigate the Faithfulness of the Thought Process: We need to investigate whether the thought process of R1-distilled models is faithful or if it is using the process as a sketchpad to derive the answer.
Optimize the Thought Process: We can use the insights gained from this experiment to optimize the thought process of R1-distilled models and improve their efficiency.
Develop New Models: We can develop new models that are designed to optimize the thought process and improve the efficiency of R1-distilled models.

References

[1] DeepSeek-R1-Distill-Llama-8B model
[2] GSM8K dataset
[3] Backtracking vector
[4] Attention pattern
[5] Cosine sim
[6] Residual at the effective steering layer

Appendix

The following are the images used in this experiment:

Q&A: Experiment: How Redundant is the Thought Process in R1-Distilled Models?

Q: What is the main goal of this experiment?

A: The main goal of this experiment is to investigate the redundancy of the thought process in R1-distilled models and explore ways to optimize it.

Q: What model and dataset were used in this experiment?

A: We used the DeepSeek-R1-Distill-Llama-8B model and the GSM8K dataset in this experiment.

Q: What was the selection criteria for the samples used in this experiment?

A: We selected samples with backtracking counts between 4 and 10, as these samples exhibited a high degree of redundancy in their thought process.

Q: What was the result of the experiment in terms of the confidence of the answer?

A: Our results showed that the confidence of the answer increased as the model backtracked more. However, when prompted again at the 1st instance, the accuracy was 0.9 and did not increase after further backtracking and arriving at the same solution.

Q: What does this suggest about the thought process of R1-distilled models?

A: This suggests that the thought process of R1-distilled models is not always necessary to arrive at the correct answer.

Q: What is the importance of understanding the thought process of R1-distilled models?

A: Understanding the thought process of R1-distilled models is important because it can help us identify areas for optimization and improve the efficiency of the model.

Q: What are some potential ways to optimize the thought process of R1-distilled models?

A: Some potential ways to optimize the thought process of R1-distilled models include steering the backtracking vector, understanding the attention pattern, and early exiting the thought process.

Q: What is the backtracking vector and how can it be used to optimize the thought process?

A: The backtracking vector is a vector that can be used to steer the model's thought process. By steering the backtracking vector, we can cause the model to either backtrack or wrap up its thoughts.

Q: What is the attention pattern and how can it be used to optimize the thought process?

A: The attention pattern refers to the way the model focuses its attention on different parts of the input. By understanding the attention pattern, we can identify areas where the model is focusing its attention and optimize the thought process accordingly.

Q: What is early exiting the thought process and how can it be used to optimize the thought process?

A: Early exiting the thought process refers to the ability to stop the model's thought process early and still arrive at the correct answer. By early exiting the thought process, we can save computational resources and improve the efficiency of the model.

Q: What are some potential future directions for this research?

A: Some potential future directions for this research include investigating the faithfulness of the thought process, optimizing the thought process, and developing new models that are designed to optimize the thought process.

Q: What are some potential applications of this research?

A: Some potential applications of this research include improving the efficiency of R1-distilled models, developing more accurate and efficient models, and improving the performance of models in real-world applications.

Q: What are some potential challenges and limitations of this research?

A: Some potential challenges and limitations of this research include the complexity of the thought process, the difficulty of understanding the attention pattern, and the need for large amounts of data to train the model.

Q: What are some potential next steps for this research?

A: Some potential next steps for this research include collecting more data, developing new models, and testing the models in real-world applications.