Hacker Newsnew | past | comments | ask | show | jobs | submitlogin

This is fascinating. I’m struggling to understand how that was causing such a large difference in the output. Is the “autoparser” vulnerable to injections somehow? How do you distinguish between user text, model text, and metadata, or is there ambiguity in the parsing?


My take, as someone who just read through the github issues conversation.

The model was outputting reasoning traces that were supposed to lead to tool calls. So the model might do something like:

    <think>
    I should use a tool
    </think>

    ... should make the tool call here
But a \n was slipping through from the last line of the reasoning trace so the parser was generating:

   <think>
   I should use a tool

   </think>
And that extra new line before the closing </think> would occasionally trigger the model to question itself with an "Actually ... " digression. In long running conversations this would end up looping because the "Actually ..." part would reason it should call a tool, then a new trailing \n would trigger an "Actually ..." and then it ends up in a loop.


> I’m struggling to understand how that was causing such a large difference in the output.

It is incremental, the more a pattern appears in the context, the more likely it was to continue appearing in future turns.

So the model was likely trained to end a reasoning trace with a single linefeed and a `</think>`. It was also likely trained that two consecutive linefeeds sometimes produce an "Actually..." sequence.

So you can think of it as:

- the first time the reasoning trace was parsed, the two trailing linefeeds followed by </think> were added to the context by the template.

- next time the model was finishing a reasoning block, it added an extra linefeed instead of just closing directly with </think>. This slightly increases the chance that the next token will begin an "actually" sequence instead of closing.

- If it caused an actually, that was added to the context, further increasing the chance of a self correction at the end of the thinking block. The more self correction paragraphs are added, the higher the chance that following turns will have more.

- Eventually it can result in a state where it enters that loop forever (or at least for a very long time).

> Is the “autoparser” vulnerable to injections somehow?

The autoparser was (and still is) incorrectly parsing a trailing linefeed as part of a reasoning block. The were two ways to fix this, both of which must be implemented for the fix to be complete IMO:

- fix the autoparser definition to ensure remove surrounding whitespace is not returned as part of the text blocks

- trim leading/trailing whitespace in the encoding phase, so it fixes bugs or even "injections" where the client deliberately adds the whitespace to trigger problems.

For this specific issue, the maintainer later fixed by trimming the extra linefeed before passing to the template.

> How do you distinguish between user text, model text, and metadata, or is there ambiguity in the parsing?

That is model specific. Ultimately, a token stream is being produced and parsed by the inference engine, and each model uses different tokens/formats. The goal of the autoparser engine was to simplify the creation of parsers for new models by inferring the delimiter tokens from the chat template.

The llama.cpp API server returns pre-parsed data, so clients don't need to do any parsing to know what is a thinking block, a text block or a tool call.




Guidelines | FAQ | Lists | API | Security | Legal | Apply to YC | Contact

Search: