claudegpt-code-logic-debugger-v0.1

Code Logic Debugger v0.1

Hardware requirements for ChatGPT GPT-4o level inference speed for the models in this repo: >=24 GB VRAM.

Note: The following results are based on my day-to-day workflows only on an RTX 3090. My goal was to run private models that could beat GPT-4o and Claude-3.5 in code debugging and generation to ‘load balance’ between OpenAI/Anthropic’s free plan and local models to avoid hitting rate limits, and to upload as few lines of my code and ideas to their servers as possible.

An example of a complex debugging scenario is where you build library A on top of library B that requires library C as a dependency but the root cause was a variable in library C. In this case, the following workflow guided me to correctly identify the problem.

<br>

Throughput

IQ here refers to Importance Matrix Quantization. For performance comparison against regular GGUF, please read this Reddit post. For more info on the techique, please see this GitHub discussion.

<br>

Personal Preference Ranking

Evaluated on two programming tasks: debugging and generation. It may be a bit subjective. DeepSeekV2 Coder Instruct is ranked lower because DeepSeek's Privacy Policy says that they may collect "text input, prompt" and there's no way around it.

Code debugging/editing prompt template used:

<code>
<current output>
<the problem description of the current output>
<expected output (in English is fine)>
<any hints>
Think step by step. Solve this problem without removing any existing functionalities, logic, or checks, except any incorrect code that interferes with your edits.

<br>

Generation Kwargs

Balanced Mode:

generation_kwargs = {
    "max_tokens":8192,
    "stop":["<|EOT|>", "</s>", "<｜end▁of▁sentence｜>", "<eos>", "<|start_header_id|>", "<|end_header_id|>", "<|eot_id|>"],
    "temperature":0.7,
    "stream":True,
    "top_k":50,
    "top_p":0.95,
}

Precise Mode:

generation_kwargs = {
    "max_tokens":8192,
    "stop":["<|EOT|>", "</s>", "<｜end▁of▁sentence｜>", "<eos>", "<|start_header_id|>", "<|end_header_id|>", "<|eot_id|>"],
    "temperature":0.0,
    "stream":True,
    "top_p":1.0,
}

Qwen2 7B:

generation_kwargs = {
    "max_tokens":8192,
    "stop":["<|EOT|>", "</s>", "<｜end▁of▁sentence｜>", "<eos>", "<|start_header_id|>", "<|end_header_id|>", "<|eot_id|>"],
    "temperature":0.4,
    "stream":True,
    "top_k":20,
    "top_p":0.8,
}

Other variations in temperature, top_k, and top_p were tested 5-8 times per model too, but I'm sticking to the above three.

<br>

New Discoveries

The following are tested in my workflow, but may not generalize well to other workflows.

• In general, if there's an error in the code, copy pasting the last few rows of stacktrace (without the library stacktrace) to the LLM seems to work.
• Adding "Reflect." after a failed attempt at code generation sometimes allows Claude-3.5-Sonnet to generate the correct version.
• If GPT-4o reasons correctly in its first response and the conversation is then continued with GPT-4-mini, the mini model can maintain comparable level of reasoning/accuracy as GPT-4o.

<br>

License

A reminder that codestral-22b-v0.1-IQ6_K.gguf should only be used for non-commercial projects.

Please use Qwen2-7b-Instruct bf16 and AutoCoder.IQ4_K.gguf as alternatives for commericial activities.

<br>

Download

pip install -U "huggingface_hub[cli]"

Commercial use:

huggingface-cli download FredZhang7/claudegpt-code-logic-debugger-v0.1 --include "AutoCoder.IQ4_K.gguf" --local-dir ./

Non-commercial (e.g. testing, research, personal, or evaluation purposes) use:

huggingface-cli download FredZhang7/claudegpt-code-logic-debugger-v0.1 --include "codestral-22b-v0.1-IQ6_K.gguf" --local-dir ./