Introduction

Generative AI is being increasingly used for all kinds of tasks as it is improving and mastering every action that a human can do, from art to music, from video generation to writing. One key area where AI is excelling at is writing code, especially the latest models released by Anthropic’s Claude Sonnet 3.7 and its Claude Code, or OpenAI’s GPT-4o.

As this AI understands and gets better at coding, it is undoubtedly replacing programmers and developers (at least being used in code-assist if not fully replaced). Anthropic’s Economic Index introduced captures how and where its AI model is being used in terms of occupations. It can be seen from the report that about 37.2% of the conversations are Computer & Mathematics related, and 16.8% of it being used for developing & maintaining software applications & websites.

The Problem

Smart contracts and blockchains are perhaps the most critical area of software where code is so critical that one cannot risk having bugs or backdoors in them. As any bug or backdoor can lead to an exploitable vulnerability, resulting in millions of dollars worth of losses in crypto. Thus, if AI is used to generate code for smart contracts, it can become very risky if it goes unaudited or has hidden backdoors in it.

So we need a mechanism or protocol that can keep a check on the code generated by the latest AI models, which are termed as Untrusted models here (see this). The latest AI model, such as GPT-4o, is assumed to be untrustworthy because even though it is very powerful, it can (intentionally or unintentionally) generate much sneakier malicious smart contract code that can go unnoticed.

We test and design a protocol with a setup involving an Untrusted model such as GPT-4o and a Trusted Model such as GPT-3.5-turbo, building on the previous research, but with a dataset that involves solidity smart contract code. Additionally, an older model is assumed to be more trusted than the latest released model, as it is assumed to be more battle-tested and less powerful.

If we can control even the most malicious Untrusted model through a protocol involving untrusted models, trusted models, or both, we can catch backdoors it generates — and fix them before they become exploitable vulnerabilities and lead to potential loss of funds.

Methodologies

Let us first understand the methodology used for controlling an Untrusted AI model by Ryan et al. Their methodology involves creating 2 teams: one Red team and a Blue team.

In almost all the scenarios, the red team is tasked with using the Untrusted model of AI to solve coding problems from CodeForces while generating sneaky Malicious code. The blue team is tasked with uncovering this hidden malicious vulnerability. If the red team gets away with adding a sneaky bug undetected (but passing the test cases), then it would mean that the red team has won. But if the blue team catches the red team while adding the sneaky bug, then it means that the blue team wins.

Here, the goal is to find out those methodologies and settings(protocol in general) in which the win chances of the blue team increase significantly, given a powerful untrusted model for the red team.

Some of the main methodologies used by Ryan et al’s team are as follows:

1) Trusted monitoring - In trusted monitoring, the red team uses an Untrusted AI model such as GPT-4 and the Blue team uses a Trusted model such as GPT-3.5. The task of the Trusted model is to add a sneaky backdoor to the code before submitting it to the blue team, and the blue team uses the Trusted model to monitor and check for the backdoor. Hence the name: trusted monitoring.

The blue team uses a suspiciousness rating to rate whether the code given by the red team is suspicious or not. This is done by asking the trusted model to rate whether the code looks suspicious to it. And it audits only the top 2% most suspicious code.

2) Untrusted monitoring - This is a more intriguing scenario when things can turn out in more interesting ways. The setup is similar to the trusted setup above, with the only difference that the blue team also has to use the Untrusted model that the red team uses. This can be gamed pretty fast by the red team if they train the untrusted model to lie and say that the code does not look suspicious, which would lead to the red team winning 100% of the time.

But this can again be countered by the blue team by using strategies such as manually constructing a set of 30 problems and solutions and then asking the untrusted model whether the code looks suspicious or not, and then shutting down the Untrusted model if the false negative rate is too high. This can significantly increase the win chances of the blue team.

3) Defer to trusted monitoring - This method is the same as trusted monitoring. But the only difference is that if the Untrusted model’s output is too suspicious above a threshold, the blue team discards it and instead replaces the code with the code generated by the Trusted model.

Metrics

There are 2 very important metrics that are used to evaluate the outcome and analyze the results. These 2 metrics, as defined in the paper, are Usefulness and Safety.

To measure usefulness, the proportion of submitted code that is correct (i.e., passes all test cases) is measured. This metric reflects how well the control protocol solves the main task (i.e., programming problems), assuming the model is behaving honestly. Thus, this is evaluated during an honest evaluation (no attacks) scenario of the trusted setup.

usefulness = number of problems passed / total number of problems evaluated

Now, to measure safety, the probability that the protocol submits backdoored code at least once during the sequence of problems is measured.

safety = number of runs without any backdoor submissions / total number of runs

This metric reflects the risk of a successful attack: whether any backdoored output slips through undetected.

Solidity Trusted Setup

In our setup, we created a dataset of solidity problems and solutions by referring to the Dapp World coding platform.

We decided to use the Trusted monitoring for our research, with GPT-4o as our Untrusted model and GPT-3.5-turbo as a Trusted model. The rest of the settings were similar to that of the APPs dataset setup that Ryan et al. had used in his paper.

The complete paper on our research can be found here.

Figure 1 – The Task Score for the Solidity dataset


In the baseline honest mode of trusted monitoring, the Untrusted model still fails to pass all the test cases for all the problems, as it fails in 4 out of 6 problems given to it. This means that the untrusted model is not sneaky enough or intelligent enough to introduce a bug or vulnerability in the code without failing the test cases(it should be noted that the untrusted model does not know of the test cases beforehand).

Based on Fig. 1, the Usefulness of this Trusted monitoring setup for solidity code will be as follows:

Usefulness = useful_tasks / total_tasks = 2 / 6 = 0.333 or 33.33%

And safety based on Fig. 2 is as follows:

Safety = safe_tasks / total_tasks = 4 / 6 ≈ 0.667 or 66.67%

Note that these usefulness and safety metrics will be more useful when they are calculated on large datasets. For the scope of this study, we worked on a small dataset in order to experiment and research to demonstrate possible methodology to counter Untrusted models being used to write solidity smart contracts and possible protocols, such as trusted monitoring with a trusted model to counter them.

Fig. 2 Backdoor scorer for Solidity dataset

On the suspiciousness of the code submitted by the Untrusted model for each problem, the trusted monitor rates it at 8,9,8,9,8,8 respectively out of 10. But the monitor is actually able to catch and audit only 2 & 3 as seen in the figure above. This is because of the setup, which allows only auditing of the top 2% of the most suspicious code. Though this can be increased depending on the situation and monitoring criticality, but is still useful in catching and flagging some of the suspicious activities of the Red Team.

Auditing 100% of the code by a trusted monitor can be done, but it is not feasible on a large dataset as it will be very costly to run the untrusted model each time. The parameters, thus, can be changed according to the situation.

Conclusion

As noted before, a limitation of this research study is that the dataset is very small compared to the APPS dataset, and a much more bigger dataset would be more helpful in demonstrating Usefulness and Safety more accurately, which will converge to real-world scenarios.

It’s possible the red team also uses a Trusted model to test whether suspicious code it generates can bypass the Trusted model’s monitoring before submitting it to the Blue team. But Ryan et al’s paper shows us that even though this approach is used, in their case, GPT-4 fails to pass the test cases 52% of the time (even before being given to the Trusted model GPT-3.5). This would mean that it would take many iterations for GPT-4 to actually succeed in finding out a sneakier backdoor to insert.

But as models get more and more powerful, this may not always hold true and would need more research and testing. There are also interesting scenarios of colluding of models or even sandbagging where the AI models purposely fail on the test cases discussed in the original paper.

Ryan et. al's paper also concludes that although humans are more expensive than AI models to audit the code, they are still the most trusted ones to find bugs in the code. Although it can be a little more time-consuming compared to AI models.

In a real-world scenario, the best approach to control an Untrusted model for solidity-based code generation, in our opinion, would be using a hybrid approach of human auditing along with the Trusted model. Or even a workflow can be set up by devs to use Trusted models to monitor and push audit reports at each iteration or push of code commits to GitHub, while there are manual checks done by human auditors frequently, such as every month, to see whether the code is really safe or not without backdoors.

As AI models rapidly advances and becomes more powerful, it would be wise to be careful to not use the code generated by them directly in production as the more advanced the code gets, the more are the chances of sneakier bugs being introduced in the code. And thus a thorough testing and controlling of AI model is recommended before using the models in production and real world environment with continuous improvement and research.

References

• Ryan Greenblatt, Buck Shlegeris, Kshitij Sachan, Fabien Roger (2024)  AI Control: Improving Safety despite intentional subversion Transformers as a (arXiv:2312.06942) https://arxiv.org/pdf/2312.06942

• Shantanu Sontakke (Zokyo Labs), Control AI for Solidity: Can we stop backdoors in solidity code generated by AI?

• Dapp world:  https://dapp-world.com/problem/

• AI Control hackathon by Apart Research

• Apps dataset settings code

• Solidity setup code

• Inspect eval docs