Vekil Bekmyradov’s paper had won the Best In-Practice Paper Award at ESEM 2026, the 20th International Symposium on Empirical Software Engineering and Measurement, held in Munich on 8 and 9 October as part of the Empirical Software Engineering International Week (ESEIW) 2026. Vekil presented the paper in Munich himself and accepted the award at the ceremony in the Augustiner-Keller.
The paper, “Evaluating LLM-Based Test Generation for a Large Industrial C++ Database System. A Case Study on SAP HANA”, was accepted in the Software Engineering in Practice Track and is published open access in LIPIcs Volume 394. Its authors are Vekil Bekmyradov, Thomas Bach, Alexander Berndt, Noah C. Puetz, Bartosz Bogacz and myself. The software behind the study is available on Zenodo.
From a master thesis at SAP to an award-winning paper
Vekil came to Gummersbach for the Automation & IT master programme, a well-established and very popular English-taught Master of Engineering at TH Köln that combines computer science, automation engineering, and data science. For his master thesis he joined SAP in Walldorf as an external student and registered the topic “Evaluating the Effectiveness of LLM-Generated Unit Tests in the Context of Industrial DBMS” in October 2025. I supervised the thesis as first examiner, and Noah C. Pütz from my group acted as second examiner.
Within the four months allowed for the thesis, Vekil carried out a large-scale empirical evaluation of state-of-the-art large language models on an industrial software engineering task, inside SAP HANA, a proprietary database system of roughly 40 million lines of code. Producing rigorous, reproducible results in a codebase of that scale demands methodological discipline, systems fluency and judgment about what can actually be measured. Vekil showed all three with a degree of independence I rarely see at the master’s level. He is among the strongest students I have worked with in recent years. The thesis is available open access from TH Köln: “Evaluating the Effectiveness of LLM-Generated Unit Tests in the Context of Industrial DBMS”. A first analysis of the results, “LLMs taking shortcuts in test generation: A study with SAP HANA and LevelDB” by Vekil, Noah and myself, appeared on arXiv in April 2026. The award-winning ESEM paper followed.
This is exactly the kind of result our cooperation with SAP makes possible. A student gets access to a production database system of serious scale. SAP colleagues Thomas Bach and Bartosz Bogacz contribute deep knowledge of that system and join the paper as co-authors. The university contributes the empirical methodology and the supervision. Alexander Berndt (Heidelberg University) completed the team. I am grateful to SAP for a partnership that lets our students do research on real industrial software rather than on toy examples.
What the paper found
Large Language Models are increasingly used to generate unit tests, and published results on open-source projects look promising. Vekil and his co-authors asked whether this holds for a large, closed-source C++ codebase that the models have never seen. On SAP HANA, the generated tests fell well short of the quality they reached on the open-source key-value store LevelDB. Iterative repair of compilation errors helped, but its effect saturated after a few iterations, and further rounds tended to produce tests that compile but assert very little. The practical lesson is clear:
results from open-source benchmarks do not transfer automatically to industrial code, and the context handed to the model matters a great deal.
What this means for TH Köln and thk-ai
A best paper award at a leading empirical software engineering conference is peer recognition of the highest kind, and it went to a paper with two TH Köln authors on it besides myself. For the thk-ai research cluster, it confirms the path we have chosen: applied AI research with industry partners, judged by rigorous empirical standards.
Vekil is now a PhD candidate at TU Delft in the Netherlands. Seeing a former student begin a doctorate with an award like this in hand is one of the real rewards of this job. Congratulations, Vekil, and congratulations to all co-authors. I wish you every success for your PhD and for the career ahead.
P.S: About software tests
Software tests are a hot topic in the AI community right now. In a widely discussed Substack note of 8 October 2026, Kun Chen argues that tests which coding agents write on their own, without being asked, are of little value. His evaluation on the DeepSWE benchmark found that agents caused no more regressions when they were not allowed to write tests at all, and that they finished faster. His key point is that a test is only an intermediate representation of the intended behaviour. If the test was written autonomously by a model, it is no more trustworthy than the code it checks, and when it fails nobody can tell whether the test or the code is wrong. Mutation testing, he adds, makes tests more sensitive but does not solve this: a more trusted source of truth is needed. Curated, human-written test suites remain an asset. Our paper reaches a compatible conclusion from the other side: on an unseen industrial codebase, generated tests compile after enough repair rounds but often assert very little. Both say the same thing. Tests generated by an LLM need a human who decides what the software is supposed to do.
Publications
- V. Bekmyradov, T. Bach, A. Berndt, N. C. Puetz, B. Bogacz, T. Bartz-Beielstein: Evaluating LLM-Based Test Generation for a Large Industrial C++ Database System. A Case Study on SAP HANA. ESEM 2026, LIPIcs Vol. 394, Paper 95. https://doi.org/10.4230/LIPIcs.ESEM.2026.95
- V. Bekmyradov, N. C. Pütz, T. Bartz-Beielstein: LLMs taking shortcuts in test generation: A study with SAP HANA and LevelDB. arXiv:2604.14437, 2026. https://doi.org/10.48550/arXiv.2604.14437
- V. Bekmyradov: Evaluating the Effectiveness of LLM-Generated Unit Tests in the Context of Industrial DBMS. Master thesis, TH Köln, 2026. https://doi.org/10.57684/COS-1441
