AI impact on developer productivity and data engineering trends

Recent industry reports and research highlight the evolving role of artificial intelligence in software engineering and data management. A study involving 331 developers at Microsoft and data from over 500 engineering organizations indicates that while AI tools like Cursor and Claude have increased median weekly pull request throughput by 75.5% over four quarters, this speed does not necessarily equate to increased value. The research notes that the share of time dedicated to new capabilities versus maintenance has only risen by two percentage points.

Researchers have identified a trend termed the ‘hollow orchestrator,’ where developers may become faster at approving work without improving their ability to judge its quality, leading to concerns regarding skill atrophy.

In the data engineering sector, focus remains on Open Table Formats, such as the Apache Iceberg REST specification, and updates within the Amazon data ecosystem. Additionally, a meta-analysis of over 1,600 papers on prompt engineering suggests that techniques like providing exemplars and requesting data in JSON or XML formats can improve LLM performance, though the industry expects these workflows to eventually become integrated directly into models.