JOCA: Task-Driven Joint Optimisation of Camera Hardware and Adaptive Camera Control Algorithms

The quality of captured images significantly impacts the performance of downstream perception tasks. Recent works that co-design camera systems with perception tasks have demonstrated improved task performance. However, prior approaches primarily focus on optimising fixed camera parameters determined at manufacturing, whereas many parameters, such as exposure settings, require adaptive control at runtime. This paper presents a method that jointly optimises camera hardware and adaptive camera control algorithms alongside downstream vision tasks. We propose a unified optimisation framework that combines gradient-based and derivative-free methods to support continuous and discrete parameters, non-differentiable image formation processes, and a neural network-based adaptive camera control algorithm. To handle non-differentiable rendering of some image effects, such as motion blur, we propose DF-Grad, a hybrid optimisation method that supervises the neural adaptive control algorithm using signals from the derivative-free optimiser, in addition to unsupervised task-driven learning. Experiments show that the proposed method outperforms baselines that optimise camera hardware and camera control algorithm separately, particularly under challenging conditions such as low light and fast motion. We demonstrate that joint optimisation of both static camera hardware parameters and adaptive control algorithms leads to improved perception task performance, offering a unified approach to task-driven camera system design.