Scaling geospatial data processing without blocking synchronisation
The challenge
A platform was developed for Rabobank’s Acorn programme to collect and process geographical data from agricultural plots around the world. Data was collected using mobile applications and synchronised later. Extensive data quality checks, including preventing overlapping plots, caused timeouts and scalability issues when multiple users uploaded their data simultaneously.
My contribution
Together with a business architect, I designed a backend architecture using NestJS, CQRS, and PostGIS. I took responsibility for a substantial part of the implementation, working alongside a junior developer who joined later. I also provided guidance and knowledge transfer. We separated synchronisation from validation using event-driven processing, and I coordinated integrations with an external software provider.
The solution
An architecture in which data collection and computationally intensive validation could take place independently. This addressed a major synchronisation bottleneck and created room for further scaling.