How we are solving our biggest development challenge

Striking the balance between dynamic data and large-scale, high-fidelity maps is a challenge that the Planwisely team knows well.

Planwisely gives users access to complex spatial datasets and the ability to visualise and analyse them on the fly. By giving users greater flexibility in their analyses, Planwisely can help uncover relevant insights that can guide a planner’s decision making and deliver better planning outcomes.

Our team’s goal has always been to deliver datasets that maintain a high level of interactivity and functionality without users having to suffer long wait times (due to a dataset’s size) but in the past we too have inevitably limited the interactivity of a dataset’s aggregations – things like filters, sums, averages, mins, maxes and other ways of interrogating spatial data to extract insights.

‍

To grapple with this problem, we’ve been working on our next-generation map engine and specifically targeting very large datasets – large in terms of details, coverage and complexity of the attached data.

One of our more recent datasets is an Australia-wide transport dataset, with 2.8 million ‘links’ – segments of a route such as roads, PT line sections and more – and 1.1 billion rows of data associated with those links.  

To visualise this, we:

• Selected 33,000 links
• Selected 400,000 rows of link data
• Summed the data across the user-specified demographics and travel purposes  
• Then joined the resulting sums and links to be rendered in our vector maps

Best of all was that it did all this in less than 1 second (and without grinding to a halt from concurrent users)! To do this, we built and profiled a variety of prototypes to test multiple architectures, data formats and AWS technologies until we found a solution that was performant and scalable enough to bring to production, while still being capable of the size of datasets we wanted to throw at it.

‍

The thing we’re most excited about is our map data engine’s ability to perform calculations on the fly. The above dataset has link volume data for 3 modes and 24 scenarios, broken down by 8 purposes and 4 demographics. Without the ability to perform calculations on the fly, maps of comparisons between scenarios would result in a dataset that’s 300x larger. Similarly, allowing any combination of purposes and demographics would also result in a dataset 300x larger, too. By performing last-mile calculations on our datasets, we can create even more expressive maps to highlight relevant insights.

We’re still working on our map data service, with plenty of ideas to optimise performance and storage costs, while also building out more sophisticated analysis and reporting capabilities. The map data engine and our associated data pipeline is implemented in Haskell, which we use to support quality throughout the pipeline and clearly define the capabilities of the service as we make regular improvements.  

Our map engine’s growing ability to quickly deliver dynamic data at highly granular geometries is exciting for its functionality and future potential. In addition to giving Planwisely users a more intuitive and functional experience, the opportunities it unlocks to visualise even the largest transport and planning datasets (and support mind-blowing visualisations around people and place) is an enticing prospect – both as a boon for Planwisely users and as a challenge for our developers.