Angus Black, Director, Barnowl Data Solutions (BDS)
I have spent a significant portion of my career dealing with the complexities of data management, particularly in the insurance industry.
When we talk about data standards, we are really talking about the backbone of how we communicate information accurately and effectively between systems. In the insurance world, this is vital. Without clear standards, the data we share would be subject to misinterpretation, making it impossible to maintain regulatory compliance or provide accurate insights.
At its core, a data set consists of three key elements. First, you need a list of data fields or elements that meet the business’s specific needs. Each field must have a well-defined purpose and clear instructions on its data type—whether it is a number, text, or some other form of data—and any restrictions that apply, such as length limits. Second, any data standard must include a minimum set of required fields essential to keeping the system functional. Some fields may be optional, but without these core elements, the data exchange simply will not work. Lastly, there are validation rules that guide how the data should be collected, how it can be verified for accuracy, and what dependencies exist between fields.
In the insurance sector, the need for such standards is driven largely by regulation. While there are always slight variations in how different companies approach their business, regulation ensures that the foundational elements remain consistent across the industry. The challenge, though, is that things change rapidly. You might achieve uniformity on day one, but by day 100, regulatory requirements could have shifted, and companies must continuously adapt their systems to stay compliant.
One of the biggest hurdles we face is the sheer variety of platforms and systems in use. Some companies rely on off-the-shelf solutions, others build custom systems, and many use a combination of both. The diversity is so broad that it was once humorously described to me as being like “a box of Smarties.” Each system has its quirks and getting them all to communicate effectively is a challenge. One of the main obstacles we encounter is repeatability. If the data coming from the same source is inconsistent from one day to the next, it becomes nearly impossible to manage. We also deal with ambiguity—take “gross written premium,” for example. Does it include VAT or not? Is the broker’s commission factored in? Every data element must be clearly defined to avoid confusion.
Empower Your Decisions by Streamlining Your Data
With BARNOWL Insurance Data Warehouse: The Ultimate Data Solution for the Insurance Industry in South Africa.
For more information on how to empower your decisions by streamlining your data, visit bds.co.za
Flexibility is crucial in our approach. Initially, we were too rigid, expecting every source system to perfectly align with our standards. That was a mistake. Now, we are much more pragmatic, collecting what we can and handling data transformations on our end rather than expecting them to be fixed at the source. In some cases, especially with older, historical data, it is impossible to meet current standards, and we need to account for those variations.
Another major challenge we face is dealing with problematic data. Even when you have established your own standards and educated your clients, issues inevitably arise. The key to overcoming these challenges is to agree on consistent metrics for comparison. We assess the data using system-generated bordereaux, which provides a structured format for reviewing the data. By applying repeatable rules, we ensure consistency month over month. It is critical that our internal processes are just as repeatable; you do not want to analyse the data with 50 rules one month and then suddenly shift to 100 rules the next month. This would only create frustration for everyone involved.
In terms of automation, most of our processes are indeed automated. We are dealing with millions of records each month, so manual checks just are not feasible. However, despite the high level of automation, there are still times when human intervention is required. This is where having a team that understands both the data and the nuances of the insurance industry comes into play. Sometimes, it is necessary to flag issues to the client when something does not look right. Automation takes care of the bulk of the work, but a skilled eye is still needed to ensure the data we are processing is accurate and meaningful.
At the end of the day, managing data in the insurance industry is a balancing act. It is about maintaining the right level of flexibility without sacrificing accuracy, and about continuously adapting to the ever-evolving regulatory landscape. Data does not just serve as a regulatory tool—it has far-reaching applications that can help businesses make smarter decisions and streamline operations. As we continue to develop our data services, it is exciting to see just how much potential there is for growth in this space.
There are countless use cases for data beyond just ticking regulatory boxes, and we are only beginning to scratch the surface of what is possible.