The last mile: Why a semantic layer matters

Credit unions are ramping up their adoption of Big Data & Analytics. The frustration with valuable business information being trapped in multiple data silos has boiled over. Projects to integrate these disparate data sources into a single source of truth are being launched every day. Now, many credit unions have the ability to slice and dice their data and drill from individual transactions up to aggregated totals.

Behind the scenes, a complex data structure has been constructed that is specially tuned to allow a multitude of questions to be asked. Experienced technical report writers who understand this architecture are able to mine the depths of these structures to handle a multitude of questions that were previously almost impossible to answer.

The next natural step in the Big Data & Analytics evolution is pushing out ad hoc reporting to the broader organizational audience. Yet, this turns out to be more difficult than anticipated. Less technical personnel are faced with a blizzard of arcane data names and a mountain of hard-to-understand tables. They are forced to be dependent on the technical report writers for any new reporting. The nimble nature of the Big Data & Analytics opportunity is lost for them.

These less technical employees make up the majority of the credit union’s workforce and, if they could do reporting themselves, would be the most valuable users of the Big Data & Analytics tools. Yet, those who are most in need of the information are the least able to obtain meaning from this incredible resource. It is as if a magnificent transportation system was built but ended a mile short of the where the travelers live.

Does this mean credit unions are forced to rely solely on scarce technical resources to accomplish anything with Big Data & Analytics? Fortunately, there is a solution that can fulfill the goal of “self-service” reporting. A semantic layer can be developed to bridge the “last mile”.

The definition of “semantic” relates to the meaning of something. Non-technical users have a difficult time deriving meaning through direct access of the data structure. The names of data elements are unfamiliar and the structure is purposely designed to perform efficiently using programming languages such as SQL.

A semantic layer is created by the very technical folks who were previously forced to be the bottleneck. These are the experts who thoroughly understand the complexities of the data structure. They also know how to translate technical jargon into recognizable business names and organize them into a format that seems logical to the front-line business person. The result is non-technical users can create queries and build reports based on terminology that is familiar and meaningful.

Creating a Semantic Layer is a virtual necessity if a credit union hopes to push the power of Big Data & Analytics out to the wider organization. However, it is far from trivial to construct. Despite the name, a Semantic Layer is not a monolithic object. It is better described as a technique for bringing the data closer to the user in a meaningful way.

Also, multiple versions of a Semantic Layer usually need to be built based on subject matter. For example, a Semantic Layer of loan data would make it easier for ad hoc questions to be asked by the lending team. The Accounting and Finance group will need a Semantic Layer focused on the general ledger. Operations will need a version that focuses on branches. Marketing may need both a Semantic Layer that focuses on campaign management and one the is geared for geo-demographic analysis.

To design and build these varieties of versions, developers will need very solid requirements from the business. Business users must be highly engaged in specifying the types of reporting and analytics they plan to do. This includes the required levels of granularity, the types of time series, and typical calculations used.

Once in place, the beauty of the Semantic Layer is readily apparent. In most cases, typical reporting and analytics tools can easily access the underlying data. This is a pleasant surprise to many organizations that previously purchased expensive reporting software only to find there was a considerable amount of technical configuration necessary to make it work.

It will take a strong organizational commitment to build the Semantic Layer, especially after an already large effort to implement the overall data structure upon which it is based. However, if credit unions want to complete the “Last Mile”, the Semantic Layer is the road that will lead to truly realizing the promise of Big Data & Analytics.