Author Archives: infolib

Today I will answer three questions that will help you to get traction, and succeed with your metadata initiatives.

• 1. What are the key elements needed?
• 2. Where does metadata fit in?
• 3. How to get started?

Knowing the answers to these three questions can literally pay big dividends for your organization. With a modest budget, a good starting point, and a clear idea of the problem you are trying to solve. You can be successful in establishing a metadata management program for your company.

#1 You need buy-in , budget, and timeframe.

This does not necessarily mean you need to boil the Atlantic Ocean. What about a smaller initiative to get things started? A novel idea. The alternative is a high risk of failure. Or even worse, a project that suffers from the inertia of never getting of the ground in the first place.

#2 Where does metadata management it fit in?

Taxonomy + Ontology = Zoology

Again, where do we get started? Do we need to comply with the most sophisticated theories out there? Is having basic metadata impact analysis a good value for low cost and quick turn-around? Is this not a good place to start?

Identify a pain point and solve that pain only. This is the best place to start. To identify where the metadata fits in requires only one thing, identify a problem you are trying to solve.

#3 How to get started?

Today I identified the three main aspects that will help you to better approach your metadata management initiatives.

Conclusion

Remember that a clear understanding of the problem you are trying to solve holds the key; and that you will probably be more likely to integrate metadata management in your company by finding a project to which metadata is an integral part. Example : Master Data – Customer Data focus.

Metadata for Data Governance and Stewardship

Master Data

For compliance reasons, you generally have to be able to prove that the metadata accurately describes the master data. The easiest way to furnish this proof is to show that the processes that handle the master data are derived directly from the metadata or vice versa. For example, the mappings and transformations done by the ETL tools might be driven directly from the metadata, or the ETL tools might populate the metadata repository whenever a transformation pipeline is created. Business rules should be taken directly from the metadata, whenever possible. This is especially important in MDM, because the business rules often determine which of several alternate values is used to populate a data element.

Data Governance and Stewardship

• Data governance is the process of defining the rules that data has to follow, and data stewardship makes sure that the data follows those rules.
• Documenting how your company manages its data, and then developing metrics to monitor the effectiveness of those management practices.
• Data governance uses metadata management to enforce management discipline on the collection and control of data.
• Governance and Stewardship are especially important functions in an MDM process.
• It’s not enough to have the numbers; you have to be able to show that the numbers are based on accurate and verifiable data. This means both a governance function to demonstrate that the right controls are in place, and a stewardship function to ensure that the controls are enforced.

Data Governance Rules

• Who can read, create, update, and delete data.
• What validity checks are required for data.
• Which application is the preferred source for data items.
• How long data is retained.
• What privacy policies are enforced.
• How confidential data is protected.
• And what disaster-recovery provisions are required.
• The data-governance function should include leaders of both the IT and business groups of a company. This partnership used to be a hard thing to achieve; but with CEOs going to jail, and data on stolen laptops making front-page news, management is paying a lot more attention.

Stewardship

• The role of a data steward is to ensure that the master data and metadata is clean consistent, and accurate.
• In cases in which data quality could not be determined with automated rules and workflows and manual intervention is needed. The people doing this manual intervention are the data stewards.
• The right person to be a steward for a particular collection of master data is the person who understands the data the best. In many cases, only someone who understands the business can make the tough calls about which value is correct for a particular data element.
• Data Stewardship is all about fixing data quality issues.
• This process is best implemented using a routing an approval system of some kind.

Metadata in Data Governance

• Metadata provides the linkage between the business need or desire (policy) and the information or data value.
• The effective management of metadata is one of the essential activities of a data steward within a governance practice, enabling data management policy and access to information.
• Metadata management refers to the activities associated with ensuring that meta data is created/captured at the point creation and that the broadest possible portfolio of meta-information is collected, stored in a repository for use by multiple applications, and controlled to remove inconsistencies and redundancies.
•  In short, data governance uses meta data management to impose management discipline on the collection and control of data.
• Metadata management is a critical component of any robust data governance practice, and metadata is one of the foundational contributors to creating and maintaining full business value of an organization’s data.

Data Quality Metadata

Some Data Quality Metadata

• Policies and Process
• Mission statement
• Guiding Principles
• Work flows
• Best Practices
• Role descriptions

Summary

Metadata management is key to the following

• Master data management
• Data Governance
• Compliance
• Data Quality

Metadata for Regulatory Compliance and Personal Information Protection.

Metadata Describing Private Data

Ensuring that the private data stored by a business is adequately protected is increasingly the responsibility of a chief privacy officer (CPO). Typically, CPOs are responsible for ensuring privacy laws are complied with, identifying the data in a company that needs to be protected, developing policies to protect that data, and responding to any incidents that occur. Another common issue that must be dealt with is the changing definition of what information is regarded as personal, and therefore, needs to be safeguarded.

This cannot be accomplished without metadata management!

Basel II

The Basel II Accord, also referred to as the “New Capital Accord,” created a set of compliance requirements that provide risk management guidelines to banks around the world. Developed by an International Banking Committee based in Basel, Switzerland, local regulatory agencies will enforce these guidelines.

The accord has a direct effect on the amount of reserves that banks need to set aside for handling unexpected losses. If a bank’s risk management practices comply with the Basel II guidelines, the bank can keep less in their capital reserves and thus have more capital available for revenue generation.

But banks only realize that benefit if they can prove they have a trustworthy risk management system that does valid risk-related calculations that can be authenticated and traced in an audit. They must also maintain a transparent system that allows customers, investors, and other interested parties to evaluate the efficacy of the bank’s risk management practices.

While Basel II makes no references to metadata, banks can only meet key requirements in the accord if they effectively capture and preserve metadata and Basel requires a history of metadata.

Sarbanes Oxley

• Sarbanes-Oxley (S-OX) includes among its many requirements, “Internal controls and procedures for financial reporting,” which require CEO’s and CFO’s to:
• Certify the contents of their corporate reports.
• Provide an assessment of internal controls for all systems used to generate and deliver financial reports.
• Not only document internal procedures, but demonstrate adherence.

SEC rules mandate “policies and procedures in addition to the control environment… including adequate safeguards over access to programs and data files.” Auditors will tell you that rules that apply to data also apply to metadata (information about information), and business rules making a controlled, auditable process for metadata management, a critical business imperative for public companies.

Metadata for Compliance Points to Ponder

• When it comes to data quality and information integrity, information about the data (metadata) is as important as the data itself. Metadata plays a key role in how people generate, store, track, retrieve, report, and deliver information. Including information within financial statements.
• Metadata provides the underlying definition and audit ability that defines how information flows.
• As such, metadata touches the complete family of sources (documents, spreadsheets, databases and applications) that are used when compiling financial reports.
• Oversight represents a best-practice and critical-to-quality element, which should be reviewed during an IT system audit.
• One urgent, aspect of Sarbanes-Oxley compliance involves enterprise-level control of metadata.
• Inconsistency produces comparison problems or errors in reported data. If you think of structural and business metadata as the underlying definition of the information structures themselves, then what’s needed is a metadata repository that supports reconciling metadata across heterogeneous environments; and helps resolve interoperability problems: which can consume 30-50% of integration time and effort.

Metadata Management Software Framework

Now that we covered the basics to establishing a program within your organization, let’s have a look at software tools to help drive our initiatives forward.

Remember Metadata Management is a Dependency of:

• Data Quality.
• Data Governance.
• Master Data.
• Big Data.
• Data management best practices.

The human brain metadata sucker device is not terribly practical or realistic. Reverse engineering 3rd party source tools will always require brute force, and there is no magic bullet when it comes to getting our arms around all our IT systems. In fact ETL is part of the problem not the solution.

The key here is what are you trying to do, what do you need to do it. Technologically, what we need to succeed already exists. The trick is identifying the starting point and the problem to be solved so that we can begin to do the work of managing our metadata.

Here we will look at a typical metadata framework technology required for any metadata management program.

• We need to put it somewhere ( Metadata Repository(s)).
• We need to make it searchable (Search engine).
• Provides a way to manage business semantics, such as business glossaries, synonyms, business terms, rules, custom types,  and taxonomies.
• Make is useable to integration and consume, share, and collaborate on its changes (Metadata Portal and Web Services components).
• We must be able to manage it ( Stewardship and Governance management tools).

Software Frameworks

In computer programming, a software framework is an abstraction in which common code providing generic functionality can be selectively overridden or specialized by user code; thus providing specific functionality.

Frameworks are a special case of Software Libraries in that they are reusable abstractions of code wrapped in a well-defined API, yet they contain some key distinguishing features that separate them from normal libraries.

Software frameworks have these distinguishing features that separate them from libraries or normal user applications:

• Inversion Control- In a framework, unlike in libraries or normal user applications. The overall program’s flow control is not dictated by the caller, but by the framework.
• Default behavior – A framework has a default behavior. This default behavior must actually be some useful behavior.
• Extensibility – A framework can be extended by the user usually by selective overriding, or specialized by user code providing specific functionality.

Metadata Management Framework:

• A metadata management framework provides the building blocks to help you integrate and manage your metadata.
• A metadata management framework is a set of software tools that captures, manages. and publishes rich metadata. Especially business metadata.
• A framework provides all that is needed to plug into the technical metadata contained in your environment and make it searchable.
• A framework also typically provides a repository and portal which is similar to Google and Wiki for searching, data entry, impact analysis, and collaboration.
• Existing applications can query the metadata to enrich functionality via the framework.
• A good framework can eliminate the need to develop a complete custom software application.
• A framework is the glue that connects the dots within your environment.

Metadata Management Framework Essential Components

• A repository to store the metadata. Not all of it, just the important stuff.
• Powerful tools to discover as much metadata as possible automatically.
• Flexibility of adapters to accommodate changes over time.
• Tools to map metadata across sources.
• Scalable options to accommodate phased efforts.
• Robust search engine technology bundled.
• Ability to perform impact analysis.
• Simple automated maintenance and scalability.
• Should allow you to add custom properties and types.
• Provide a historical perspective.
• Provide rich business metadata, terms, rules, and taxonomy capabilities.
• Business glossaries and semantics capabilities.
• Should not retain your metadata in an overly complex difficult to query super meta-model, but make it available to query easily.

Buy vs. Build

Build your own Metadata Framework

•   Cost in terms of labor, resources, and time to market testing and debugging.
•   Do we have the resources to develop a solution?
•   Do we have skill set?
•   Most homegrown efforts fail!
•   Developing a framework in-house is a very difficult task.

Buy a COTS (commercial off the shelf solution)

•  Cost – Can we get budget?
•  What is our budget?
•  Not many vendors to choose from, evaluation process should be easy!
•  Do we need customization. (Do we have a finely defined scope, which will allow us to use a product off the shelf).
•  Can save a lot of time and money.
•  A Software Framework provides the modules + adaptability and customizability.

Summary

Your success is dependent on identifying a business problem, finding a solution to that problem, and selling that solution to management. Followed up by a well defined and managed project leveraging a metadata integration framework.

Ready to Develop a Metadata Program?

In Previous Articles in this Series we have Covered:

• Why are we not leveraging our metadata today?
• Why projects fail.
• Common perceptions that prevent us from utilizing our metadata.
• How and why metadata management is important.
• That business metadata holds the key (focus should be here).
• That technical metadata is best leveraged for searching and impact analysis.
• Business terms, business rules, and taxonomies are building blocks.
• Metadata entry and stewardship is part of the required tasks.

First lets have a look at some basic components of a typical metadata management program:

• Defining the problem to be solved.
• Determining initial scope and requirements.
• Getting a budget and executive sponsorship.
• Knowing where the business metadata and technical metadata fit in.
• Knowing your options buy vs build or hybrid.
• Project execution.

Common Problems Solved with Effective Metadata Management:

• Can’t reliably search information from other programs.
• Terminology often means different things in different contexts.
• Information from other programs may be contradictory,  e.g. research information may conflict with regulations.
• No glossary business terms.
• No data dictionaries exist.
• No way of knowing how things inter-relate.
• How is a particular item calculated?
• Redundancy of effort.
• Duplicate data, or poor data quality.
• Lost knowledge due to employee turnover.
• Cannot identify information for compliance, audits, and governance.
• What is the impact of changes to a data point? Downstream or upstream.
• Is there a single version of the truth? Can I trust this data?
• Employees spend too much time searching for information to do their jobs.
• Metadata management is required for master data, data governance, data quality, and big data efforts.

Common Mistakes

• Often we want a magic bullet to reverse engineer sophisticated products that their own vendors cannot provide useful metadata for.
• Metadata ETL hell. (Gory details – leave that to ETL Tools).
• Tendency to get bogged down in details, theories, and architectures  (leave theories to the academics).
• Analysis paralysis.
• Scope too huge. Try to sell metadata mega project ( RFP a good example).
• Expectation or belief that standards actual matter.
• Opportunistic vendors. I.E:  BI tool vendor claim they are metadata management tool.
• ETL is part of the problem , not the solution.
• No budget or buy-in.

Determining Initial Scope and Requirements

In order for any project to be successful, the better job we do up front of defining requirements and scope; the more likely we’ll be successful. I cannot emphasize this enough.

Getting Executive Sponsorship and Budget

This is key, for obvious reasons; without budget and buy in there will be no project. In order to provide a good metadata return on investment to the decision makers, you will need to leverage the requirements gathered to identify the key pain points and drive the message home as it pertains to the problem you will be solving.

This is the most difficult part of your effort, as it will involve getting some hard numbers and building a clear presentation to management in language they understand. Typically, technical folks have difficulty with this and usually need to ask for help in this area.

Knowing Where the Business Metadata and Technical Metadata Fit.

I have included this, because I’ve seen way too many efforts get unwieldy because there was simply too much focus put on the technical and the reverse engineering activities. We need to know where the real benefit is. Propose a phased solution that targets the business need, which means we’ll be focusing on business metadata. You’ll get a win.

Buy vs Build

Here, I have to say that if you expect to build a metadata solution from scratch, you are in for a lot of pain. Take it from me, unless you have a lot of developers and resources to spend the time it will take to go through full blown development projects, bugs, and iterations; you are doomed to fail.

Unbiased as I am not. I recommend that you are much better off finding a vendor to work with as a consulting partner, that can help you with requirements, developing a road map, and ROI for your program. Take advantage of commercially available tools that are a fit to your companies needs. It’s not only about tools or technologies, standards or hype, it’s about solving business problems.

Ultimately, the key to your success will hinge on your willingness to get the help you need to develop a program that will solve business problems, and get a win with your metadata management program. The tools will support your effort.

Project Execution

All successful projects have one thing in common, good project management. In order to meet the budget and objectives, the project needs to be expertly managed by seasoned professionals, especially, when dealing with the dynamics related to metadata integration and management programs. However, there are many success stories, case studies and proven ROI samples available to back this up.

Here are Some Lessons Learned

• Must manage expectations and scope. If you try to boil the Atlantic ocean you will get burned. Scope creep is a killer of any IT project.
• Because the term metadata is abstract and not widely understood in a corporate environment, asking someone “How they use metadata” in their job is  a question many people struggle with.
• Trying to define an elaborate set of roles and in doing so introducing a whole new strain of resource requirements and terminologies. Don’t over-complicate.
• Metadata is only useful if it is current and accurate.
• Keep it simple (to start).
• Identify the (a) problem you are trying to solve , then work from there.
• Phased approach.
• The whole idea is to know what you have and where it is!

Summary

Your success is dependent on identifying a business problem, finding a solution to that problem, and selling that solution to management. Followed up by a well defined and managed project leveraging a metadata integration framework.

The Big Picture Perspectives

We’ve broken down the types of metadata into three main perspectives Technical, Business, and Enterprise as depicted in this diagram.

Aspects for Metadata Management

Technical Metadata Aspect

Here we address capturing technical metadata within a repository such as schemas, configurations, and code using automated processes and adapters. Since this aspect is mostly automated, it provides searching and impact analysis capabilities directly against the technical metadata.

Business Metadata Aspect

Notice that the business metadata aspect is broken out into two parts; tools and methods respectively. When we start to look at capturing business metadata, we’ll be faced with much more challenges to capture specific business metadata. In fact we may find that most business metadata lives in Excel Spreadsheets or people’s heads. This is where the tools and methods needs to be considered. Essentially, what is needed is a set of tools that allow for the creation, management, mapping, stitching, classification, and loading of business metadata from many possible sources. In cases where data entry will be required; collaboration, metadata governance, and stewardship will be critical to a system for capturing business metadata.

Enterprise Metadata Aspect

When we think of the enterprise metadata aspect, we address functionality and requirements as they relate to governance, stewardship, collaboration, security, change management, retention, and publication or sharing of metadata.

Metadata Repository

• A metadata repository has a facility to store your important metadata. Need a place to put it?
• Much less volume than data!
• Does not have to be singular,and can federate for searching.
• Contains mostly business metadata. Direct to technical.
• Technical metadata is descriptive only.
• Does not make sense to reverse engineer everything into a monolithic repository.
• Searchable and can create reports.
• A robust portal. (Web front end).
• Taxonomy driven navigation.

Some Data Entry Required

• No way to do business metadata without some data entry.
• Value returned versus cost of data entry is huge. (Its worth it).
• Most of the important metadata, business metadata lives in peoples heads, or in spreadsheets.
• Someone needs to connect the dots. Subject matter experts, data stewards, librarians.
• Can make part of a process with the right solution.
• Contributors use wiki style data entry screens.
• Workflow integration for approval and metadata stewardship.
• Automated scanning and refresh of technical metadata.
• Model driven architecture.  Example, logical model may contain business definitions; therefore can be captured as part of scanning process and automated.
• Requirements driven Business Analyst web form. Use controlled vocabulary (Business Terms) to build the report spec.

In the last post, I discussed why we need metadata management, and some of the common reasons we see projects false starts, and failures.

In this post I will focus on what I feel is the number one inhibitor of successful establishment of a metadata management program. A propensity to focus on the technical and lack of good requirements. Lets talk about business metadata. Since this is where the un-tapped value really lies.

Business Metadata VS Technical Metadata

Business Metadata  describes technical computer data and applications in a way that is understandable to the business users.  Business Metadata hides technological constraints by mapping business language or glossary to the technical systems, data points, applications, and processes.

Technical metadata includes things such as field definitions, data dictionaries, tasks, transformations, and code. All of which is very useful to the technical staff to search and perform impact analysis to support IT.

The differentiators for Business Metadata is that it provides:

• Context
• Understandability
• Search ability
• Usability

Business Metadata answers the following questions:

• What are the valid values?
• Where did this data come from?
• How do I get more information?
• Who is responsible for the data?
• How is this data calculated?

Business Metadata provides a logical drill down:

• Is this table going to work?
• Show me a list of reports, applications, terms, or rules.
• Can I trust this data?
• How fresh is the data?
• When was the data last updated?
• Is there a single version of the truth?

The Un-tapped Value of Business Metadata (Much of which is in peoples heads)

It is the Business Metadata that we need to focus on. Not the technical so much. That should be automated for the most part.

Example: An ETL developer will want to see the gory details and will use their preferred ETL tool for that. What is needed is a pointer to “what happened” between [a] and [b]. Business users want to know what happened in business language.

Good examples of Business Metadata:

• Applications.
• Products.
• KPI’s.
• Business terms.
• Business rules.
• Taxonomies.
• Guiding principles.
• Data governance metrics such as policies, rules, and valid values.

Business Metadata holds the key to the real meaning of data, applications, and processes. In order to be successful we need to know the requirements as they pertain to the pain or need, and then plan or define a road map to begin to capture and manage the metadata.

Most projects I’ve seen are very successful, if they are started with a requirement phase and the focus is on addressing the business needs for information, where we can fill in the blanks.

Overview

This is the first article in a series blog I will be updating regularly. I will discuss the many aspects of metadata management including pitfalls and the secret sauce to successfully starting, leveraging and making use of the valuable metadata buried within your IT infrastructure.

IT folks all know and believe that metadata management is a key dependency for projects such as master data management, data quality, data governance, data warehousing and business intelligence efforts.
However, many metadata projects die on the vine or fail and here are some of the main reasons we’ve seen as to why that happens.

• With data to day issues, starting a metadata initiative just seems too strategic, difficult.
• Very difficult to get budget and buy in. Unfortunately IT folks are not sales people.
• Management has no idea what this “metadata” is all about and how it helps them.
• Perceptions that it would require too much effort and that our systems are so special it just won’t work.
• Fear of “who is going to maintain all of this?”
• The industry itself has a unique talent for over-complicating the issue.
• IT tends to get bogged down in technical metadata over analysis.
• RFP’s (this is a big one) we see the same 10 year old RFP metadata template floating around which do not address the discovery required to get at the paint points and problems to be solved. They favor the biggest vendors. We often walk away from these RFP’s.
• Failure of the client to engage a vendor for help early in the process. Particularly in the requirements gathering phase. Experts are there to help.
• Metadata is an afterthought, just like documentation is to development efforts.

Why is leveraging your metadata so important?

• Aid developers and technical people in sourcing information.
• Determine the impact of making system changes.
• Capture metadata from legacy system.
• Identify redundant processes.
• Data governance and regulatory compliance.
• Help new employees learn technical systems.
• Allow business users to navigate, find and analyze information , specifically how it relates to other pieces of information.
• Classify information for improved search and navigation to the right information.
• Provide documentation to other business units.
• ETL projects. (I’ve seen integration Project blow budget by 20 million.)
• Too many to list here.

However, Metadata is Being Leveraged Successfully

• Intel claims 6 dollars for every dollar spent ROI on metadata management initiative. Intel Metadata Case Study – the combined repository will helps cut the time — which can peak at 30% of a developer’s day — it takes for developers to find information assets.
• SOA would not exist without metadata.
• Master data is driven by metadata.
• The .NET framework is metadata driven.
• XML is metadata rich!
• Many working implementations and examples out there.

Please stay tuned for the next blog in this series. I will be covering metadata in detail, including strategies to get started in your environment.