InfoLibrarian Metadata Adapters™ | Batch, Streaming & Event Driven Metadata Integration
Metadata is everywhere, in continuous motion, and change... capture metadata from diverse sources, run rules engine based classification and canonicalization of the payload streaming metadata. Automatically capture metdata to the open InfoLibrarian Repository™ or stream to other third party destinations or publishing endpoints.
The adapters can be later scheduled in batch, or run in a continuous streaming change capture in batch or realtime. This makes all your Metadata available via the portal, for sql like querying or meta-analysis with third party tools like Jupyter Notebooks, Panda Dataframes and Python.
InfoLibrarian™ Universal Adapters Bundle!
We've bundled a set of the most common and useful point and click adapters with every version of the product.
The standard adapter bundle enables you to configure crawling of metadata from most sources in most environments.
Once metadata is onboarded, the portal will provide search, impact analysis results. Below is a list of the adapters
included with the product:
- Import adapter for Oracle and SQL Server and OLEDB-ODBC-JDBC sources.
- Excel adapter for tabular metadata from Excel documents).
- ANSI Information_schema (Relational).
- Oracle Metadata Schema Adapter (Relational).
- SQLServer Metadata Schema Adapter (Relational).
- DB2 Metadata Schema Adapter (Relational).
- XMI Universal Adapter (XMI-CWM payloads).
- Web API Adapter (Document Web API's).
- InfoLibrarian™ Transformer Canonicalizer.
- Command Line (CLI) tool for scheduling re-scans using one of the above adapter configurations.
Note: The XMI (XML Metdata Interchange format) Adapter can integrate metadata from most popular BI tools, ETL processes, data models, as well as other tools that exmit their metadata via XML or JSON. The XML adapter can
be configured to point to several hot folders containing XML files for fast multithreaded integration of this metadata.
Metadata can be tranformed by InfoLibrarian™ on the fly to virtually any defined meta-model, industry or user defined standard structure.
Note: Other specialized and customer adapters are available upon request.
The InfoLibrarian™ repository supports polyglot persistence. Store your metadata in open data lake formats, NOSQL JSON or as persisted graph models!
Scanning Performance and Load
The InfoLibrarian Metadata Adapters™ were designed to perform in the most demanding environments and are capable of
scanning millions of objects. The adapters have a small footprint and show negligible performance impact against the
sources they scan. The Auto-Scanner is fully multi-threaded taking advantage of SMP hardware. Adjustments can be made
to the batch size for the queries that run against your source systems, giving you full control in minimizing any
impact on your production systems. Our adapters have been deployed in some of the largest production I.T. environments in the USA
against thousands of servers in large server farms and data centers and in big data cloud ecosystems.
What is the Scan Engine?
The InfoLibrarian Scan Engine™ is the architectural component that drives all functions related to processing scans of metadata. The InfoLibrarian Scan Engine™ is used by adapters and scanners to process sources systems
metadata and to stream or load into the InfoLibrarian Repostory™. Additionally, the InfoLibrarian Scan Engine™ provides the interfaces for the adapter templates used to develop new custom adapters.
InfoLibrarian Scan Engine™ - Enterprise Class Performance & Scalability
Traditional metadata repositories have one major flaw, they don't scale. This has never been the case with InfoLibrarian™, we designed the Scan Engine™ and repository with scalability in mind but we are never satisfied.
We've made huge improvements to the InfoLibrarian Scanning Engine™.
Previous version of the Infolibrarian Scan Engine™ were efficient and scaled well. Now they are simply fast and scale to the Petabytes of metadata. We've added improvements
for scanning of huge file systems and data environments, and we've reduced the overall time to complete those scans, allowing you to scour entire infrastructure in smaller windows.
Additionally, the impacts of the scans, already being minimal; have been further optimized to scale in the most demanding clustered, SMP environments and running in the cloud.
Automated Scanning
The InfoLibrarian Auto-Scanner™ refreshes and maintains historical changes as they occur over time. The InfoLibrarian Auto Scanner™ ships with the InfoLibrarian Studio™ product, but the big difference with the server version is the engine behind it.
Running the CLI Command line interface server console utility server version provides much more performance boost than with the standard desktop components. The InfoLibrarian CLI™ server version is fully multithreaded, taking
advantage of server processors and memory. The command line utility also offers minimal logging modes and other optimization settings to further tune performance.
InfoLibrarian Auto-Scanner™
- Re-scanning single catalogs.
- Re-scan all catalogs with rescan flag set.
- Creates snapshots and captures deltas.
- Age-off settings and steps.
- Settings to handle deletions in source.
- GUI interface.
- Command line mode.
- Full logging.
- Calculates new, changed, and deleted objects and properties.
- Reduce project analysis time.
- Diagnose code deployment and configuration problems.
- Manage physical assets.
- Provide reports for governance and compliance auditing.
- Can be scheduled using third party schedulers.
InfoLibrarian Studio Auto Scanner
InfoLibrarian Adapter SDK™ API's
The InfoLibrarian Adapter SDK™ is a MSDN style library of documentation and component API’s to allow organizations to create custom adapters;
intranet and/or windows based integrated solutions using the InfoLibrarian Framework™. InfoLibrarian API™ supports languages such as C#, C++, Java (via INTEROPT), and web services interfacing.
Enterprise versions of the product come with templates that allow you to quickly and easily develop your own custom metadata adapters. The toolkit provides the basic templates for OLEDB and file systems files which allow you to quickly modify the simple mapping template to create unlimited custom adapters.
Source Code Provided for Adapters Templates
When you purchase an enterprise adapter the source code is available for customization, changes, and enhancements. An example would be an Oracle adapter maybe changed to map in different data by changing the queries in the adapter and redeploying that adapter as a new custom adapter. This saves a lot of time since much of the implementation details are already part of that adapter template. We have several universal adapters that are perfect for this type of customization.
Template Code Pre-Built and Available for Most Interfacing Methods
- ODBC/OLE DB (Connect to hundreds of DBMS and related products).
- XML (Load metadata from XML files in with any schema).
- XMI (Pull in XMI metadata from XMI compliant tools).
- API (Up for the challenge, hook up to a 3rd party tool API).
- Web Services (Query web service based applications and services for metadata).
- Messaging middleware in real time.
- File Processing (Process any files anywhere, even Active Directory or LDAP authentication is built in).
We'll Build Any Adapter You Need
Want to outsource the work to build an adapter? InfoLibrarian™ has the expertise, experience, and skill for the job. After all, we have built over 300 different adapters so far.
Years of developing and maturing the framework makes solving the problems possible .
Build Your Own Adapters - Low Code
Load metadata from virtually any source using the Adapter API. With only a few lines of code you can build your own metadata adapter.
And we'll provide all the training you need to get the job done. Simple batch loader example below:
//Code snipit from Task Scheduler Adapter
foreach (SchedulerWS SchedulerObject in SchedulerList)
{
Mapper.MapBaseProperties(SchedulerObject.FolderPath.ToString(), MetadataMapper.BaseProperties.ObjectName);
Mapper.MapBaseProperties(SchedulerObject.Definition.ToString(), MetadataMapper.BaseProperties.ObjectDescription);
Mapper.MapBaseProperties(this.ScanArgs.Catalog.ToString(), MetadataMapper.BaseProperties.ObjectParent);
Mapper.MapBaseProperties("1", MetadataMapper.BaseProperties.ObjectParentID);
Mapper.MapBaseProperties(JobFolderType.ToString(), MetadataMapper.BaseProperties.ObjectType);
long CurrentFolderObjectInstance = Mapper.UpdateObjectInRepository();
long retval = Mapper.UpdateObjectInRepository();//Create the Task Object Instance
}
Features at a Glance
- Create object and types.
- Map your source metadata to types.
- Create cross reference mappings such as needed for ETL source target mappings.
- Bulk and chunk processing, CDC pattern or Steaming and microbatch for huge metadata source data sets.
- InfoLibrarian Rules Engine™ included.
- Only code the mapping logic, the rest is done for you.
Need to Engineer a Custom Metadata Integration?
One of the most common use cases we see relates to custom metadata integration requirements.
While turnkey repository tools and catalogs exist, in most cases some degree of custom metadata integration
is required to address a business need. We've done everything from Streaming media "Syndication TV streaming metadata" to custom Business Taxonomies, all require some form of integration to harness
vital metadata. Data Warehouses are not the only source of metadata in an organization! We specialize in those edge cases, and difficult to solve
metadata integration challenges beyond a simple turnkey implementation of a data catalog or commercial DG point solution.
Please contact us and we will be happy to assist you in scoping out your custom metadata integration requirements.
Lets solve some real challenges together.