How to get visibility into your BizTalk environment

In today's fast-paced business environment, the ability to monitor and understand the flow of data within an organisation is crucial for both IT and business teams. Microsoft BizTalk Server offers powerful built-in capabilities to track messages and business data, provide telemetry to support understanding of system health for IT support teams and collecting of key business metrics to support near real-time insights to business stakeholders. This article details how BizTalk supports these capabilities so organisations can quickly understand what is happening in their BizTalk environment.

BizTalk's Built-in Message Tracking Capability

One of the core features of BizTalk Server is its robust message tracking capability. This functionality is integral for IT support teams to monitor the health of messages and orchestrations as data flows through the system. Using the BizTalk Health Monitor (BHM) functions, IT teams can gain visibility into the state of messages, identify bottlenecks, and troubleshoot issues promptly.

How Message Tracking Works

Message Flow and Tracking Points:

BizTalk processes messages through orchestrations, pipelines, ports, and adapters. Tracking points are specific locations within these components where message data can be logged. These tracking points include receive ports, send ports, orchestration start/end points, and pipeline stages.

Tracking Configuration:

Administrators can configure tracking settings using the BizTalk Administration Console. This involves specifying which message properties and bodies should be tracked. Proper configuration ensures that only necessary data is captured, minimizing performance impact.

Tracked Data:

BizTalk tracks various types of data, including message properties (context properties), message content (body), and event data such as errors or warnings. This information is stored in the Tracking Database (DTA), which is indexed and can be queried for reporting and troubleshooting.

Tracking Database (DTA):

The DTA database is the central repository for all tracked data. Tools like the BizTalk Administration Console and the Health and Activity Tracking (HAT) tool allow users to access and analyze this data. The database schema includes tables for message events, service instances, and other tracked information.

Health and Activity Tracking (HAT):

HAT is a tool provided with BizTalk for viewing and analysing tracked data. It enables users to search and filter tracked messages, view message flow, and diagnose issues. HAT also allows for correlating messages across different stages of processing, helping to identify bottlenecks or errors.

BizTalk Administration Console:

The BizTalk Administration Console provides an interface to configure tracking settings and view tracked data. Administrators can set tracking options for receive locations, send ports, orchestrations, and pipelines, making it easier to manage and monitor message flow.

Analytics and Reporting:

Tracked data is crucial for generating reports and analytics. BizTalk integrates with tools like SQL Server Reporting Services (SSRS) and third-party monitoring solutions to create custom reports. These reports provide insights into message flows, performance metrics, and system health.

Performance Considerations:

While tracking is essential for monitoring and troubleshooting, it can impact performance. Administrators must balance the level of detail tracked with the system's performance requirements. Careful planning and configuration are necessary to ensure optimal performance without sacrificing essential tracking data.

BizTalk Activity Monitoring (BAM) for Business Insights

In addition to message tracking, BizTalk Server includes BizTalk Activity Monitoring (BAM), a powerful tool for tracking business data and collecting insights for near real-time reporting. BAM allows organisations to monitor key business processes and gain visibility into critical business metrics.

Defining BAM Activities and Views:

• Activities: BAM activities represent business processes or transactions you want to monitor. Each activity is defined by a set of milestones and business data.
• Views: BAM views aggregate data from one or more activities to provide a comprehensive picture of the business process. Views can include aggregations, charts, and KPIs.

BAM Configuration and Tools:

• BAM Definition Workbook: Business analysts and developers use Excel-based BAM Definition Workbooks to define activities and views. These workbooks specify the milestones, business data, and dimensions for analysis.
• BAM Deployment: Once defined, activities and views are deployed to the BizTalk BAM infrastructure using the BAM management utility (BM.exe). This deployment process creates the necessary databases and OLAP cubes for storing and analysing BAM data.

• Tracking Profiles: BAM data collection is configured using tracking profiles, which map business data from BizTalk orchestrations, pipelines, and ports to BAM activities. Tracking profiles are created and managed using the Tracking Profile Editor (TPE).
• Real-Time Data Collection: As messages flow through BizTalk, the mapped data is collected and stored in the BAM Primary Import database. This real-time data collection enables immediate visibility into business processes.

Analysing BAM Data:

• BAM Portal: The BAM Portal is a web-based interface that allows business users to view and analyse BAM data. Users can monitor real-time business activities, track KPIs, and generate reports. The portal provides customisable dashboards and interactive data exploration.
• Excel Workbooks: BAM data can also be analysed using Excel, providing flexible reporting and data visualisation capabilities. Users can create pivot tables, charts, and other visualisations to gain insights into business performance.

Data Archiving and Aggregation:

• Data Archiving: To manage the volume of data, BAM includes archiving capabilities that move old data from the BAM Primary Import database to the BAM Archive database. This ensures that the BAM system remains performant while retaining historical data for analysis.
• OLAP Cubes: BAM can create OLAP cubes in SQL Server Analysis Services (SSAS) for advanced data analysis and reporting. These cubes enable complex queries and multidimensional analysis, supporting deeper insights into business processes.

Performance Monitoring

Finally, to ensure optimal performance of the BizTalk environment, it is essential to monitor various performance metrics. BizTalk Server provides special performance counters that can be used in conjunction with SQL Server performance counters. These counters help IT teams understand how the environment is performing with respect to core compute, memory, and disk usage.

BizTalk Performance Counters

The performance counters provided by BizTalk Server cover the following areas:

• Message Box
• Orchestrations
• Host Throttling
• Messaging
• Data Tracking
• Adapters
• BAM  

This enables a very detailed look into how the BizTalk core processes are performing and will give you some ideas on how to configure your environment for more suitable performance, not to mention if there are some key performance issues you are investigating.

SQL and Windows Server Performance Counters

The BizTalk Performance Counters will not tell you the whole story, so it is important to look at the performance of the underlying SQL Server and the Windows Server operating systems that both BizTalk and SQL Server are installed on.

SQL Server provides an easy method to access performance counters, by exposing them as a dynamic management view which can be queried using TSQL.

Some performance issues can only be resolved by understanding how the operating system is managing Compute, Memory, Storage or networking. Reviewing the performance counters in conjunction with SQL Server and BizTalk will give you a wholistic view of what is happening your environment.