6.2 Logging, Monitoring,and Notification
These three functions are essential for cloud integration because they provide visibility into message flows, performance, and errors, allowing for timely responses to issues
🡆Logging: This is a record of all events and messages within an integration flow. It captures detailed message processing logs, which can be stored, searched, and analyzed to trace anomalies and reconstruct incidents (p. 18). Some services even allow for advanced analytics by querying logs with SQL .
🡆Monitoring: This is a dashboard and toolset for observing the real-time status of integrations. It provides a high-level overview of system health and performance metrics and allows users to drill down into individual message details to troubleshoot errors .
🡆Notification: This is the process of alerting relevant teams when predefined conditions are met. You can define alerting policies based on metrics or log data, such as when an error count exceeds a threshold. Alerts can be sent to designated channels like email, Slack, or other integrated services.
Example Flow:
🡆A Matillion orchestration job runs to load data into Snowflake.
🡆If the job fails, Matillion’s internal audit log records the failure details.
🡆Simultaneously, the configured CloudWatch Logs Agent streams the failure logs to AWS CloudWatch for further analysis.
🡆The Metrics Insight Dashboard or an integrated external monitoring tool (e.g., Datadog) flags the job failure based on the collected metrics.
🡆A pre-configured webhook in Matillion triggers a notification to a designated Slack channel, including details about the failed job and the error message, alerting the data engineering team.
A email notification is sent to a wider audience, informing them of the job’s failure.