Databricks to Teams notifications via workflows

Note: As the office 365 connectors are planned to be retired by January 31, 2025, the recommended upgrade is to create incoming webhooks with workflows.

Receiving notifications of failures in pipelines is a crucial part of Data Engineering work. Here, I will lead through how to make teams notifications from Databricks (in this case using Databricks Standard) to Microsoft Teams using workflows.

First, we want to create a new teams channel, where the notifications will be coming through:

In my case I will call it "Databricks - Error notifications PROD"

Press Create, then the new channel in Microsoft Teams is created.

Once the channel is created, we will start to setup a workflow connection:

We will select More options next to channel and select workflows:

Pressing the workflows, we will get a tab on what workflow we want to connect:

In this case, I want to post to a channel when a webhook request is received:

We pick that one, it automatically picks my email:

Press Next: Then the Microsoft teams channel where the webhook connector can be added will pop up:

We press Add workflow. Easy as that - the new workflow is added succesfully.

We copy the URL to the notepad.

Workflow has been set in Microsoft teams now.

For connecting to Databricks we want to go to Databricks workspace. Press on your name to get this tab:

We go to settings:

In settings we press notifictions:

We want to manage notification destinations:

Then I press add a destination:

Then this tab will appear:

Under the destination type I want to choose Microsoft Teams:

Once the Microsoft Teams destination is chosen, the only two things you will need to choose is name of this connection and workflow URL we copied previously to our notepad.

I will call it Workflow connector

Press save. Then we go to the ultimate test: Open a job to in the Databricks workspace and try adding up the notifications based on our desires:

For this I have created a mock job which will fail just for different tests:

So we go to the job notifications, press Edit Notifications:

We press add notification:

Select Destination:

We pick the workflow connector we just created a few minutes ago:

Save, then we run the workflow to see if we get a notification in our designated teams channel.

And we can see the notification via workflow appeared in my designated Teams channel:

Happy connecting!