For the configure button for the operation, there are many options. You can specify a Slack webhook along with the channel, display name, emoji/icon url, and even the layout of the message. By default, whenever you create a payload via the "Create Payloads" page, it is tagged as alertable - any time a new callback is created based on that payload, this slack webhook will be invoked. If you want to prevent that for a specific payload, go to the payloads page, select the "Actions" dropdown for the payload in question, and select to stop alerting. If you have the Slack webhook set on the operation overall, other payloads will continue to generate alerts, but not the ones you manually disable. You can always enable this feature again in the same way.