Skip to main content
Submit or View Requests
Deutsch Español (España) Français (France) Italiano (Italia)

Understanding automated Call Scoring

Automated call scoring enables you to evaluate calls using AI Assist Pro based on customizable criteria—without needing to trigger evaluations manually. By setting up automation rules, you can ensure consistent, scalable call quality monitoring across your organization.

IMPORTANT:
Automated call scoring requires **AI Assist Pro** licenses for both the admin configuring the automation and the agent whose calls will be evaluated. Only **scorecards** with **AI-evaluable questions** are supported.

How automation rules work

Automation rules define which calls will be automatically evaluated. You can create tailored rules using the following filters:

  • Users: Individual agents or entire teams
  • Phone numbers: Specific lines or all company numbers
  • Call direction: Inbound, outbound, or both
  • Call duration: Minimum and/or maximum length

  • Call tags: call contains all of the selected tags

The automation engine runs on two events:

  • Call ends

    • CALL_CONVERSATION_UPDATED

  • Call tags updated

    • TAG_CREATED

    • TAG_DELETED

For each event, the engine scans your active rules from top to bottom and applies the first matching rule. The scorecard attached to that rule is then evaluated by AI. If an evaluation already exists for that call, it is replaced. Only one evaluation is stored per call at any time.

Setting up automation rules

Step 1: Access automation rules

  1. Go to Aircall Dashboard > AI Assist > Scorecards.
  2. Click the Automation rules tab.
  3. Select Create automation to start a new rule.
Aircall Conversation Center showing Scorecards and Automation rules overview with options to create new scorecards or automations.

Step 2: Configure rule criteria

  • Scorecard selection: Choose the scorecard to apply. Only scorecards with AI-evaluable questions support automation.
  • User targeting: Select individual users or full teams. Use All users to apply the rule across the organization.
  • Number assignment: Apply the rule to all numbers or limit it to specific lines. You can filter by line type or campaign.
  • Call direction
    • Inbound: Calls initiated by customers
    • Outbound: Calls initiated by agents
  • Duration requirements: Set minimum and/or maximum call lengths for the rule to apply.

  • Call tags: Apply the scorecard when the call contains any of the selected tags.

  • Agent to evaluate: When a call involves multiple agents, such as a transferred call, use the Agent to evaluate dropdown to choose which agent the scorecard applies to:

    • All agents (default): Evaluates every agent who participated in the call. This is the existing behavior and applies automatically if you do not change this setting.
    • First agent: Evaluates only the agent who first handled the call.
    • Last agent: Evaluates only the agent who last handled the call.
TIP:
The most common use case to use agent to evaluate rule is transferred calls. For example, if a receptionist picks up and transfers to a specialist, setting this to Last agent evaluates only the specialist, without the handoff portion affecting their score.
NOTE:
Existing automation rules that were not configured with the agent to evaluate rule, are not affected by it. They will continue using All agents unless you update them manually. If a call involved more than two agents, only the first or last can be selected. There is no option to evaluate an agent in the middle.
Aircall Scorecards page with Automation rules panel open, allowing users to configure scorecard, users, numbers, call type, and duration

Step 3: Set rule priority

When multiple rules could apply to the same call:

  • Only the highest priority rule will execute.
  • Drag and reorder rules to adjust priority.
  • Lower-priority matches will be ignored.

Tag-based triggers for automated call scoring

This feature enhances Automated Call Scoring by introducing tag-based triggers. Previously, automation rules could apply scorecards based on criteria such as users, numbers, call direction, or duration. With this update, scorecards can also be applied automatically when a call includes specific tags.

Because tags are often added after a call ends, the automation engine will now re-evaluate rules each time call tags are updated, not just at the time the call ends.

Supervisors can now ensure automated evaluations are aligned with existing workflows. For example, a call tagged “Escalation” can automatically trigger a QA scorecard, while a “Demo” tag can trigger a Sales scorecard. This improvement increases the accuracy and relevance of automated scoring coverage.

New trigger type: Call tags

Admins can now configure automation rules to apply a scorecard when any tag in a defined list is present on a call.

  • Example: If a call has the tag “Escalation” or “High Priority”, apply the Support QA scorecard.

Trigger timing

  • Rules are re-evaluated whenever tags are updated after a call ends.
  • If tag conditions are met post-call, the scorecard is applied retroactively.
  • If another scorecard has already been applied, it will be overridden.

Evaluation handling

  • When multiple rules could apply (for example, a tag-based rule and an existing rule), the system respects the priority order as it does today.
  • Only one rule will fire per call to prevent duplicate evaluations.

Managing automation rules

In the Automation rules tab, you can view and manage all your rules.

  • View rule names, descriptions, active status, criteria summaries, and priority order.
  • Edit or disable rules
    • Click the ellipsis (⋯) to edit a rule’s configuration.
    • Use the Active/Inactive toggle to pause or resume automation without deleting the rule.
NOTE:
Active rules will evaluate matching calls. Inactive rules remain saved but will not trigger evaluations.

Rule examples

Use case Users/Teams Numbers Direction Duration
Sales team cold calls 10 sales agents 2 outbound lines Outbound 10+ minutes
Support quality checks 2 support teams Support line Inbound Any length

Viewing automated results

After a call is completed, evaluations are available in:

  • The call record, displaying the scorecard and evaluation results
  • Scorecard analytics, showing team performance over time
NOTE:
Automation rules only apply to calls that occur after the rule is created. Past calls are not retroactively evaluated.

AI Scorecard Evaluator tool

Use the AI Scorecard Evaluator to improve the quality of your call scoring questions and ensure they are AI-compatible.

What it can do

  • Evaluate a single question or an entire scorecard
  • Rate each question’s AI scoring difficulty (easy, medium, or hard)
  • Suggest improved wording for clarity and accuracy
  • Recommend a complete scoring guide (e.g. Yes/No or Very Poor → Excellent)

Try the AI Scorecard Evaluator (ChatGPT tool)

NOTE:
This tool runs on OpenAI’s ChatGPT. Do not paste sensitive customer data, personally identifiable information (PII), or confidential business information into the tool. Learn more in [OpenAI’s guide](https://openai.com/enterprise-privacy).

You will need a ChatGPT account to access the tool, but will not need a paid license to use it.

How to get the best results

  • Format questions in Yes/No or ranged style (e.g. 1–5, Poor to Excellent)
  • Include your scoring guide to help the tool provide better suggestions
  • If no scoring guide is provided, the tool will suggest one

FAQs

Can I run multiple automations on the same call?
No. Only the highest priority rule that matches the call will be executed.

What happens if a rule has no matching calls?
The rule stays active, but won’t generate evaluations until matching calls occur.

Can I pause an automation without deleting rules?
Yes. Use the Active/Inactive toggle to temporarily disable a rule.

Do automation rules work with manual questions?
No. Automation only applies to AI-evaluable questions in scorecards.

Jordan W.