What Are Competency Questions? They Help AI Know Your Business
Good ontologies – or good products – don’t start with answers. They start by asking the right questions as part of the design process.
Key Takeaways
For AI to be valuable to a business, it requires organizing information in a way that makes it easy for machines to understand and use. How you do that in a consistent and trustworthy way is through design.
Part of the deep work to realize the benefits of AI involves developing ontologies, which codify concepts, relationships, and constraints within the context of your business. One of the ways to ensure the ontology is effective is to use competency questions (CQs).
Let’s learn what that means and why implementing CQs is important to designing trustworthy AI experiences.
Here’s what we’ll cover:
What are competency questions?
Ask questions, design better AI systems
How do CQs affect ontology engineering?
Structure questions to match the business need
Closing the gap
What are competency questions?
CQs are questions that set the boundaries for what knowledge an ontology should include. A simple way to think about it is:
- Start with real human questions.
- Translate them into machine-verifiable definitions of success.
- Use these to measure gaps, improve, and iterate.
Like fixing the electrical wiring in your home, building a proper ontology isn’t glamorous and has many facets. Every choice about how our systems understand context, maintain meaning across interactions, or know when to ask for clarification is design work applied to intelligence.
Ontology design involves multiple layers of validation and verification. One of those techniques is to use CQs, which ensure an ontology strikes the right balance for a broad spectrum of customer needs. It would be difficult to provide a generalized model of all the areas common across domains and account for the nuances of each domain.
CQs help capture the right level of granularity.
Ask questions, design better AI systems
Recently, my colleague Irina Malkova, Vice President of Data Science at Salesforce, compared CQs to AI evals. Her realization: “I thought AI evals were this brilliant new innovation. Turns out, we’ve just rediscovered a 30-year-old recipe for automating human knowledge.”
What if, Irina wondered, instead of building first and hoping for adoption, teams designed backwards from the questions that drive decisions?
It’s not, for example, only about building dashboards. It’s about designing a shared understanding between humans and AI, organizing metadata, and designing findability and sense-making. The work is to build the scaffolding for better decisions.
“Ontology folks have been doing structured reasoning long before LLMs made it cool.”
How do CQs affect ontology engineering?
Let’s get technical. The utility of CQs spans the entire ontology engineering lifecycle, from initial scoping to testing. First, let’s learn about the key stages in the ontology development lifecycle:
| Stage | Description |
|---|---|
| Scoping | The lifecycle begins with a use case that defines the problem and objectives the ontology should address. |
| Elicitation | In this stage, the ontology’s functional and non-functional requirements are defined. |
| Ontology design | Involves codifying the concepts, relationships, and constraints within the specified domain. |
| Implementation | Focused on data mapping, knowledge graph construction, reasoning enrichment, and testing; usually executed in parallel with later phases. |
| Deployment | Covers the operational aspects: pipeline automation, scaling, monitoring, maintenance, and security. |
| Consumption | Defines usage guidelines, integration patterns, query interfaces, AI/ML models, and BI tools; developed alongside previous phases. |
| Evaluation | The final assessment of success against original metrics. Includes documenting lessons learned and pitfalls to inform the next iteration. |
How CQs function:
| Phase | Primary CQ Function |
|---|---|
| Scoping and elicitation | Help define the operational boundary of the domain, confirming which concepts are essential and which fall outside the scope of the model. |
| Ontology design | Guide the ontology engineer in identifying the specific classes, object properties, and data properties necessary to represent the domain knowledge effectively. |
| Implementation | Inform the properties that must hold true in the construction of a graph. |
| Deployment, consumption, and evaluation | CQs evolve into formal test cases, formalized as executable SPARQL queries, ensuring the constructed ontology correctly represents the intended knowledge and fulfills established requirements consistently. |
Structure questions to match the business need
CQs help ontology modelers and engineers to connect the concepts that matter to users: what they’re trying to do, what they’re searching for, and what conditions should trigger a next step.
These requirements then become the ontology’s formal evaluation framework. When it works, the ontology can answer the questions it was designed to answer.
- Example business need: Performance managers need to analyze their organization’s explicitly defined goals to track progress and support resource allocation.
- A potential competency question: What are the strategic business goals set by the company ACME for the 2026 fiscal year, including their description, target KPIs, priority factor, and associated performance indicators?
In this example, to support effective querying, an ontology model should be able to filter business goals (strategic, operational) set by a business entity (company, department), during a time horizon (fiscal year, quarter, date) and provide information about the goal itself (description, status), target and observed metrics (key performance indicators) that are relevant for a given user (person, role).
By framing what the model needs to answer, CQs make sure the right building blocks are in place and become a built-in test for whether different systems can work together.
Types of CQs
| CQ Type | Purpose | Example |
|---|---|---|
| Scoping – broad questions that help define the territory; they don’t need a formal answer, but they shape what the model needs to cover. | Confirms the necessary content coverage for the ontology. | Progress tracking and resource allocation requirements influence modeling by introducing time-related and categorical classes that represent historical and current states. While the details of these classes might not be fully defined at this stage, the modeler knows that the design must include a temporal dimension. |
| Validations – test instruments for the conceptual model. | Verifies that the ontology accurately reflects the domain and that the established requirements have been fulfilled. | A time-related class can be expressed as an interval or an instant. |
| Foundations – introduce a layer of semantic clarity. | Sets up the bigger-picture context for an entity, so the model can reason more deeply about it based on what that context implies. | An ontology can represent time in a way that’s meaningful to the business. If intervals make the more sense for consumption, the model should accommodate notions such as fiscal year, a quarter, a release cycle. |
| Relationships – designed to establish the logical characteristics of a specific property (relationship) within the ontology. | Helps to test the application of domain and range. | Unlike activities, a place might not logically require an association with a time interval. |
| Metaproperties – interrogates the entities’ ontological characteristics. | Aids in classifying the entities correctly for reusable ontological design. | A specific person can adopt and lose a role during their tenure in an organization. Is Jane (an instance of a person) at all times of her existence an instance of a “performance manager” (an instance of a role)? Could the model support Jane if she took a new role? |
Closing the gap
Building AI that understands your business starts with asking the right questions.
CQs force alignment between what a business needs, what a system is built to do, and what an AI agent can reliably act on. Ontologies grounded in well-formed CQs close the gap between a model that technically works and one that actually works.
That can be the difference between a product that gets used versus one that sits on the shelf.
