AI-103: Developing AI Apps and Agents on AzureDemo

You have a Microsoft Foundry project that contains an agent. The agent generates summaries from retrieved policy documents.

Users report that some responses omit required regulatory clauses, even though the clauses are present in the retrieved content.

You need to improve the completeness of the responses.

You have a Microsoft Foundry project that contains an agent. The agent generates summaries from retrieved policy documents.

Users report that some responses omit required regulatory clauses, even though the clauses are present in the retrieved content.

You need to improve the completeness of the responses.

You have a Microsoft Foundry project that contains an agent. The agent generates summaries from retrieved policy documents.

Users report that some responses omit required regulatory clauses, even though the clauses are present in the retrieved content.

You need to improve the completeness of the responses.

You have a Microsoft Foundry project that contains an agent. The agent generates summaries from retrieved policy documents.

Users report that some responses omit required regulatory clauses, even though the clauses are present in the retrieved content.

You need to improve the completeness of the responses.

Overview

Company Information

Contoso, Ltd is a multinational retail company that builds, deploys, and manages generative AI and agent-based solutions using Microsoft Foundry.

Existing Environment

Identity Environment

Contoso relies on Microsoft Entra ID for identity management, authentication, and authorization, enabling agents to access organizational resources and services. Contoso recently created a new AI engineering team called Agent1Dev Team to optimize and maintain existing AI solutions. This team works with solution architects, DevOps engineers, and security engineers to design, implement, monitor, and secure AI applications. Contoso also has a team named Agent1Test Team that is responsible for validating AI solutions before the solutions are deployed.

Generative Environment

Contoso has a Microsoft Foundry deployment that contains two projects named Project1 and Project2.

Project1

Project1 contains a customer support agent named Agent1 that helps customers with product inquiries and troubleshooting requests. Agent1 has the following configurations:

• Agent1 uses a base model deployment.
• A safety evaluation pipeline is NOT enabled.
• Tool invocation approval workflows are NOT enabled.
• Conversation memory constraints are NOT configured.

Agent1 interacts with customers through digital support channels and answers general questions about Contoso products. Project1 is deployed to an Azure region located in the European Union (EU). Agent1Dev Team will use Project1 to optimize and maintain Agent1.

Project2

Project2 contains a deployed video generation model. The marketing department at Contoso has access to Project2 and plans to use the model to build a video creation solution. Development of the solution is not yet complete.

Data Environment

Contoso stores product-related information in Azure resources that support AI applications. The Azure environment includes an Azure Blob Storage account named storage1 that stores product detail sheets for all Contoso products. The product sheets include specifications, feature descriptions, and product support information that Agent1 can use to answer customer questions. The product sheets are stored in PDF format.

Problem Statements

Contoso identifies the following issues:

• Agent1 has only general knowledge of the Contoso products.
• A recent chat interaction with Agent1 was analyzed for sentiment. The results of the analysis have NOT been processed yet.
• Agent1 does NOT use the detailed product information in the product sheets stored in storage1 when responding to customer questions.
• The finance department at Contoso reports that vendor invoices must be reviewed manually to ensure that the invoices match the terms defined in the vendor contracts. The invoices contain tables, logos, and varied layouts that make the documents difficult to process consistently.

Requirements

Planned Changes

Contoso plans to implement the following changes:

• Implement a solution for Project1 that analyzes the vendor invoices by evaluating both the visual layout and the textual content of the invoices, so that the invoice details can be verified against the vendor contract terms.
• Update the base model deployment used by Agent1 and standardize the model version to ensure continuity and consistent responses.
• Enable Agent1 to retrieve and use the detailed product information from the product sheets stored in storage1.
• Implement an indexing solution for the product sheets that Agent1 can use to answer customer questions.
• Complete the development of the video creation solution.

Technical Requirements

Contoso identifies the following technical requirements:

• The model deployment used by Agent1 must support scalable, high-throughput generative AI workloads and dynamically scale to handle variable customer support traffic, without requiring reserved throughput capacity.
• The product sheets must be processed using an indexing pipeline that enables semantic and vector search, so that Agent1 can retrieve the relevant product information.
• Responses generated using the product sheet information must be relevant, complete, and accurate.
• Agent1 must be able to use the product sheets to answer natural language questions about product details.
• The model version used by Agent1 must remain consistent to ensure stable responses.
• The data processed by the model must remain within the EU.

Security and Compliance Requirements

Contoso identifies the following security and compliance requirements:

• API keys must NOT be used to access Foundry-deployed models.
• Access to the Azure resources must follow the principle of least privilege.
• The developers at Contoso must authenticate to Microsoft Foundry resources using Microsoft Entra authentication.
• Access to Project1 must be assigned to the members of Agent1Dev Team using a security group named SC_Agent1_Dev.
• Access to Project1 must be assigned to the members of Agent1Test Team using a security group named SC_Agent1_Test.
• Agent1 must never reveal customer information, even if a document that contains customer data is added erroneously to the product sheet repository in storage1.
• The product sheets might contain images that include embedded text. Agent1 must be protected from malicious instructions potentially hidden within the images.

Business Requirements

Contoso identifies the following business requirements:

• Users that interact with Agent1 must have a personalized experience in future interactions, including the ability for Agent1 to retain conversation context and recall relevant information from previous interactions.
• Agent1 must answer questions only about the products sold by Contoso.

Overview

Company Information

Contoso, Ltd is a multinational retail company that builds, deploys, and manages generative AI and agent-based solutions using Microsoft Foundry.

Existing Environment

Identity Environment

Contoso relies on Microsoft Entra ID for identity management, authentication, and authorization, enabling agents to access organizational resources and services. Contoso recently created a new AI engineering team called Agent1Dev Team to optimize and maintain existing AI solutions. This team works with solution architects, DevOps engineers, and security engineers to design, implement, monitor, and secure AI applications. Contoso also has a team named Agent1Test Team that is responsible for validating AI solutions before the solutions are deployed.

Generative Environment

Contoso has a Microsoft Foundry deployment that contains two projects named Project1 and Project2.

Project1

Project1 contains a customer support agent named Agent1 that helps customers with product inquiries and troubleshooting requests. Agent1 has the following configurations:

• Agent1 uses a base model deployment.
• A safety evaluation pipeline is NOT enabled.
• Tool invocation approval workflows are NOT enabled.
• Conversation memory constraints are NOT configured.

Agent1 interacts with customers through digital support channels and answers general questions about Contoso products. Project1 is deployed to an Azure region located in the European Union (EU). Agent1Dev Team will use Project1 to optimize and maintain Agent1.

Project2

Project2 contains a deployed video generation model. The marketing department at Contoso has access to Project2 and plans to use the model to build a video creation solution. Development of the solution is not yet complete.

Data Environment

Contoso stores product-related information in Azure resources that support AI applications. The Azure environment includes an Azure Blob Storage account named storage1 that stores product detail sheets for all Contoso products. The product sheets include specifications, feature descriptions, and product support information that Agent1 can use to answer customer questions. The product sheets are stored in PDF format.

Problem Statements

Contoso identifies the following issues:

• Agent1 has only general knowledge of the Contoso products.
• A recent chat interaction with Agent1 was analyzed for sentiment. The results of the analysis have NOT been processed yet.
• Agent1 does NOT use the detailed product information in the product sheets stored in storage1 when responding to customer questions.
• The finance department at Contoso reports that vendor invoices must be reviewed manually to ensure that the invoices match the terms defined in the vendor contracts. The invoices contain tables, logos, and varied layouts that make the documents difficult to process consistently.

Requirements

Planned Changes

Contoso plans to implement the following changes:

• Implement a solution for Project1 that analyzes the vendor invoices by evaluating both the visual layout and the textual content of the invoices, so that the invoice details can be verified against the vendor contract terms.
• Update the base model deployment used by Agent1 and standardize the model version to ensure continuity and consistent responses.
• Enable Agent1 to retrieve and use the detailed product information from the product sheets stored in storage1.
• Implement an indexing solution for the product sheets that Agent1 can use to answer customer questions.
• Complete the development of the video creation solution.

Technical Requirements

Contoso identifies the following technical requirements:

• The model deployment used by Agent1 must support scalable, high-throughput generative AI workloads and dynamically scale to handle variable customer support traffic, without requiring reserved throughput capacity.
• The product sheets must be processed using an indexing pipeline that enables semantic and vector search, so that Agent1 can retrieve the relevant product information.
• Responses generated using the product sheet information must be relevant, complete, and accurate.
• Agent1 must be able to use the product sheets to answer natural language questions about product details.
• The model version used by Agent1 must remain consistent to ensure stable responses.
• The data processed by the model must remain within the EU.

Security and Compliance Requirements

Contoso identifies the following security and compliance requirements:

• API keys must NOT be used to access Foundry-deployed models.
• Access to the Azure resources must follow the principle of least privilege.
• The developers at Contoso must authenticate to Microsoft Foundry resources using Microsoft Entra authentication.
• Access to Project1 must be assigned to the members of Agent1Dev Team using a security group named SC_Agent1_Dev.
• Access to Project1 must be assigned to the members of Agent1Test Team using a security group named SC_Agent1_Test.
• Agent1 must never reveal customer information, even if a document that contains customer data is added erroneously to the product sheet repository in storage1.
• The product sheets might contain images that include embedded text. Agent1 must be protected from malicious instructions potentially hidden within the images.

Business Requirements

Contoso identifies the following business requirements:

• Users that interact with Agent1 must have a personalized experience in future interactions, including the ability for Agent1 to retain conversation context and recall relevant information from previous interactions.
• Agent1 must answer questions only about the products sold by Contoso.

Overview

Company Information

Contoso, Ltd is a multinational retail company that builds, deploys, and manages generative AI and agent-based solutions using Microsoft Foundry.

Existing Environment

Identity Environment

Contoso relies on Microsoft Entra ID for identity management, authentication, and authorization, enabling agents to access organizational resources and services. Contoso recently created a new AI engineering team called Agent1Dev Team to optimize and maintain existing AI solutions. This team works with solution architects, DevOps engineers, and security engineers to design, implement, monitor, and secure AI applications. Contoso also has a team named Agent1Test Team that is responsible for validating AI solutions before the solutions are deployed.

Generative Environment

Contoso has a Microsoft Foundry deployment that contains two projects named Project1 and Project2.

Project1

Project1 contains a customer support agent named Agent1 that helps customers with product inquiries and troubleshooting requests. Agent1 has the following configurations:

• Agent1 uses a base model deployment.
• A safety evaluation pipeline is NOT enabled.
• Tool invocation approval workflows are NOT enabled.
• Conversation memory constraints are NOT configured.

Agent1 interacts with customers through digital support channels and answers general questions about Contoso products. Project1 is deployed to an Azure region located in the European Union (EU). Agent1Dev Team will use Project1 to optimize and maintain Agent1.

Project2

Project2 contains a deployed video generation model. The marketing department at Contoso has access to Project2 and plans to use the model to build a video creation solution. Development of the solution is not yet complete.

Data Environment

Contoso stores product-related information in Azure resources that support AI applications. The Azure environment includes an Azure Blob Storage account named storage1 that stores product detail sheets for all Contoso products. The product sheets include specifications, feature descriptions, and product support information that Agent1 can use to answer customer questions. The product sheets are stored in PDF format.

Problem Statements

Contoso identifies the following issues:

• Agent1 has only general knowledge of the Contoso products.
• A recent chat interaction with Agent1 was analyzed for sentiment. The results of the analysis have NOT been processed yet.
• Agent1 does NOT use the detailed product information in the product sheets stored in storage1 when responding to customer questions.
• The finance department at Contoso reports that vendor invoices must be reviewed manually to ensure that the invoices match the terms defined in the vendor contracts. The invoices contain tables, logos, and varied layouts that make the documents difficult to process consistently.

Requirements

Planned Changes

Contoso plans to implement the following changes:

• Implement a solution for Project1 that analyzes the vendor invoices by evaluating both the visual layout and the textual content of the invoices, so that the invoice details can be verified against the vendor contract terms.
• Update the base model deployment used by Agent1 and standardize the model version to ensure continuity and consistent responses.
• Enable Agent1 to retrieve and use the detailed product information from the product sheets stored in storage1.
• Implement an indexing solution for the product sheets that Agent1 can use to answer customer questions.
• Complete the development of the video creation solution.

Technical Requirements

Contoso identifies the following technical requirements:

• The model deployment used by Agent1 must support scalable, high-throughput generative AI workloads and dynamically scale to handle variable customer support traffic, without requiring reserved throughput capacity.
• The product sheets must be processed using an indexing pipeline that enables semantic and vector search, so that Agent1 can retrieve the relevant product information.
• Responses generated using the product sheet information must be relevant, complete, and accurate.
• Agent1 must be able to use the product sheets to answer natural language questions about product details.
• The model version used by Agent1 must remain consistent to ensure stable responses.
• The data processed by the model must remain within the EU.

Security and Compliance Requirements

Contoso identifies the following security and compliance requirements:

• API keys must NOT be used to access Foundry-deployed models.
• Access to the Azure resources must follow the principle of least privilege.
• The developers at Contoso must authenticate to Microsoft Foundry resources using Microsoft Entra authentication.
• Access to Project1 must be assigned to the members of Agent1Dev Team using a security group named SC_Agent1_Dev.
• Access to Project1 must be assigned to the members of Agent1Test Team using a security group named SC_Agent1_Test.
• Agent1 must never reveal customer information, even if a document that contains customer data is added erroneously to the product sheet repository in storage1.
• The product sheets might contain images that include embedded text. Agent1 must be protected from malicious instructions potentially hidden within the images.

Business Requirements

Contoso identifies the following business requirements:

• Users that interact with Agent1 must have a personalized experience in future interactions, including the ability for Agent1 to retain conversation context and recall relevant information from previous interactions.
• Agent1 must answer questions only about the products sold by Contoso.