shiny-aiconversation

name: shiny-aiconversation description: Generate code for Shiny.AiConversation - a centralized AI service library for .NET MAUI apps with chat client abstraction, wake word detection, speech-to-text/text-to-speech, acknowledgement modes (None/AudioBlip/LessWordy/Full), persistent message store, optional AI chat history lookup tool, and configurable sound effects auto_invoke: true triggers: - shiny ai - shiny maui ai - ai service - aiservice - iai service - iaiservice - chat client provider - ichatclientprovider - message store - imessagestore - wake word - wakeword - listen and talk - speech to text - text to speech - ai acknowledgement - aiacknowledgement - audio blip - ai chat message - aichatmessage - chat history - ai tool lookup - chat lookup - talk to ai - ai response - ai state - addshinyaiconversation - add shiny ai conversation - ai conversation service - aiconversationservice - iaiconversationservice - ai service options - aiserviceoptions - quiet words - quietwords - voice interruption - speech interruption - expects response - conversation continuation - speech to text options - text to speech options - github copilot - copilot chat - addgithubcopilotchatclient - addstaticopenaichatclient - openai static - openaistaticChatprovider - request access - requestaccess - voice selection - voice tools - change voice - play voice sample - get available voices - voiceselection - addvoiceselectiontools - add voice selection tools - voice sampling - tts voice - switch voice references: - ai-service.md - registration.md - message-store.md - chat-client-provider.md - chat-lookup-tool.md - voice-selection-tools.md

Shiny.AiConversation Skill

You are an expert in the Shiny.AiConversation library, a centralized AI service for .NET MAUI applications that integrates chat, speech recognition, wake word detection, text-to-speech, and persistent message storage.

Library Overview

GitHub: https://github.com/shinyorg/speech (Shiny.AiConversation lives under the ai/ directory) NuGet: Shiny.AiConversation Namespace: Shiny.AiConversation Infrastructure Namespace: Shiny.AiConversation.Infrastructure (internal implementations)

The library provides:

• IAiConversationService: Central orchestrator for AI interactions — manages access checking, state (Idle/Listening/Thinking/Responding), wake word detection, speech-to-text capture, chat client communication, text-to-speech response, acknowledgement modes, sound effects, persistent chat history, conversation continuation (auto-listens when AI asks a question), and voice interruption (quiet words to stop TTS, or speak over the AI to redirect the conversation)
• IChatClientProvider: Abstraction for obtaining an IChatClient (from Microsoft.Extensions.AI) — a default implementation (InjectedChatClientProvider) resolves IChatClient from DI; custom implementations handle authentication, token management, and client construction
• IMessageStore: Abstraction for persisting and querying chat message history — implementations provide storage (SQLite, file system, cloud, etc.)
• ChatLookupAITool: AI tool that allows the AI to search past conversations via IMessageStore — automatically added by ContextProvider when an IMessageStore is registered
• VoiceSelectionContextProvider: Optional IContextProvider that adds three AI tools — get_available_voices, play_voice_sample, and change_voice — enabling the AI to list voices, play audio samples, and change its own TTS voice mid-conversation. Enabled via opts.AddVoiceSelectionTools().
• AiChatMessage: Record representing a persisted chat message with Id, Message, Timestamp, and Direction (User/AI)
• IContextProvider: Visitor-pattern abstraction for populating an AiContext per request. Each provider's Apply(AiContext) method receives a mutable context and adds its contributions. The ContextProvider handles time-based prompts, acknowledgement-aware voice prompts, and DI-registered AITool instances. Implement custom providers to add domain-specific system prompts, tools, or override speech settings.
• AiContext: Mutable context object passed to IContextProvider.Apply() — contains Acknowledgement, SystemPrompts, Tools, QuietWords, SpeechToTextOptions, and TextToSpeechOptions that providers populate or modify
• AiConversationOptions: Fluent configuration for DI registration — sets chat client provider, message store, sound provider, and additional context providers via AddContextProvider<T>()

Built-in Provider Packages:

• Shiny.AiConversation.OpenAi (OpenAiStaticChatProvider): Static OpenAI-compatible provider. Accepts API key, endpoint URI, and model name. Works with OpenAI, Azure OpenAI, Ollama, or any OpenAI-compatible API. Register with opts.AddStaticOpenAIChatClient(apiToken, endpointUri, modelName).
• Shiny.AiConversation.Maui.GithubCopilot (GitHubCopilotChatClientProvider): MAUI-specific provider using GitHub device code OAuth flow and the Copilot API. Self-contained auth — shows a popup with the device code, copies to clipboard, opens browser, polls until authorized. Tokens stored in SecureStorage. Register with opts.AddGithubCopilotChatClient(). Additional API: StartAuthentication(), CancelAuthentication(), SignOut(), IsAuthenticated, AccessTokenChanged event.

Dependencies

• Microsoft.Extensions.AI — IChatClient, ChatMessage, ChatRole, AITool, ChatOptions
• Shiny.Speech — ISpeechToTextService, ITextToSpeechService, IAudioPlayer for voice interactions and sound effects

When to Use This Skill

Invoke this skill when the user wants to:

• Set up an AI chat service in a .NET MAUI app
• Register and configure IAiConversationService with dependency injection
• Implement IChatClientProvider for a specific AI backend (OpenAI, GitHub Copilot, Azure, etc.)
• Implement IMessageStore for persistent chat history
• Add wake word detection to an app
• Configure acknowledgement modes (None, AudioBlip, LessWordy, Full)
• Set up sound effects for AI state transitions
• Configure voice interruption with quiet words
• Set up speech-to-text and text-to-speech options (culture, voice, speech rate, etc.)
• Add the optional ChatLookupAITool for AI-driven history search
• Enable voice selection tools so the AI can list voices, play samples, and switch its own voice
• Build a chat UI that integrates with IAiConversationService
• Handle AI state changes (Idle, Listening, Thinking, Responding)
• Use TalkTo or ListenAndTalk for AI interactions
• Check speech/microphone access before starting voice features

Code Generation Instructions

1. Registration (MauiProgram.cs)

Always register with AddShinyAiConversation():

using Shiny.AiConversation;

// Option A: Register an IChatClient in DI (simplest approach)
builder.Services.AddChatClient(new OpenAIClient("your-api-key").GetChatClient("gpt-4o").AsIChatClient());

// Option B: Use a built-in provider
builder.Services.AddShinyAiConversation(opts =>
{
    // OpenAI-compatible (OpenAI, Azure OpenAI, Ollama, etc.)
    opts.AddStaticOpenAIChatClient("your-api-key", "https://api.openai.com/v1", "gpt-4o");

    // OR GitHub Copilot (MAUI only — self-contained auth with SecureStorage)
    opts.AddGithubCopilotChatClient();

    opts.SetMessageStore<MyMessageStore>(); // optional — ChatLookupAITool added automatically
});

• Built-in providers: AddStaticOpenAIChatClient() for OpenAI-compatible APIs, AddGithubCopilotChatClient() for GitHub Copilot on MAUI
• SetChatClientProvider<T>() is for custom providers — if not set and no built-in provider is used, the default InjectedChatClientProvider resolves IChatClient from DI
• SetMessageStore<T>() is optional — enables persistent history; the ContextProvider automatically adds ChatLookupAITool when a store is present
• AddVoiceSelectionTools() is optional — registers VoiceSelectionContextProvider, giving the AI tools to list voices, play samples, and change its own voice
• AddContextProvider<T>() registers additional IContextProvider implementations
• SetSoundProvider<T>() registers a custom ISoundProvider implementation
• System prompts, tools, quiet words, and speech options are provided via IContextProvider implementations registered in DI (a ContextProvider is auto-registered)

2. Chat Client Setup

Simple approach — register IChatClient in DI (the default provider resolves it automatically):

builder.Services.AddChatClient(new OpenAIClient("your-api-key").GetChatClient("gpt-4o").AsIChatClient());

Advanced approach — implement IChatClientProvider for on-demand auth or token refresh:

public class MyChatClientProvider : IChatClientProvider
{
    public async Task<IChatClient> GetChatClient(CancellationToken cancelToken = default)
    {
        // Obtain/refresh tokens, authenticate if needed, build client
        return new OpenAIChatClient(...);
    }
}

• Handle token expiry and re-authentication inside GetChatClient
• Can inject INavigator to navigate to a login page if authentication is needed on-demand

3. Implementing IContextProvider (Custom)

The ContextProvider is registered automatically and provides time-based prompts, acknowledgement-aware voice prompts, and any AITool instances from DI. To add custom system prompts, tools, or modify speech settings, implement IContextProvider using the visitor pattern and register it:

public class MyContextProvider : IContextProvider
{
    public Task Apply(AiContext context)
    {
        context.SystemPrompts.Add("You are a helpful assistant for our company.");
        // context.Tools.Add(...) to add AI tools
        // context.Acknowledgement is available to adjust behavior per mode
        // context.QuietWords — modify quiet words for voice interruption
        // context.SpeechToTextOptions — set/override speech recognition options
        // context.TextToSpeechOptions — set/override text-to-speech options
        return Task.CompletedTask;
    }
}

// Register in DI — multiple providers are supported, executed in sequence
builder.Services.AddShinyAiConversation(opts =>
{
    opts.AddContextProvider<MyContextProvider>();
});

4. Implementing IMessageStore

public class MyMessageStore : IMessageStore
{
    public Task Store(string? userTriggeringMessage, ChatResponse response, CancellationToken cancellationToken) { ... }
    public Task Clear(DateTimeOffset? beforeDate = null) { ... }
    public Task<IReadOnlyList<AiChatMessage>> Query(
        string? messageContains = null,
        DateTimeOffset? fromDate = null,
        DateTimeOffset? toDate = null,
        int? limit = null,
        CancellationToken cancellationToken = default) { ... }
}

5. Using IAiConversationService

// Check access before using voice features
var access = await aiService.RequestAccess();
if (access != AccessState.Available)
{
    // Speech is not available — handle accordingly
    return;
}

// Send a text message
await aiService.TalkTo("What's the weather?", cancellationToken);

// Listen via microphone and send to AI
await aiService.ListenAndTalk(cancellationToken);

// Wake word detection (loops until stopped)
await aiService.StartWakeWord("Hey Assistant");
aiService.StopWakeWord();

// Query chat history
var history = await aiService.GetChatHistory(limit: 25);
var filtered = await aiService.GetChatHistory(messageContains: "weather", startDate: yesterday);

// Clear history
await aiService.ClearChatHistory();
await aiService.ClearChatHistory(beforeDate: oneWeekAgo);

// React to state changes
aiService.StatusChanged += (state) => { /* update UI with new AiState */ };
aiService.AiResponded += (response) =>
{
    // response.Response — the complete ChatResponse
    // response.Response.Text — the text content of the response
    // response.WasReadAloud — whether TTS was used
    // response.ExpectsResponse — true if AI asked a question (will auto-listen for reply)
};

6. Acknowledgement Modes

7. AI States

Best Practices

1. Use IContextProvider for configuration — System prompts, tools, quiet words, and speech options are all configured via IContextProvider.Apply(AiContext) — not set directly on the service
2. Handle auth on-demand — IChatClientProvider should handle authentication lazily, not force login at startup
3. Use TwoWay binding for Acknowledgement in settings UIs
4. Subscribe/unsubscribe to StatusChanged and AiResponded in page lifecycle (OnAppearing/OnDisappearing)
5. Use CancellationToken for all TalkTo/ListenAndTalk calls
6. MessageStore is optional — The service works without it but GetChatHistory/ClearChatHistory will throw
7. ChatLookupAITool is opt-in — Pass addAiLookupTool: true to SetMessageStore to allow the AI to search past conversations
8. No reflection — All registrations must be explicit; do not use ActivatorUtilities.CreateInstance or reflection-based patterns