Chapters
Chapters

Chapters

Hide chapters

Apple Foundation Models

First Edition · iOS 26.5 · Swift 6.3 · Xcode 26.4

Before You Begin

Section 0: 5 chapters
Show chapters Hide chapters

Section I: Apple Foundation Models

Section 1: 9 chapters
Show chapters Hide chapters

8. Using Foundation Models for Voice Notes
Written by Bill Morefield

Heads up... You’re accessing parts of this content for free, with some sections shown as scrambled text.

Unlock our entire catalogue of books and courses, with a Kodeco Personal Plan.

Unlock now

Heads up... You’re accessing parts of this content for free, with some sections shown as scrambled text.

Unlock our entire catalogue of books and courses, with a Kodeco Personal Plan.

Unlock now

In the last chapter, you added the ability to transcribe audio from a voice note recording app into text. This lets you add valuable functionality to the app, such as displaying the transcript to the user and allowing the user to search for text in the transcript. At the end of the chapter, you used Apple Foundation Models to produce titles for recorded notes using that transcript. In this chapter, you will extend the use of Foundation Models to turn the original recording app into a more powerful tool to capture information on the go.

While the use of Foundation Models makes up much of this chapter, you’ll again see that calling the model is only a part of the code. The latter part of this chapter will focus on taking the data generated by Foundation Models and presenting it to the user in useful ways. The goal is to use artificial intelligence to provide value in your app.

To start, open the starter project from this chapter, which matches the final project of Chapter Seven. Repeat the steps to add the NSSpeechRecognitionUsageDescription key to your Target:

  1. Go to the Project for the app in Xcode and select the VoiceNotes target.
  2. Go to the Info tab, and you will see the existing list of properties.
  3. Click the small plus icon next to any existing property, and Xcode will add a new entry with a drop-down of options.
  4. Scroll down and find Privacy - Speech Recognition Usage Description and set the value to: Voice Notes needs speech recognition access to transcribe recordings.

Producing Data Structures for Analysis

In Chapter Five, you learned about perhaps the most powerful feature of Foundation Models, guided generation. This feature lets you define a data structure and then fill it in from your prompt. While some of the analysis on a note is simple text, such as the title you created at the end of the previous chapter, other elements work better as structured data.

Create a new Swift file under Models named VoiceNoteAnalysis.swift. Replace the contents of the file with:

import Foundation
import FoundationModels

@Generable(description: "A concise analysis of a transcribed voice note.")
struct NoteAnalysis {
  @Guide(description: "A concise title of a few words that summarizes the note contents.")
  let title: String

  @Guide(description: "A two to three sentence summary of the voice note.")
  let summary: String

  @Guide(description: "Up to five short lowercase topic tags.", .count(1...5))
  let tags: [String]

  @Guide(description: "People referenced in the note.")
  let people: [String]

  @Guide(description: "Specific action items or tasks mentioned in the note.")
  let actionItems: [GeneratedNoteActionItem]
}

This struct uses the @Generable macro, which allows the model to respond to prompts by creating an instance of the type. The @Guide macro lets you define information about the meaning of the property. Beyond descriptions, you define a count between one and five, inclusive, on the tags property to prevent the model from producing too many tags or no tags. This struct defines the analysis you can do on the note. You’ll create the title, a summary, and tags for the note. You will also identify potential people and action items defined in the note, the latter of which includes another struct, GeneratedNoteActionItem. Define GeneratedNoteActionItem used in the actionItems property of the NoteAnalysis struct by adding the following code after NoteAnalysis:

@Generable(description: "An actionable item or task extracted from the voice note.")
struct GeneratedNoteActionItem {
  @Guide(description: "Indicates if the action item has been completed.")
  let isCompleted: Bool
  @Guide(description: "The task or action to be completed.")
  let task: String
  @Guide(description: "People mentioned near or as part of the task.")
  let people: [String]
}

This struct defines the three items the model generates for an actionable item: its completion status, the task, and any people referenced in the task. Here, you state you want people mentioned near or as part of the task in the transcription.

You built the struct for generating action items from the voice notes, but it’s not the right format to persist this data alongside the note. You will often find that you need to adjust the data generated by Foundation Models or convert it to a format better suited for use within the app. In this case, a vital missing element is a unique id for the action item. You never want to use an LLM to generate anything unique or random, such as passwords, keys, or unique identifiers. The deterministic and pattern-matching behavior of LLMs makes them very poor at this type of task.

First, find VoiceNote.swift under the Models folder and add the following new code after the VoiceNote struct:

struct NoteActionItem: Identifiable, Codable, Equatable {
  let id: UUID
  var isCompleted: Bool
  let task: String
  let people: [String]

  init(from generatedItem: GeneratedNoteActionItem) {
    id = UUID()
    isCompleted = generatedItem.isCompleted
    task = generatedItem.task
    people = generatedItem.people
  }
}

This new struct implements several protocols to allow the existing store to persist the NoteActionItem. Since all the types are natively supported, you don’t need to do any additional work to implement them. You do include a custom initializer to make it easier to create a NoteActionItem from a Foundation Models-created GeneratedNoteActionItem object. Note that this initializer handles creating the unique ID using the UUID() initializer.

We’ll add these new fields to the note, saving them along with the other note information. Go to the VoiceNote struct and add these new properties after var transcript:

var summary: String?
var tags = [String]()
var people = [String]()
var actionItems = [NoteActionItem]()

You define the summary as an optional string since it will not exist until the analysis runs. You create the others as empty arrays, which is their initial state. It’s also a valid final state for both the people and actionItems properties, since not every note will reference other people or include action items.

Now that you’ve updated the app to store the analysis, it’s time to use Foundation Models to fill it out. In the next section, you’ll work on using Foundation Models to do this note analysis.

Using Foundation Models to Analyze a Note

Open NoteAnalysisService.swift. You’ll begin by producing some error states. Add the following code to the top of the file after the imports:

enum NoteAnalysisError: LocalizedError {
  case missingTranscript
  case transcriptTooLarge

  var errorDescription: String? {
    switch self {
    case .missingTranscript:
      "No transcript is available to analyze."
    case .transcriptTooLarge:
      "This transcript is too long for the current model."
    }
  }
}

private func fitsInContext(_ prompt: String) async -> Bool {
  let tokenLength: Int
  // 1
  let promptRatio = 0.33

  // 2
  if #available(iOS 26.4, *) {
    // 3
    let promptCalc = try? await
      SystemLanguageModel.default.tokenCount(for: prompt)
    if let promptCalc = promptCalc {
      tokenLength = promptCalc
    } else {
      tokenLength = prompt.count * 3 / 4
    }
  } else {
    // 4
    tokenLength = prompt.count * 3 / 4
  }
  
  // 5
  return tokenLength < Int(Double(SystemLanguageModel.default.contextSize) * promptRatio)
}

func analyze(transcript: String) async throws -> NoteAnalysis {
  // 1
  guard !transcript.isEmpty else {
    throw NoteAnalysisError.missingTranscript
  }

  // 2
  let session = LanguageModelSession()
  let prompt = """
    Analyze the following voice note transcription.

    Create:
    - A concise title of a few words
    - A two to three-sentence summary focused on the overall topic and
      key points.
    - Up to five short lowercase tags, each one up to three words.
    - Action items that the speaker intends to do, has committed to doing,
      or that are clearly implied.
    - A list of people mentioned in the note

    Do not invent details, deadlines, assignees, or people.
    If no action items are present, return an empty actionItems array.
    If no people are mentioned, return an empty people array.

    Transcription: \(transcript)
    """

  // 3
  guard await fitsInContext(prompt) else {
    throw NoteAnalysisError.transcriptTooLarge
  }

  // 4
  let response = try await session.respond(to: prompt, generating: NoteAnalysis.self)
  // 5
  #if DEBUG
  if #available(iOS 26.4, *) {
    let promptSize = (try? await
      SystemLanguageModel.default.tokenCount(for: prompt)) ?? 0
    let responseSize = SystemLanguageModel.default.contextSize
    let ratio = Double(promptSize) / Double(responseSize)
    print("Prompt Size: \(promptSize) Response Size: \(responseSize)")
    print("Ratio: \(ratio)")
  }
  #endif
  // 6
  return response.content
}

let analysis = try await noteAnalysis.analyze(transcript: noteTranscript)
updateTitle(analysis.title, for: note.id)
Pulling the title from larger note analysis.
Rovqogs tfu zeype ldos gaztij giyi okajxfig.

Updating Notes with Analysis

Open VoiceNoteStore.swift and begin by adding a reference to the analysis service after transcriptionService:

private let analysisService = NoteAnalysisService()

private func updateAnalysis(_ analysis: NoteAnalysis, for noteID: VoiceNote.ID) {
  guard let index = notes.firstIndex(
    where: { $0.id == noteID }
  ) else { return }
  notes[index].title = analysis.title
  notes[index].summary = analysis.summary
  notes[index].tags = analysis.tags
  notes[index].actionItems = analysis.actionItems.map {
    NoteActionItem(from: $0)
  }
  notes[index].people = analysis.people
  saveNotes()
}

func performAnalysis(_ transcript: String, for noteId: VoiceNote.ID) async {
  guard !transcript.isEmpty else { return }

  do {
    let analysis = try await analysisService.analyze(transcript: transcript)
    updateAnalysis(analysis, for: noteId)
  } catch {
    permissionMessage = (error as? LocalizedError)?.errorDescription
      ?? "This voice note could not be analyzed."
  }
}

let transcript = await transcribeRecording(note)
guard let transcript = transcript else { return }
await performAnalysis(transcript, for: note.id)
Pulling the title from larger note analysis.
Batzacj rda rognu qgit racnut tobu obobfzeg.

@State private var analyzingNote = false

if let transcript = note.transcript {
  Button {
    Task {
      analyzingNote = true
      await store.performAnalysis(
        transcript,
        for: note.id
      )
      analyzingNote = false
    }
  } label: {
    Image(systemName: "sparkles.2")
      .accessibilityLabel("Perform Analysis")
  }
  .disabled(analyzingNote)
}

Showing Note Analysis

With the analysis in place, the next step is to present this information to the user. It would be useful to replace the truncated transcript on the list of notes with a summary when available. To do this, open VoiceNoteRow.swift and find the private TranscriptSummary view. Find the line that reads } else if let transcript = note.transcript, !transcript.isEmpty {. Insert the following code before that line:

} else if let summary = note.summary {
  Text(summary)
    .font(.subheadline)
    .foregroundStyle(.primary)
    .lineLimit(2)
    .padding(.top, 4)
Showing summary instead of truncated transcript.
Cqixomr jurcujt orhteum id kwezrakeb tferykluvw.

import SwiftUI

struct VoiceNoteTextSection: View {
  let text: String?
  let title: String
  
  var body: some View {
    VStack(alignment: .leading, spacing: 12) {
      Text(title)
        .font(.headline)
      if let text = text {
        Text(text)
          .font(.body)
          .textSelection(.enabled)
          .fixedSize(horizontal: false, vertical: true)
      } else {
        Text("No \(title) available.")
          .font(.body.italic())
      }
    }
    .frame(maxWidth: .infinity, alignment: .leading)
    .padding(16)
  }
}

#Preview {
  VoiceNoteTextSection(
    text: "Sample Text",
    title: "Summary"
  )
}

VoiceNoteTextSection(text: note.summary, title: "Summary")
Voice note showing summary of the note.
Caule mebi flehipk fidvofv ad jvo wizo.

struct VoiceNoteTagsSection: View {
  var tags: [String]
  var title: String

  var body: some View {
    if !tags.isEmpty {
      VStack(alignment: .leading, spacing: 12) {
        Text(title)
          .font(.headline)
        FlowLayout(spacing: 8) {
          ForEach(tags, id: \.self) { tag in
            Text(tag)
              .font(.subheadline.weight(.medium))
              .foregroundStyle(.secondary)
              .padding(.horizontal, 12)
              .padding(.vertical, 6)
              .background(Color(.systemGray6), in: Capsule())
          }
        }
      }
      .padding(16)
    }
  }
}

#Preview {
  VoiceNoteTagsSection(
    tags: ["running", "marathon", "city park"],
    title: "Tags"
  )
}

VoiceNoteTagsSection(tags: note.tags, title: "Tags")
VoiceNoteTagsSection(tags: note.people, title: "People")
Showing people and tags for the note.
Ssajuyk goupri act zidb woj tsi tiri.

func updateTaskCompletion(_ status: Bool, for actionItem: UUID) {
  for noteIndex in notes.indices {
    let taskIndex = notes[noteIndex].actionItems.firstIndex(
      where: { $0.id == actionItem }
    )
    if let taskIndex = taskIndex {
      notes[noteIndex].actionItems[taskIndex].isCompleted = status
    }
  }
  saveNotes()
}

import SwiftUI

struct VoiceNoteTasksSection: View {
  var actionItems: [NoteActionItem]
  
  var body: some View {
    if !actionItems.isEmpty {
      VStack(alignment: .leading, spacing: 12) {
        Text("Action Items")
          .font(.headline)
        VStack(spacing: 8) {
          ForEach(actionItems) { task in
            ActionItemRow(task: task)
          }
        }
      }
      .frame(maxWidth: .infinity, alignment: .leading)
      .padding(16)
    }
  }
}

struct ActionItemRow: View {
  @EnvironmentObject private var store: VoiceNoteStore
  let task: NoteActionItem
  
  var body: some View {
    HStack(alignment: .center, spacing: 12) {
      Button {
        withAnimation {
          store.updateTaskCompletion(!task.isCompleted, for: task.id)
        }
      } label: {
        Image(systemName: task.isCompleted ?
              "checkmark.circle.fill" : "circle"
        )
        .font(.body)
        .foregroundStyle(.secondary)
        .frame(width: 20, height: 20)
        .padding(.top, 1)
      }
      VStack(alignment: .leading, spacing: 4) {
        Text(task.task)
          .font(.body)
          .strikethrough(task.isCompleted)
          .textSelection(.enabled)
          .fixedSize(horizontal: false, vertical: true)
        if !task.people.isEmpty {
          ForEach(task.people, id: \.self) { person in
            Label(person, systemImage: "person")
              .font(.caption)
              .strikethrough(task.isCompleted)
              .foregroundStyle(.secondary)
          }
        }
      }
    }
    .frame(maxWidth: .infinity, alignment: .leading)
    .padding(12)
    .background(
      Color(.secondarySystemGroupedBackground),
      in: RoundedRectangle(cornerRadius: 10)
    )
  }
}

VoiceNoteTasksSection(actionItems: note.actionItems)
Showing action items identified in the note.
Kkerusx uhheen okeff onoklomeav ey hlo tiyu.

Searching Note Analysis

After implementing transcripts for the voice notes, you added the ability to search for text within the transcripts. Now you’ll allow the user to search within these new fields. Open ContentView.swift and add a new enum to the top of the file:

enum SearchScope {
  case all
  case transcript
  case actionItems
  case people
}

@State private var searchScope: SearchScope = .all

.searchScopes($searchScope) {
  Text("All").tag(SearchScope.all)
  Text("Transcript").tag(SearchScope.transcript)
  Text("Action Items").tag(SearchScope.actionItems)
  Text("People").tag(SearchScope.people)
}

func matchesPeople(_ text: String) -> Bool {
  // 1
  people.contains { $0.localizedStandardContains(text) } ||
  // 2
  actionItems.contains {
    $0.people.contains { $0.localizedStandardContains(text) }
  }
}

func matchesActionItems(_ text: String) -> Bool {
  actionItems.contains {
    $0.task.localizedStandardContains(text) ||
    $0.people.contains { $0.localizedStandardContains(text) }
  }
}

func matchesTranscript(_ text: String) -> Bool {
  transcript?.localizedStandardContains(text) == true
}

func anyFieldMatches(_ text: String) -> Bool {
  title.localizedStandardContains(text) ||
  matchesTranscript(text) ||
  summary?.localizedStandardContains(text) == true ||
  matchesPeople(text) ||
  matchesActionItems(text)
}

var visibleNotes: [VoiceNote] {
  if searchText.isEmpty {
    return store.notes
  }
  
  return store.notes.filter { note in
    switch searchScope {
    case .all:
      note.anyFieldMatches(searchText)
    case .transcript:
      note.matchesTranscript(searchText)
    case .actionItems:
      note.matchesActionItems(searchText)
    case .people:
      note.matchesPeople(searchText)
    }
  }
}
Searching for a person mentioned in a note.
Xeopkquxv lax o benwol hoybaemav ih u note.

Showing Action Items

Create a new SwiftUI view named ActionItemsListView.swift. This view will show all action items contained in any voice note. Replace the contents of the view with:

import SwiftUI

struct ActionItemsListView: View {
  @EnvironmentObject private var store: VoiceNoteStore
  @State private var showCompleted = true

  var body: some View {
    VStack {
      // 1
      Toggle("Show Completed", isOn: $showCompleted)
        .padding()
      // 2
      ForEach(store.notes) { note in
        Text(note.title)
          .font(.headline)
        // 3
        ForEach(note.actionItems) { item in
          if !item.isCompleted || showCompleted {
            ActionItemRow(task: item)
          }
        }
        Divider()
      }
    }
    .navigationTitle("Action Items")
  }
}

#Preview {
  NavigationView {
    ActionItemsListView()
      .environmentObject(VoiceNoteStore.mock)
  }
}

Section("Action Items") {
  ActionItemsListView()
}
Collected Action Items on the first page of the view.
Fiwtihtag Aypeiz Arazn ux cye sushy kide ic yqu sied.

Conclusion

In this chapter, you took the data added through machine learning in Chapter Seven to produce a transcript and applied Apple Foundation Models to analyze and find useful information in that transcript. This takes what started as a voice note that a user could only listen to and expands it to produce a title and summary. It also collected possible action items and people mentioned in the recording. In two chapters, you turned a basic voice recording app into the start of a powerful tool for capturing and recalling information.

Key Points

  • When using Foundation Models, you can produce simple text responses or more specialized data structures using the @Generable and @Guide macros, which avoid parsing unstructured text responses. And you can mix and match as it is appropriate for your app.
  • You often need to convert data produced by Foundation Models or split it for presentation.
  • Never ask an LLM to generate unique identifiers, passwords, or anything requiring true randomness. The pattern-matching nature of language models makes them bad at these tasks.
  • It’s good to check the context length for prompts. You can determine this through experimentation. If you expect to exceed the context size often, consider chunking and summarization to reduce the data the model sees, as you learned in Chapter Four.
  • You can use non-streaming respond(to:generating:) for background analysis tasks that the user won’t watch in real time. Reserve streamed responses for interactions where the user is waiting and watching, as in the other apps in this book.
  • Placing the search logic into the data structure keeps the view layer clean and makes it easier to update or extend searching as the model changes.
7. Extending an App with Foundation Models
9. Conclusion
Have a technical question? Want to report a bug? You can ask questions and report bugs to the book authors in our official book forum here.
© 2026 Kodeco Inc.

You’re accessing parts of this content for free, with some sections shown as scrambled text. Unlock our entire catalogue of books and courses, with a Kodeco Personal Plan.

Unlock now