Skip to main content

We're excited to be creating more content for you! Additional modules are being added regularly.

Cartographer Consulting

Module 2.4

Tokens & Token Management

When we chat with AI models, we think in terms of words and sentences. However, the model thinks in tokens – smaller chunks of text that might be full words, parts of words, or even individual characters. Think of tokens as the building blocks that language models use to understand and generate text.

The Importance of Tokens

Tokens comprise of everything that the AI sends and receives. A single word might be comprised of one token, multiple tokens, or even share tokens with another word, depending on how common the word or pattern is. Consider the following examples:

  • "cat" = 1 token (common word)
  • "hamburger" = 2 tokens ("ham" + "burger")
  • "indescribable" = 4 tokens ("in" + "describ" + "able" + "e")

Understanding them is crucial because every interaction with a language model has a token limit – a maximum amount of text it can process at once. This limit applies to both your input (the prompt) and the model's output (the response). If you exceed these limits, you may lose context, have your response truncated (shortened), receive an incomplete analysis, or even receive no response at all (in the event that capacity limits are reached).

Fortunately, tokens are fairly easy to understand since they tend to be broken down in predictable ways:

  • Common Words and Patterns - "The quick brown fox" might be processed as: ["The", "quick", "brown", "fox"]. Each word is a single token because they're common in English.

  • Technical Terms - "Neural network architecture" might become: ["Neural", "net", "work", "architecture"]. Technical terms often break into multiple tokens unless they're very common in the training data.

  • Special Characters - "email@example.com" might tokenize as: ["email", "@", "example", ".", "com"]. Special characters usually become their own tokens.

Examples of Token Management

As you continue refining your prompt engineering skills, consider the question: "Can I make this shorter without losing my objective?" Ultimately, the concept of tokenization boils down to being as direct and concise as possible. This will holistically improve your output. Consider the following examples to see how you could create an ideal prompt.

Example 1: Summarizing an Article

  • Summarize this extremely long blog post about artificial intelligence and machine learning and their applications in modern technology and how they affect our daily lives and what we can expect in the future...
  • Summarize key points about AI applications and future impacts. Focus on: technology, daily life, future trends.

Example 2: Technical Documentation

  • Please provide a comprehensive and detailed technical documentation including all possible use cases, implementation details, architectural considerations, deployment strategies, and maintenance procedures for this system...
  • Please create technical documentation for the following: core functionality, implementation guide, deployment steps, maintenance plan

Example 3: Creative Writing

  • Write a story that incorporates all of these elements: a mysterious old house, a curious child, a magical artifact, a hidden doorway, a wise old mentor, a dangerous quest, unexpected allies, and a surprising twist at the end...
  • Write a children's adventure story about finding a magical artifact in an old house. Include a mentor figure and a surprise ending.

Practice Tokenizing

Start a new conversation. Ask the AI to rate your prompts and to provide suggestions for simplifying. Compare the differences.

AI: How can I help you?  
  You: I want to practice tokenizing with my prompts. Please provide me with a few exercises to help me do so. When I respond, please rate my prompt using a 5 point scale and provide a few suggestions for how it could be improved (including example prompts that I could have done).
AI: Exercise 1: Conciseness vs. Context
  • Task: Write a prompt that clearly and concisely requests a summary of an article about climate change's economic impact.
  • Focus: Reduce unnecessary words while maintaining clarity and intent.
  • Example: "Summarize the main points of an article about how climate change affects global economies, focusing on key industries and policy responses."
  
  You: Please summarize this article: https://www.noaa.gov/education/resource-collections/climate/climate-change-impacts

Provide key takeaways and realistic action items I can do
AI: Rating Your Prompt:
  • Clarity: 4/5 – The request is straightforward, but it could benefit from slight rewording for specificity (e.g., defining the depth of "key takeaways").
  • Focus: 5/5 – Your focus on summarization and actionable steps is clear.
  • Conciseness: 5/5 – The prompt is succinct and avoids unnecessary details.
  • Creativity: 4/5 – Adding a unique perspective (e.g., tailoring actions to a personal or professional context) could enhance engagement.
  • Effectiveness: 5/5 – The AI is likely to respond well to this, as it clearly outlines the task.
  • Overall Score: 4.6/5

Suggestions for Improvement
  1. Add more context about the depth of the summary (e.g., "1-2 paragraphs" or "bullet points").
  2. Specify the type of action items you're interested in, such as those for personal lifestyle, community involvement, or policy advocacy.
  3. Use active language to reinforce the need for tangible outputs.
Revised Prompt Examples
  1. "Summarize the NOAA article on climate change impacts in 1-2 paragraphs. Highlight 3-5 key takeaways and provide 3 realistic action items I can implement in my daily life or community."
  2. "Read the NOAA resource on climate change impacts and distill the main points into 3 bullet points. Also, suggest specific actions I can take as an individual to reduce my carbon footprint and advocate for change."
  3. "From the NOAA article on climate change impacts, summarize the top concerns it identifies. Provide actionable steps for someone living in a coastal area to mitigate and adapt to these challenges."
Try refining your prompt and let me know when you're ready for another evaluation!		  

 

Strategies for Token Efficiency

There are a couple key areas to focus on improving your use of tokens:

  1. Be Concise but Clear
    1. Instead of: "I would like you to please create a comprehensive analysis of the following text, taking into account all possible interpretations and meanings..."
    2. Use: "Analyze this text's key themes and meanings:"
  2. Use Structure Efficiently
    1. Instead of: "First, I want you to review the document. Then, I want you to identify the main points. After that, I want you to summarize each section. Finally, I want you to provide recommendations."
    2. Use: "Review document: Identify main points, Summarize sections, Provide recommendations"
  3. Leverage Context
    1. Instead of repeating information in every prompt, establish context once and refer back to it:
    2. First prompt: "We're analyzing quarterly sales data for a retail company."
    3. Follow-up: "Compare Q1 to Q2."
  4. Ask Questions
    1. Instead of assuming that the AI has all of the information, ask for clarification:
    2. First prompt: "Here is all the data. Is there anything else you need?"
    3. Response: "Thank you for the information. I see some gaps..."

Putting It All Together

In our next section, we'll explore how to maintain context effectively while managing tokens, ensuring your conversations with AI models remain both efficient and productive. We'll look at strategies for breaking down complex tasks into token-efficient chunks while maintaining clarity and purpose.

Previous Module
Keep Learning