List All Pages of a Project with CustomGPT RAG API – Step by Step Developers Tutorial

Watch the demo video embedded at the top of this post for a live walkthrough of this tutorial!

1. Introduction

Hey there! In this tutorial, we’re going to learn how to list all pages of a project using the CustomGPT RAG API in Google Colab. Even though the code refers to “projects,” we like to call them “agents” now, so keep that in mind as you follow along. We’ll guide you through creating an agent with a sitemap, and then we’ll show you how to retrieve all its pages. Let’s dive in!

Make sure you have read our Getting Started with CustomGPT.ai for New Developers blog to get an overview of the entire platform.

Get the cookbook link here – https://github.com/Poll-The-People/customgpt-cookbook/blob/main/examples/List_all_pages_of_a_project.ipynb

2. Setting Up the Environment

Before you start, let’s set up our Google Colab notebook. We need to define the RAG API endpoint, set our RAG API token, and import the required libraries.

What we’re doing:

• RAG API URL & Token: We specify the base URL for the RAG API and use our RAG API token for authentication.
  
• Headers: We prepare our headers to include the content type and our authorization token.
  
• Imports: We import the requests library for HTTP calls and json to handle JSON data.
  

# setup RAG API URL and RAG API Token

api_endpoint = 'https://app.customgpt.ai/api/v1/'

api_token = 'ADD_YOUR_RAG_API_TOKEN_HERE'

headers = {

    'Content-type': 'application/json',

    'Authorization': 'Bearer ' + api_token

}

# imports

import requests

import json

Now that our environment is set up, let’s check that you have everything you need to proceed.

3. Prerequisites to Use CustomGPT RAG APIs

Before you dive into the code, make sure you have the following:

• CustomGPT.ai Account: Sign up and log in at CustomGPT.ai.
  
• RAG API Key: Generate your RAG API token from your account dashboard.
  
• Basic Python Knowledge: Familiarity with Python and REST APIs will help you follow along.
  
• Google Colab: We’ll use Google Colab to run the notebook—no local setup required!
  

4. Get Your RAG API Key

Your token authenticates your requests and grants you access to manage your agents securely. Make sure your key is ready and correctly inserted into the code.

To get your RAG API key, there are two ways:

Method 1 – Via Agent

1. Agent > All Agents.
2. Select your agent and go to deploy, click on the RAG API key section and create an RAG API. 

Method 2 – Via Profile section.

1. Go to profile (top right corner of your screen)
2. Click on My Profile
3. You will see the screen something like this (below screenshot). Here you can click on “Create API key”, give it a name and copy the key.

Please save this secret key somewhere safe and accessible. For security reasons, You won’t be able to view it again through your CustomGPT.ai account. If you lose this secret key, you’ll need to generate a new one.

Now that you have your prerequisites in place, let’s create your first agent!

5. Creating an Agent (Project) for Your Account

We’ll begin by creating an agent using a sitemap. Although the code still uses “project,” think of this as your agent’s workspace where your chatbot will pull its content from.

What the code does:

• Agent Details: Sets the name of your agent and provides a sitemap URL as its content source.
  
• Payload Creation: Converts these details into a JSON-formatted string.
  
• API Request: Sends a POST request to the API endpoint to create your agent.
  
• Output: Prints the response which includes a unique ID for your newly created agent.
  

# Give a name to your project

project_name = 'Example ChatBot using Sitemap'

sitemap_path = 'https://adorosario.github.io/small-sitemap.xml'

payload = json.dumps({

    "project_name": project_name,

    "sitemap_path": sitemap_path

})

url = api_endpoint + 'projects'

create_project = requests.request('POST', url, headers=headers, data=payload)

print(create_project.text)

After running this section, you should see output similar to the sample provided, which confirms your agent has been created. Now, let’s move on to listing all the pages associated with your agent.

6. Getting (Listing) All Pages of the Agent

This section explains how to list all pages that are part of your agent. We will extract the agent’s unique ID from the previous step and use it to build the URL for fetching pages. Then, we’ll send a GET request to retrieve all page details.

What the code does:

• Extract Agent ID: Parses the JSON response from agent creation to get the agent’s ID.
  
• Build Pages URL: Constructs the URL by appending the agent’s ID and the /pages endpoint.
  
• GET Request: Sends the GET request to list all pages of your agent.
  
• Output: Prints the JSON response, which contains all pages associated with your agent.

data = json.loads(create_project.text)["data"]

project_id = data["id"]

# Update the project

url = api_endpoint + 'projects' + f"/{project_id}" + '/pages'

project_pages = requests.request('GET', url, headers=headers)

print(project_pages.text)

Once you run this section, you should see a detailed JSON response listing all the pages of your agent. This completes the main part of our tutorial!

7. Troubleshooting Common Issues

Even if everything is straightforward, sometimes issues can pop up. Here are a few common problems and some simple solutions:

• Invalid API Token: Make sure you’ve replaced ‘ADD_YOUR_RAG_API_TOKEN_HERE‘ with your actual RAG API token.
  
• JSON Parsing Errors: If you encounter errors while parsing, print out the raw response to inspect its structure.
  
• Connection Problems: Verify your internet connection and confirm that the RAG API endpoint URL is correct.
  
• Unexpected Outputs: Double-check that your payload and URLs are correctly formatted and refer to the right endpoints.
  

If you continue to face issues, consult the official CustomGPT documentation or join the community support forum for help.

8. Conclusion

Great work! In this tutorial, you learned how to:

• Set up your Google Colab environment to interact with the CustomGPT RAG API.
  
• Create an agent using a sitemap as the content source.
  
• List all pages associated with your agent by extracting and using its unique ID.
  

By following these steps, you can easily manage and explore the content of your agent. With this beginner-friendly approach, you’re now ready to experiment further and build even more advanced chatbot solutions using CustomGPT.

If you have any questions or need further assistance, feel free to check out the CustomGPT documentation or ask in our community. Happy coding, and enjoy building your custom chatbot solutions with CustomGPT!