Reindex a Page of a Project Using CustomGPT RAG API – A Detailed Tutorial

Watch the video above for a live demo of this tutorial! (coming soon)

1. Introduction

Hi there! In this tutorial, we’re going to learn how to reindex a page of a project using the CustomGPT RAG API. In our discussions, we refer to “projects” as “agents” (though the code still shows “project”). Reindexing a page can be useful when you update your content and want the agent to reflect those changes. We’ll guide you through setting up your environment, creating an agent, retrieving its pages, and finally reindexing a page—all in simple, easy-to-follow steps.

Make sure you have read our Getting Started with CustomGPT.ai for New Developers blog to get an overview of the entire platform.

Get the cookbook link here – https://github.com/Poll-The-People/customgpt-cookbook/blob/main/examples/Reindex_page_of_a_project.ipynb

2. Setting Up the Environment

Before we start, we need to get our workspace ready in Google Colab. This involves setting up the API endpoint, your API token, and importing the necessary libraries.

Code Sample:

# setup API URL and API Token

api_endpoint = 'https://app.customgpt.ai/api/v1/'

api_token = 'ADD_YOUR_API_TOKEN'

headers = {

    'Content-type': 'application/json',

    'Authorization': 'Bearer ' + api_token

}

# imports

import requests

import json

What This Does:

• API Endpoint & Token: Sets the base URL for all API calls and uses your API token for authentication. Make sure to replace ‘ADD_YOUR_API_TOKEN‘ with your actual token.
  
• Headers: Prepares the HTTP headers including the content type (JSON) and the authorization token.
  
• Imports: Loads the requests library for making HTTP calls and the json module for handling JSON data.

Your token authenticates your requests and grants you access to manage your agents securely. Make sure your key is ready and correctly inserted into the code.

Get API keys

To get your API key, there are two ways:

Method 1 – Via Agent

1. Agent > All Agents.
2. Select your agent and go to deploy, click on the API key section and create an API. 

Method 2 – Via Profile section.

1. Go to profile (top right corner of your screen)
2. Click on My Profile
3. You will see the screen something like this (below screenshot). Here you can click on “Create API key”, give it a name and copy the key.

Please save this secret key somewhere safe and accessible. For security reasons, You won’t be able to view it again through your CustomGPT.ai account. If you lose this secret key, you’ll need to generate a new one.

Now that our environment is set up, let’s move on to what you need before diving into the code.

3. Prerequisites

Before you start, ensure you have the following:

• CustomGPT.ai Account: Sign up and log in at CustomGPT.ai.
  
• API Key: Generate your API token from your account dashboard.
  
• Basic Python Knowledge: Familiarity with Python and REST APIs will help you follow along.
  
• Google Colab: We’re using Google Colab, so you don’t need to set anything up locally!
  

With these prerequisites checked, we’re ready to create an agent.

4. Creating an Agent (Project)

Even though the code uses the word “project,” think of it as creating your agent’s workspace where your chatbot pulls content from. In this step, we’ll create an agent using a sitemap as the content source.

Code Sample:

# Give a name to your project

project_name = 'Example ChatBot using Sitemap'

sitemap_path = 'https://adorosario.github.io/small-sitemap.xml'

payload = json.dumps({

    "project_name": project_name,

    "sitemap_path": sitemap_path

})

url = api_endpoint + 'projects'

create_project = requests.request('POST', url, headers=headers, data=payload)

print(create_project.text)

What This Does:

• Agent Details: Sets the agent’s name and assigns a sitemap URL as its source.
  
• Payload Creation: Converts these details into a JSON string that the API understands.
  
• POST Request: Sends a POST request to create your agent on the CustomGPT platform.
  
• Output: Prints the JSON response containing details about the created agent, including its unique ID.
  

Now that your agent is created, let’s fetch its pages.

5. Getting the Agent’s Pages

Next, we retrieve the list of pages associated with your agent. This helps you identify which page to reindex.

Code Sample:

data = json.loads(create_project.text)["data"]

project_id = data["id"]

# Update the project

url = api_endpoint + 'projects' + f"/{project_id}" + '/pages'

project_pages = requests.request('GET', url, headers=headers)

print(project_pages.text)

What This Does:

• Parse Response: Parses the JSON response from the agent creation to extract the agent’s unique ID.
  
• Build Pages URL: Constructs the URL to access the agent’s pages using the extracted ID.
  
• GET Request: Sends a GET request to fetch the list of pages.
  
• Output: Prints the JSON response containing details of all pages associated with the agent.
  

After running this section, you should see a list of pages. Now, let’s pick a page and reindex it.

6. Reindexing a Page of the Agent

With your agent’s pages in hand, you can now reindex one of the pages. Reindexing ensures that any updates to the page’s content are reflected in your agent.

Code Sample:

page_id = json.loads(project_pages.text)['data']['pages']['data'][0]['id']

# Reindex the project page

url = api_endpoint + 'projects' + f"/{project_id}" + '/pages' + f"/{page_id}/reindex"

project_pages = requests.request('POST', url, headers=headers)

print(project_pages.text)

What This Does:

• Extract Page ID: Parses the list of pages to get the ID of the first page. You could choose another page by modifying this selection.
  
• Build Reindex URL: Constructs the URL for reindexing the selected page by appending /reindex to the page’s URL.
  
• POST Request for Reindexing: Sends a POST request to the reindex endpoint to trigger the reindexing process.
  
• Output:
  Prints the response from the API, which confirms that the reindexing request has been processed.
  

With your page reindexed, your agent is now updated with the latest content. Let’s quickly go over what to do if something goes wrong.

7. Troubleshooting Common Issues

Here are a few common issues you might encounter and some tips to resolve them:

• Invalid API Token: Double-check that you replaced ‘ADD_YOUR_API_TOKEN’ with your actual API token.
  
• JSON Parsing Errors: If you get an error while parsing JSON, print out the raw response to inspect its structure.
  
• Connection Problems: Make sure your internet connection is stable and the API endpoint URL is correct.
  
• Unexpected Output: Verify that the payloads and URLs are correctly formatted. Use the API documentation as a reference if needed.
  

If these troubleshooting steps don’t help, consider reaching out to the CustomGPT support community.

8. Conclusion

Awesome work! In this tutorial, you learned how to:

• Set up your Google Colab environment and configure the API endpoint, token, and headers.
  
• Create an agent (project) using a sitemap as the content source.
  
• Retrieve the list of pages for your agent.
  
• Reindex a specific page to update its content on the platform.
  

By following these steps, you can ensure that your agent always reflects the latest updates in your content. If you have any questions or need further assistance, feel free to check out the official CustomGPT documentation or join the community forum.

Happy coding, and enjoy building and maintaining your custom chatbot solutions with CustomGPT!