Training AI Chatbot with Website or Sitemap using MyBotChat API

MyBotChat support both website and sitemap. For website, our web crawler will try to discover every link on your website and scrap all content to train the bot. For sitemap, it will only scan each link in the sitemap for content.

Prerequisite

• You need to have an account with MyBot and created a chatbot. If you have not done this step yet, visit this get-started turorial page to start.
• Genrated an API Key

Custom SSE Event

Because fetching website content or sitemap could take a long time, it is important to use Server Side Event to get real time status and to avoid HTTP timeout error.

There are three type of event the API generate: status, end and failed

• status: status update of the server
• failed: error status from server
• end: end of training, request is terminated

Example Raw SSE data:

event: status
data: Training in progress

event: failed
data: Error occured on server

event: end
data:

1. Train with Sitemap

We are using this url for traning chatbot with sitemap: /api/Training/Website/Sitemap/{ChatbotId} with POST method

Example in C#

using System.Net.Http.Headers;
using System.Net.Http.Json;

namespace SSEConsumer
{
    class Program
    {
        static async Task Main()
        {
            string chatbotId = "772af073-234f-465a-b44c-5a72fddce458";
            string apiKey = "e5fe8a65-90cd-4b72-9a1e-b583c1190694";
            string sitemapUrl = "https://example.com/sitemap.xml";

            string url = $"https://mybot.chat/api/Training/Website/Sitemap/{chatbotId}";
            using (var client = new HttpClient())
            {
                client.DefaultRequestHeaders.Authorization = new AuthenticationHeaderValue("Bearer", apiKey);
                client.DefaultRequestHeaders.Add("Accept", "text/event-stream");

                var payload = new { sitemapUrl };
                var jsonContent = JsonContent.Create(payload);

                using var response = await client.PostAsync(url, jsonContent);
                response.EnsureSuccessStatusCode();

                using var stream = await response.Content.ReadAsStreamAsync();

                using var reader = new StreamReader(stream);

                string currentEventName = "";
                string data = "";

                while (!reader.EndOfStream)
                {
                    string line = await reader.ReadLineAsync();

                    if (!string.IsNullOrEmpty(line))
                    {
                        if (line.StartsWith("event:"))
                        {
                            currentEventName = line.Substring(6).Trim();
                        }
                        else if (line.StartsWith("data:"))
                        {
                            data = line.Substring(5).Trim();
                        }
                        Console.WriteLine(line);
                    }
                    else
                    {
                        // Process event data with the event name
                        Console.WriteLine($"Event: {currentEventName}, Data: {data}");
                        if (currentEventName == "end")
                        {
                            Console.WriteLine("Finished training");
                            client.CancelPendingRequests();
                        }
                        else if (currentEventName == "failed")
                        {
                            Console.WriteLine($"Failed training, reason: {data}");
                        }
                    }
                }
            }
        }
    }
}