JoeBot

JoeBot is the chatbot behind the button in my nav. If you ask it how it was made, it'll tell you it's a GPT wrapper I built into my site, which is true. I think the more interesting part is everything I had to put around it so it's safe to leave on a public site, and so it sounds a bit more like me.

The first version

The first version of the endpoint basically took whatever the browser sent and passed it straight to OpenAI, with my API key on the server. I thought keeping the key on the server was enough, but anyone could still send whatever they wanted through it, and someone found it and ran my balance down before I'd even noticed.

What the route checks

Now the route checks everything before it goes anywhere near OpenAI. The conversation can only contain normal messages from the visitor and the bot, twenty at most and each under two thousand characters, and if the last one isn't from the visitor the whole request gets rejected. The model and the system prompt are set on the server, and so is a cap on how long a reply can be, so nobody can point it at a different model or get it to write them an essay. The only thing the browser can actually send is the messages.

On top of that there's a rate limit of ten requests a minute per IP, kept in memory.

function isRateLimited(ip: string): boolean {
  const now = Date.now()
  const timestamps = (rateLimitBuckets.get(ip) ?? []).filter(
    (t) => now - t < RATE_LIMIT_WINDOW_MS
  )
  if (timestamps.length >= RATE_LIMIT_MAX_REQUESTS) {
    rateLimitBuckets.set(ip, timestamps)
    return true
  }
  timestamps.push(now)
  rateLimitBuckets.set(ip, timestamps)
  if (rateLimitBuckets.size > 10_000) rateLimitBuckets.clear()
  return false
}

Rate limiting

The problem with that one is it only lasts as long as the server instance holding it, and Vercel runs more than one when there's traffic, so the same person can land on a fresh instance and their count starts from zero again. I'd call it best effort really. The limit that actually holds is a Vercel firewall rule in front of the route, ten requests every sixty seconds per IP, which blocks anything over that before it reaches my code. Since then I've also set a hard spend cap in the OpenAI dashboard, so even if something does get through there's a ceiling on what it can cost me. There's no API for that, so you have to go in and click it yourself.

The prompt

Most of the actual design work went into the system prompt. I wrote it a bit like a brief for someone standing in for me, so it talks about me in the third person and sticks to British English. I also banned phrases like "happy to help" and "let me know if you need anything", because I think that's what gives a chatbot away. There are a few things I don't want it bringing up at all either, like my day rate or who I'm talking to about work.

The other thing people try with any chatbot is getting it to ignore its instructions, or to repeat them back in some other form. So the prompt has a rule against giving away any version of itself, and it covers pretty much every way I could think of someone asking. Before I trusted it with anyone I spent a while trying to get the prompt out of it myself, and I couldn't.

Replies

I didn't want long replies arriving as one big paragraph, because that feels more like getting an email than talking to someone. So replies get split into a few bubbles that come in one at a time, and the pause before each one depends on how long the previous one takes to read, with a cap so a long answer doesn't drag. If you've got reduced motion turned on, it shows the whole reply at once.

So yeah, it's still a wrapper, but I'm a lot more comfortable leaving it on a public site now, and I think it's quite nice to talk to.

I've put the code on GitHub if you want to build one for your own site. Everything in there works the way it does here apart from the prompt, which is a template with my details taken out, because publishing my real prompt would've kind of defeated the point of keeping it private.