Report crawler hits

Crawlers never run the tracking snippet, so the browser never sees ChatGPT fetching your docs or Googlebot walking your sitemap. A few lines on your server report those requests to Lynq, and the Bots screen appears for the site as soon as the first one lands.

Only requests that look like a bot leave your server. A request from a person is never sent, so a visitor who opted out stays invisible on this path too. Lynq then decides what each one is: the same list of crawlers for every site, updated on our side, so a crawler that appears after you install is not missed.

What you need

An API key for the site with Send events from a server checked. Put it in your deployment’s secrets as LYNQ_API_KEY; never in a repository.

Next.js

Add a middleware. It runs before the response and reports after it, so it never slows a page and never fails one: if Lynq is unreachable the request is dropped.

middleware.ts

import type { NextFetchEvent, NextRequest } from "next/server";
import { NextResponse } from "next/server";
 
// Coarse on purpose: anything bot-shaped is sent, and Lynq decides what it is.
// A person's browser never matches, so their requests never leave the server.
const BOTLIKE = new RegExp(
  [
    "bot", "crawl", "spider", "slurp", "fetch", "scrap", "preview",
    "externalhit", "http\\.rb", "curl", "wget", "python", "go-http", "okhttp",
    "axios", "headless", "lighthouse", "-user\\b", "whatsapp", "bluesky",
    "telegram", "discord", "slack", "twitter", "linkedin", "facebook",
    "pinterest", "reddit", "embedly", "iframely",
  ].join("|"),
  "i"
);
 
export function middleware(req: NextRequest, event: NextFetchEvent) {
  const ua = req.headers.get("user-agent") ?? "";
  const key = process.env.LYNQ_API_KEY;
  if (key && BOTLIKE.test(ua)) {
    event.waitUntil(
      fetch("https://lynq.byharsh.com/api/bots", {
        method: "POST",
        headers: {
          authorization: `Bearer ${key}`,
          "content-type": "application/json",
        },
        body: JSON.stringify([{ ua, path: req.nextUrl.pathname, at: Date.now() }]),
      }).catch(() => {})
    );
  }
  return NextResponse.next();
}
 
// Pages and files crawlers ask for; not the framework's own assets.
export const config = { matcher: ["/((?!_next/|api/).*)"] };

In Next.js 16 the same file can be named proxy.ts with the function called proxy. If you already have a middleware, add the if block to it.

Node with Express

Report once the response has finished, so the status code goes along:

server.js

const BOTLIKE = new RegExp(
  [
    "bot", "crawl", "spider", "slurp", "fetch", "scrap", "preview",
    "externalhit", "http\\.rb", "curl", "wget", "python", "go-http", "okhttp",
    "axios", "headless", "lighthouse", "-user\\b", "whatsapp", "bluesky",
    "telegram", "discord", "slack", "twitter", "linkedin", "facebook",
    "pinterest", "reddit", "embedly", "iframely",
  ].join("|"),
  "i"
);
 
app.use((req, res, next) => {
  res.on("finish", () => {
    const ua = req.get("user-agent") || "";
    const key = process.env.LYNQ_API_KEY;
    if (!key || !BOTLIKE.test(ua)) return;
    fetch("https://lynq.byharsh.com/api/bots", {
      method: "POST",
      headers: { authorization: `Bearer ${key}`, "content-type": "application/json" },
      body: JSON.stringify([{ ua, path: req.path, status: res.statusCode, at: Date.now() }]),
    }).catch(() => {});
  });
  next();
});

Any other server works the same way: on each request that matches, POST a JSON array of { ua, path, status, at } to https://lynq.byharsh.com/api/bots with the key as a bearer token. Up to 50 entries per request, so a busy server can batch them. status and at are optional.

Check it works

Ask your site for a page the way a crawler would, then open the site in Lynq:

curl -A "Mozilla/5.0 (compatible; GPTBot/1.2; +https://openai.com/gptbot)" https://your-site.com/

Bots appears in the site’s navigation once the first hit is stored, with GPTBot on it. If it does not, check the key is the site’s own, has the server scope, and is set where the middleware runs; a wrong key gets a 401, a key without the scope a 403. Requests carrying a browser Origin header are refused, on purpose: a key that leaks into front-end code cannot be replayed from a page.

What is stored

For each site, day, crawler and page: how many times, and the last status seen. No IP address, no headers beyond the user agent, and nothing that describes a person. Crawler hits live in their own table and never touch your visitor numbers. They are kept for the same 24 months as everything else. The privacy page says the same in one paragraph.

Check it works The script tag