Report crawler hits

Crawlers never run the tracking snippet, so the browser never sees ChatGPT fetching your docs or Googlebot walking your sitemap. A few lines on your server report those requests to Lynq, and the Bots screen appears for the site as soon as the first one lands.

Only requests that look like a bot leave your server. A request from a person is never sent, so a visitor who opted out stays invisible on this path too. Lynq then decides what each one is: the same list of crawlers for every site, updated on our side, so a crawler that appears after you install is not missed.

What you need

An API key for the site with Send events from a server checked. Put it in your deployment’s secrets as LYNQ_API_KEY; never in a repository.

And a way to tell a crawler’s user agent from a browser’s. The snippets use isbot, a small package with nothing else underneath it:

npm install isbot

Matching the user agent yourself gets this wrong in both directions. Some phone models have bot in their name, so their owners would be reported as crawlers; and plenty of real crawlers carry no such word at all, so they would be missed.

Any check you trust does the job, though. Lynq drops anything it does not recognise as a crawler, so the only thing that matters on your side is that a request from a person is not sent.

Next.js

Add a middleware. It runs before the response and reports after it, so it never slows a page and never fails one: if Lynq is unreachable the request is dropped.

middleware.ts

import { isbot } from "isbot";
import type { NextFetchEvent, NextRequest } from "next/server";
import { NextResponse } from "next/server";
 
// Only a crawler's request is reported, and Lynq decides which crawler it was.
// A person's request never leaves your server.
export function middleware(req: NextRequest, event: NextFetchEvent) {
  const ua = req.headers.get("user-agent") ?? "";
  const key = process.env.LYNQ_API_KEY;
  if (key && isbot(ua)) {
    event.waitUntil(
      fetch("https://lynq.byharsh.com/api/bots", {
        method: "POST",
        headers: {
          authorization: `Bearer ${key}`,
          "content-type": "application/json",
        },
        body: JSON.stringify([{ ua, path: req.nextUrl.pathname, at: Date.now() }]),
      }).catch(() => {})
    );
  }
  return NextResponse.next();
}
 
// Pages and files crawlers ask for; not the framework's own assets.
export const config = { matcher: ["/((?!_next/|api/).*)"] };

In Next.js 16 the same file can be named proxy.ts with the function called proxy. If you already have a middleware, add the if block to it.

Node with Express

Report once the response has finished, so the status code goes along:

server.js

const { isbot } = require("isbot");
 
app.use((req, res, next) => {
  res.on("finish", () => {
    const ua = req.get("user-agent") || "";
    const key = process.env.LYNQ_API_KEY;
    if (!key || !isbot(ua)) return;
    fetch("https://lynq.byharsh.com/api/bots", {
      method: "POST",
      headers: { authorization: `Bearer ${key}`, "content-type": "application/json" },
      body: JSON.stringify([{ ua, path: req.path, status: res.statusCode, at: Date.now() }]),
    }).catch(() => {});
  });
  next();
});

Any other server works the same way: for each request whose user agent belongs to a crawler rather than a browser, POST a JSON array of { ua, path, status, at } to https://lynq.byharsh.com/api/bots with the key as a bearer token. Up to 50 entries per request, so a busy server can batch them. status and at are optional.

Check it works

Ask your site for a page the way a crawler would, then open the site in Lynq:

curl -A "Mozilla/5.0 (compatible; GPTBot/1.2; +https://openai.com/gptbot)" https://your-site.com/

Bots appears in the site’s navigation once the first hit is stored, with GPTBot on it. If it does not, check the key is the site’s own, has the server scope, and is set where the middleware runs; a wrong key gets a 401, a key without the scope a 403. A key does not work from a browser page, on purpose: one that leaks into front-end code is useless there.

What is stored

For each site, day, crawler and page: how many times, and the last status seen. No IP address, no headers beyond the user agent, and nothing that describes a person. Crawler hits are counted apart from visits and never touch your visitor numbers. They are kept for the same 24 months as everything else. The privacy page says the same in one paragraph.

Check it works The script tag