The Square's crawler: what it reads and how to reach us
If you see StateGovTheSquare in your server logs, that is us. The Square is a free app from StateGov that shows Arkansans what their local boards are deciding. Its collector reads public meeting records, one request at a time.
Last checked .
To report a problem or ask us to slow down, write to tal@naturalstateintel.com. Include your site's address and what you are seeing.
Who we are
StateGov builds tools for following Arkansas government. The Square is our free app for local government: city councils, quorum courts, school boards and the boards under them. It is live on the web at square.stategov.io and coming to the App Store and Google Play. It covers five Arkansas towns today: Little Rock, Hot Springs, Russellville, Blytheville and Osceola. To show residents what their local boards are deciding, it reads what those boards post.
The crawler is called the collector inside StateGov. It runs on StateGov's servers, not on anyone's personal computer.
What it reads
Public meeting records that anyone can open without an account:
- Meeting calendars and notices.
- Agendas, agenda packets and minutes.
- Lists of board members, such as council members, justices of the peace and school board members.
It never logs in. It reads the pages where your documents are listed, then the documents themselves.
How to recognize us
Every request the collector sends to a city, county or school site carries this User-Agent:
StateGovTheSquare/1.0 (StateGov The Square; +https://stategov.io/square/crawler; contact: tal@naturalstateintel.com)It does not pose as a web browser, and it does not route requests through proxies. We do not have a fixed IP address to publish, so match on the User-Agent.
How fast it reads
- One at a time
- The collector sends one request at a time to each site and waits for the answer before it sends the next.
- Gap
- At least 5 seconds from the end of one request to the start of the next. Some sites get a longer gap: www.popecountyar.gov gets 10 seconds.
- Shared platforms
- Boards on one hosted platform (for example BoardBook, Diligent or Thrillshare) share one gap, so the platform sees one request at a time however many boards use it.
- Retries
- After a
429, a server error or a network error, it tries the same page at most twice more, waiting longer each time and honoringRetry-Afterup to 60 seconds. It does not retry a403.
Report a problem or ask us to slow down
Write to tal@naturalstateintel.com. Include your site's address, what you are seeing in your logs, and what you would like us to change.
Questions
Why is a crawler reading our site?
The Square shows residents what their local boards are deciding. Your board's agendas and minutes are public records, and reading them where you post them is how The Square keeps up.
How much traffic will we see?
One request at a time, with at least 5 seconds between requests. If that is too much for your server, write to tal@naturalstateintel.com.