Building a Community Intelligence Tracker with Jev
If you are building at the cutting edge of AI models and infrastructure, you’ve probably come across TypeSafe’s new model, Jev.
Instead of asking a model to generate more text, you can ask Jev to decide between explicit choices — what matters, how important it is, who should own it, and what should happen next.
I wanted to try Jev on a problem where the output could directly change what a team does next.
Community conversations felt like a good test: bugs, feature requests, support questions and interesting use cases are already there, but they arrive mixed into hundreds of ordinary messages.
So, I built a small tracker that turns that stream into a queue of requests that needs attention:
Start with the decision you need
My first instinct was to classify messages: bug, feature request, docs question, general chat.
That was useful, but still left most of the work to a human. What I really wanted to know was: Does this need attention? Who should look at it? What should happen next?
So each conversation gets a small set of judgments:
- Signal type
- Priority
- Whether someone needs a response
- User state
- Suggested owner
- Suggested action
If a field cannot change someone’s next action, I probably don’t need it.
My mental model was, if a field cannot change someone’s next action, I probably don’t need it.That is also what made Jev interesting for this use case. The task is less about generating prose and more about making a series of bounded decisions.
Keep the ingestion layer replaceable
The tracker does not need to care whether the community lives in Discord, Slack or a forum. For a real deployment, messages should come through the platform’s official bot, API, webhook or export mechanism.
The rest of the pipeline only needs a normalized message with things like its ID, author, text, timestamp and reply context.
Iteration & Testing
I tested the pipeline on a dataset of 500 community messages. Honestly, real community data is messy. It contains timestamps, reaction text, attachment labels, duplicates and quoted parent messages.
Like one interesting issue i found was quoted replies. For example - If three replies all contain the same original product question, a model may reasonably identify three product signals. The model is not necessarily wrong — the input has made one underlying conversation look like three independent ones.
The unit you ask the model to judge matters.In a production version, I would preserve parent messages and reply context as separate fields instead of flattening the entire thread into one block of text.
Jev becomes the judgment layer
Once a message is clean, Jev turns it into structured decisions. A message can enter the system as plain community text and leave with something closer to:
priority: high owner: engineering signal: bug action: investigate
This output is much easier to build around than a paragraph explaining what the model thinks. It can be filtered, routed, counted or handed to another system.
Testing the output
I initially pushed the results into a Google Sheet. That was intentional. I wanted to inspect rows and ask a very basic question: Would someone on the team actually find this useful?
I then built a lightweight frontend using vercel called Jevvy with three views:
- Needs attention
- Product signals
- Interesting builds
From the 436-message run, the tracker surfaced: 36 high-priority signals 35 high product signals 30 bugs 16 feature requests 9 documentation gaps 96 messages needing a staff response
What I learned
The code for this project was not the hard part, i literally spent less than 2 hours building this and a few cents on Jev.
The more interesting decisions were:
- what should the model actually judge?
- what counts as one conversation?
- how much context should it see?
- which outputs are useful enough to change someone’s next action?
The next thing I would measure is trust
I would not immediately automate replies or create engineering tickets from this. First I would manually review a sample of the high-priority queue and label each result: good signal / useful but routed incorrectly / noise
For this kind of system, I think precision matters more than recall. If a team repeatedly opens “Needs attention” and finds things that do not need attention, the queue stops being useful very quickly.
Once the judgment layer earns trust, then the downstream automation becomes more interesting.
Want to build it? I put the implementation, setup steps and code in the GitHub repo: repo link