The CTO who guards the first five minutes
Dima Grossman is the cofounder and CTO of Novu, which he describes as the biggest open source communication platform on GitHub. He runs a team of about 15 people, backed by some 450 open source contributors. Novu is an API-first product, but a company's first impression of it happens in the dashboard, so Dima classifies the critical paths as the first five minutes of using Novu. He needs QA to keep those five minutes working in every packaging of the product.
A QA service the team couldn't see into
Novu started with Cypress, which stopped working at scale: the more test cases the team added, the more challenging the suite became. A Playwright suite came next, and then an outsourced QA service took over the Playwright work entirely, writing the tests and fixing them when something broke or turned flaky.
As a way to offload maintenance, the service worked. Somebody else dealt with flakiness and updated test cases when the product changed, and Dima says having a person handle that was extremely helpful. The catch was that there was no product behind the person. Novu never interacted with a system, only with a Slack channel.
Four packagings, one Slack channel
Novu ships as a cloud platform and an enterprise edition, across staging and production. Testing all four means new test patterns and dynamic triggers whenever the product moves. On the service, the bigger changes meant explaining the request in Slack and waiting.
"Making the bigger changes, the communication was not very helpful, and we lacked the visibility into what's happening behind the scenes."
The misalignment ran deeper than turnaround time. Dima had always assumed professional services were a step on the way to a product, and this vendor was walking in the other direction, further into services. When Novu asked about using the product, the team learned the vendor was investing in the services side instead.
Going back to maintaining their own Playwright suite was not on the table either, because, as Dima puts it, the market is not heading in this direction. So the team kept quietly looking for something that behaved like a product.
A test that signed itself up
The looking ended on X, where Dima ran into TesterArmy's posts and decided to give it a spin. The risk was small, since the worst the agent could do was log into staging. And the first thing the agent did was what Dima calls the most difficult part of establishing any testing environment: creating users, logging in, and handling credentials.
"With almost no guidance, it was able to sign up and navigate the logged-in product. The moment I saw it navigate within the system, I knew the base is there."
Once the agent was inside, the rest of the setup stopped being plumbing. Dima describes it as "mapping customer journeys in human language, almost describing what our users are doing," and letting TesterArmy walk them.
Novu was the first open source project TesterArmy supported. The paid subscription came later. Dima tried the product during the YC batch, automated his first flow quickly, and became one of the first people to post about it publicly.
"I usually consider myself an early adopter with a lot of technology, so it was quite easy to just give it a go and try it out."
User flows instead of selectors
Plain language on live staging
Novu's saved tests cover the dashboard's critical paths and run as a smoke test on staging after every merge, with Slack notifications when something breaks. Each test is written the way Dima describes the product to peers: "describe user flows in simple language and achievable steps: click that, do that, or achieve this particular goal." An average run completes in about four minutes.
What changed most is how the suite behaves when the product moves. On earlier tooling, a UI shift meant changing the whole test suite, which Dima calls the time-consuming kind of flakiness, the kind a rerun cannot fix.
"I can see from the recordings that the agent sometimes takes a different path if the UI has changed, or there is a new pop-up. The test can adapt to it, and we're not seeing those types of flakiness."
He is deliberate about what remains. The content editor, one of the most complicated areas of Novu, still produces the occasional combination that passes one day and fails the next. But the class of flakiness that used to force rewrites is gone, and by Dima's estimate flaky tests across the suite have dropped by half. What is left can be rerun instead of rebuilt.
Dynamic tests on every pull request
The capability Novu could not get before is the dynamic PR flow. When a pull request goes up, TesterArmy plans and runs tests against it before the merge. For the team, the runs are "another layer of confidence when shipping changes, especially ones touching the dashboard." Novu had tried to build the same flow with Cursor's cloud agents, which produce a similar video of a computer-use session, but in Dima's words the setup didn't nail it in the end.
The difference Dima points to is where the tests run: "TesterArmy runs against the live staging environment, not a local copy of the codebase, which gives a different angle of confidence." His ballpark for the effect is about 30 percent faster time to merge. He volunteered the caveat himself: nobody at Novu tracks it, and the number is what the added confidence amounts to in practice.
Real sign-ups without auth mocks
The other thing that removed a whole category of work is authentication. Novu used to mock or bypass parts of its auth flow so tests could get in. Now TesterArmy signs up as a real user and handles verification itself. "Previously we had to do a lot of hacks for things like OTP codes. It worked out of the box with TesterArmy, and it removed all of this complexity from our side."