GrowthBook
Data
ab testing
SHOW HN2021
How HN took it
4 positive, 20 neutral or asking, 0 mixed, 7 critical, out of 31 comments about the project. Thread score -0.11.
Design: 1 for, 6 againstIdea: 2 for, 1 againstExecution: 1 for, 0 against
The thread, comment by comment
Each row is a top-level comment and its replies, in HN's order. Select a cell to open that comment on HN.
positive neutral or mixed critical about another commenter or something else the maker
What HN said
What people liked
Polished and well executed
From 2 comments
Wow this looks very polished. (Out of frustration with Optimizely) I created and maintain a couple of A/B test open source projects[0][1] but the statistical analysis was always…
gingerlime on HNI think you guys really did a good job with something that's hard to do well. Kudos
jaggednad on HN
Bayesian approach reduces peeking problems
From 1 comment
Hi, thanks for the questions! 1. We don't store any raw user data. We pull things like mean and standard deviation from data sources, run the statistics, and store the result. 2.…
jrdorn on HN
Objections
MongoDB is not open source (SSPL) and is a poor choice for some users
From 2 comments
Hi- thanks for making this and open sourcing it! I would also suggest supporting an alternative to MongoDB. Postgres using jsonb is a great option. I try to always use and…
ensignavenger on HN> PostgreSQL using JSONB however is a way to hurt your head Really? I have used it pretty extensively and like it... I don't do a lot of complex manipulations though, it might be…
ensignavenger on HN
Short tests give untrustworthy results and the tool may encourage poor decisions
From 1 comment
Interesting, thanks. 1. How do you get around needing session level data instead of aggregate data when working with non parametric KPIs? GA in particular is notorious for…
Dyac on HN
Metrics coupled to experimentation is a design concern
From 1 comment
Coupling metrics to experimentation is a huge red flag. You should offload that onto DBT or some other data modeling tools.
ablearcher83 on HN
Configuration cannot be stored in the code repository
From 1 comment
From the looks of it, it doesn't look like the configuration can be stored in the code repository itself. This is one of the key things to do - treating configuration as a code…
tehlike on HN
Requests
Support PostgreSQL or SQLite instead of MongoDB
From 3 comments
Nice project! PostgreSQL has excellent JSON support these days, including the ability to query nested fields which may be beneficial to your project.
sojournerc on HNHonestly, I'd love to see it just use SQLite as a backend. If it's just storing results, that seems feasible and it would reduce the complexity of the tech stack.
nrjames on HNHi- thanks for making this and open sourcing it! I would also suggest supporting an alternative to MongoDB. Postgres using jsonb is a great option. I try to always use and…
ensignavenger on HN
Store configuration in code (YAML) for self-hosting
From 1 comment
That's on our roadmap. We originally built the tool as a multi-tenant hosted platform so storing configs in a database made the most sense initially. For self hosting, we want to…
jrdorn on HN
MySQL/MariaDB data source support
From 1 comment
We plan to add MySQL/MariaDB support soon which should let you use Matomo data as long as you have raw SQL access. For cloud-hosted Matomo, we would have to use the reporting…
jrdorn on HN
Mobile SDK support
From 1 comment
Would this be useful for A/B testing mobile app features as well?
dreamer7 on HN
Questions
Does it store raw user data or query data sources?
From 1 comment
Hi. Regular A/B experimenter here. This looks like a great tool! Does your system store the stats, or does it trust the stats to be stored in, eg. GA, and then just allow you to…
Dyac on HN
Can it analyse tests run in other tools?
From 1 comment
Brilliant, thank you. Can you use the system to analyse results of tests it didn't run? ie. If I run tests using some SAAS that only supports frequentist stats could I use your…
Dyac on HN
How does it handle non-parametric KPIs and sampling?
From 1 comment
Interesting, thanks. 1. How do you get around needing session level data instead of aggregate data when working with non parametric KPIs? GA in particular is notorious for…
Dyac on HN
How does it compare to Facebook Ax?
From 1 comment
How does this compare to Facebook Ax?
dimitry12 on HN
Alternatives named
Analytics Toolkit
From 1 comment
Wow this looks very polished. (Out of frustration with Optimizely) I created and maintain a couple of A/B test open source projects[0][1] but the statistical analysis was always…
gingerlime on HN
Alephbet
From 1 comment
Wow this looks very polished. (Out of frustration with Optimizely) I created and maintain a couple of A/B test open source projects[0][1] but the statistical analysis was always…
gingerlime on HN
Facebook Ax
From 1 comment
How does this compare to Facebook Ax?
dimitry12 on HN
sixpack
From 1 comment
This reminds me of https://github.com/sixpack/sixpack which I've been eyeballing for years.
JensRantil on HN