WarcDB
Data
web archive database
SHOW HN2022
How HN took it
3 positive, 9 neutral or asking, 1 mixed, 1 critical, out of 14 comments about the project. Thread score +0.08.
Idea: 3 for, 1 against
The thread, comment by comment
Each row is a top-level comment and its replies, in HN's order. Select a cell to open that comment on HN.
positive neutral or mixed critical about another commenter or something else the maker
What HN said
What people liked
SQLite offers strong well-documented guarantees and transactions for archive integrity
From 1 comment
Anecdotal evidence, but I produced a medium-size crawl in the past (~20TB compressed). I used distributed resources with off-the-shelf libraries (i.e. warcprox etc.) and managed…
uniqueuid on HN
Idea of SQLite as a portable archive format is appealing
From 2 comments
Oh hi, thanks for building this! I haven't had the chance to play with it, but my hunch is that sqlite for warc can fill a great niche and would be much more portable (and…
uniqueuid on HN> What is the advantage of moving around .sqlite-files, over just loading the (compressed) WARCs into sqlite databases when you need them? I suppose there could be made a case…
mynameismon on HN
Objections
Loading WARCs into SQLite on demand would suffice; moving .sqlite files adds little
From 1 comment
> PS: I do believe that at some point .sqlite will become the defacto standard for such initiatives. Sure, it's not text... but it's pretty close. What is the advantage of moving…
marginalia_nu on HN
WARC is already portable and standard; binary content does not need a database
From 2 comments
> PS: I do believe that at some point .sqlite will become the defacto standard for such initiatives. Sure, it's not text... but it's pretty close. What is the advantage of moving…
marginalia_nu on HNA bit late to this thread, but I think WARC is a reasonable format for raw HTTP traffic. We should definitely have better tools to ensure WARC files produced are valid, and…
ikreymer on HN
Requests
Suggestions for virtual table or Parquet/DuckDB alternatives
From 2 comments
Have you every played with SQLite virtual tables (https://sqlite.org/vtab.html) - they could allow to provide an SQLite interface while keeping the same structure on disk. Though…
nlohmann on HNDuckDB would probably be a way better option and works amazingly well on top of parquet (https://duckdb.org/docs/data/parquet)
m_ke on HN
Questions
What is the advantage of distributing SQLite files over loading compressed WARCs?
From 1 comment
> PS: I do believe that at some point .sqlite will become the defacto standard for such initiatives. Sure, it's not text... but it's pretty close. What is the advantage of moving…
marginalia_nu on HN
Alternatives named
DuckDB over Parquet
From 1 comment
DuckDB would probably be a way better option and works amazingly well on top of parquet (https://duckdb.org/docs/data/parquet)
m_ke on HN
CommonCrawl Parquet files with sqlite-parquet-vtable
From 1 comment
Good point. Actually CommonCrawl provides Parquet files for their archives too. And there's this vtable for Parquet extension. https://github.com/cldellow/sqlite-parquet-vtable…
fforflo on HN
ZIM format for offline website copies
From 1 comment
End-user facing applications usually don't consume website crawls, do they? That's impractical for many reasons, the sheer size alone being perhaps the biggest obstacle. If you…
marginalia_nu on HN
WACZ format
From 1 comment
A bit late to this thread, but I think WARC is a reasonable format for raw HTTP traffic. We should definitely have better tools to ensure WARC files produced are valid, and…
ikreymer on HN
WARCTools collection
From 1 comment
Did you add this to https://github.com/dhamaniasad/WARCTools ?
TedDoesntTalk on HN