161sh/CatgirlIntelligenceAgency
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity

(sample-exporter) Add some limits on sizes and lengths

Browse Source 
Tar files will reject entries with filenames over 100b, so we need a limit there.  Also added a maximum size limit to keep the file sizes reasonable.
This commit is contained in:
Viktor Lofgren 2024-01-25 11:51:53 +01:00
parent 0846606b12
commit 1b8b97b8ec
1 changed files with 8 additions and 1 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

9
code/features-convert/data-extractors/src/main/java/nu/marginalia/extractor/SampleDataExporter.java
View File

@ -37,7 +37,14 @@ public class SampleDataExporter {
List<WorkLogEntry> entriesAll = new ArrayList<>(100_000);
for (var item : WorkLog.iterable(crawlerLogFile)) {
if (item.cnt() < 2) continue;
if (item.cnt() < 2) // this one's too small
continue;
if (item.cnt() > 5000) // this one's too big
continue;
if (item.relPath().length() > 90) // this one's too long
continue; // TAR file name limit is 100, but we add some extra for good measure
// this one's just right
entriesAll.add(item);
}