161sh/CatgirlIntelligenceAgency
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity

(converter) Look at properties when deciding pool size

Browse Source 
Look at whether the property 'system.conserveProperty' is enabled when deciding he default pool size for the converter.

If true, a much more conservative default is used, limiting the risk of running out of memory.
This commit is contained in:
Viktor Lofgren 2024-02-12 16:24:19 +01:00
parent 5a1087dbf9
commit 02dd5c5853
1 changed files with 5 additions and 4 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

9
code/processes/converting-process/src/main/java/nu/marginalia/converting/ConverterMain.java
View File

@ -120,10 +120,11 @@ public class ConverterMain extends ProcessMainClass {
public void convert(CrawlPlan plan) throws Exception { public void convert(CrawlPlan plan) throws Exception {
final int maxPoolSize = final int defaultPoolSize = Boolean.getBoolean("system.conserveMemory")
Integer.getInteger("converter.poolSize", ? Math.clamp(Runtime.getRuntime().availableProcessors() / 2, 1, 4) // <-- conserve memory
Math.clamp(Runtime.getRuntime().availableProcessors() - 2, 1, 32) : Math.clamp(Runtime.getRuntime().availableProcessors() - 2, 1, 32); // <-- a more liberal pool size
);
final int maxPoolSize = Integer.getInteger("converter.poolSize", defaultPoolSize);
try (BatchingWorkLog batchingWorkLog = new BatchingWorkLogImpl(plan.process.getLogFile()); try (BatchingWorkLog batchingWorkLog = new BatchingWorkLogImpl(plan.process.getLogFile());
ConverterWriter converterWriter = new ConverterWriter(batchingWorkLog, plan.process.getDir())) ConverterWriter converterWriter = new ConverterWriter(batchingWorkLog, plan.process.getDir()))