6a04cdfddf
Deprecate the LoadUrl instruction entirely. We no longer need to be told upfront about which URLs to expect, as IDs are generated from the domain id and document ordinal. For now, we no longer store new URLs in different domains. We need to re-implement this somehow, probably in a different job or a as a different output. |
||
|---|---|---|
| .. | ||
| src/main/java/nu/marginalia/converting | ||
| build.gradle | ||
| readme.md | ||
Converting Models
Contains models shared by the converting-process and loading-process.
Design
The two processes communicate through a file-based protocol. The converter serializes instructions to file, which are deserialized by the loader and fed into an instructions.
The instructions implement a visitor pattern.
Conceptually the pattern can be thought of a bit like remote function calls over file, or a crude instructions-based programming language.
This
producer.foo("cat");
producer.bar("milk", "eggs", "bread");
translates through this paradigm, to this:
(producer)
writeInstruction(DoFoo("Cat"))
writeInstruction(DoBar("Milk", "Eggs", "Bread"))
(consumer)
while read instruction:
interpreter.apply(instruction)
(Interpreter)
doFoo(animal):
...
doBar(ingredients):
...
(doFoo)
DoFoo(animal):
apply(interpreter):
interpreter.foo(animal)
(doBar)
DoBar(ingredients):
apply(interpreter):
interpreter.bar(ingredients)