Viktor Lofgren
7b5ec6b98f
(executor-service) Embed dist/ in executor-service's docker image
2023-10-19 17:48:34 +02:00
Viktor Lofgren
23526f6d1a
(executor) Executor service now pulls DomainType list for CRAWL on "recrawl"
...
This is an automatic integration with the submit-site repo on github and also
crawl-queue.
2023-10-19 17:48:34 +02:00
Viktor Lofgren
c0930ead0f
(doc) Update conceptual-overview.svg
2023-10-19 17:48:34 +02:00
Viktor Lofgren
809b3ee023
(control) Update GUI for crawl specs. They are now less important than they were before.
2023-10-19 17:48:34 +02:00
Viktor Lofgren
23f0c79fba
(control) GUI for data sets/domain types.
2023-10-19 17:48:34 +02:00
Viktor Lofgren
81dd3809e9
(*) WIP Add node affinity to EC_DOMAIN
...
Very messy commit due to fractalline yak shaving
2023-10-19 17:48:34 +02:00
Viktor Lofgren
2bf0c4497d
(*) Tool for unfcking old crawl data so that it aligns with the new style IDs
2023-10-19 17:48:34 +02:00
Viktor Lofgren
93122bdd18
(run) Add two nodes to the demo setup
2023-10-16 17:37:26 +02:00
Viktor Lofgren
978550f809
(executor-service) Retire features-convert and move the corresponding packages into the executor service.
2023-10-16 15:43:46 +02:00
Viktor Lofgren
84fea0fd05
(node) Nodes auto-start their monitor actors.
2023-10-16 15:33:22 +02:00
Viktor Lofgren
2df3e0f881
(node) Nodes auto-configure on start-up instead of requiring manual configuration.
2023-10-16 14:46:35 +02:00
Viktor Lofgren
c98117f69d
(actor) FS monitor should pick up stuff in BACKUP as well.
2023-10-16 14:37:36 +02:00
Viktor Lofgren
ede5d1f890
(actor) Give process spawners more easily recognizable names.
2023-10-16 14:19:00 +02:00
Viktor Lofgren
39911e3acd
(control) Fix incorrect storage base and clean up GUI for data
2023-10-16 13:30:26 +02:00
Viktor Lofgren
3d1c15ef99
(client) Refactor liveness monitor
2023-10-16 12:34:01 +02:00
Viktor Lofgren
f718482e98
(client) Fix tests
2023-10-16 12:12:16 +02:00
Viktor Lofgren
8dafd13cd7
(client) Fix executor tests
2023-10-16 12:02:57 +02:00
Viktor Lofgren
0b19b28a64
(file-storage) Delete unused code
2023-10-16 12:02:57 +02:00
Viktor Lofgren
c245f7ce3a
(control) Bootstrapify review-domains and search-to-ban views.
2023-10-15 22:04:23 +02:00
Viktor Lofgren
607d647483
(control) Remove services listing view
2023-10-15 21:48:55 +02:00
Viktor Lofgren
9a38a455c9
(control/exec) File listings in control GUI
2023-10-15 19:15:44 +02:00
Viktor Lofgren
16e0738731
(*) Get multi-node routing working.
2023-10-15 18:38:30 +02:00
Viktor Lofgren
eacbf87979
(control) New list and form for index nodes.
2023-10-14 21:46:52 +02:00
Viktor Lofgren
108b4cb648
(service) Keep disabled multi-noded services dormant when they are configured to be disabled.
2023-10-14 20:58:55 +02:00
Viktor Lofgren
a9dff407a1
(config/db) Clean up migrations
2023-10-14 20:34:03 +02:00
Viktor Lofgren
9e26109e36
(reverse-index) Don't always POST
2023-10-14 16:48:29 +02:00
Viktor Lofgren
6308a8dfcd
(control) Node configuration
2023-10-14 16:47:52 +02:00
Viktor Lofgren
4baf9527d7
(*) WIP Control GUI redesign, executor-service, multi-node mq
...
This turned out to be very difficult to do in small isolated steps.
* Design overhaul of the control gui using bootstrap
* Move the actors out of control-service into to a new executor-service, that can be run on multiple nodes
* Add node-affinity to message queue
2023-10-14 12:08:43 +02:00
Viktor Lofgren
199c459697
(*) Add node-affinity to services, processes and file storage.
2023-10-10 12:32:22 +02:00
Viktor Lofgren
61288c5e68
(service, client) First steps towards multiple nodedness
2023-10-09 22:13:27 +02:00
Viktor Lofgren
8375237de5
(converter) Add special keyword for websites with a tilde url.
2023-10-09 17:02:32 +02:00
Viktor
c8d820c17b
Merge pull request #53 from MarginaliaSearch/standalone-index
...
Move ranking to the index-service, and query parsing to a new query-service; separate out the search-service
2023-10-09 15:42:06 +02:00
Viktor Lofgren
6319b8ef51
(api-service) Improved testability, always set content type to application/json
2023-10-09 15:39:34 +02:00
Viktor Lofgren
397a85eaa4
(query-service) Apply blacklisting to search results
2023-10-09 15:18:53 +02:00
Viktor Lofgren
3889c4bdd9
(refactor) Remove features-search and update documentation
2023-10-09 15:12:30 +02:00
Viktor Lofgren
c899f1cb85
(docs) Update documentation to reflect new query service
2023-10-09 14:56:59 +02:00
Viktor Lofgren
d8956c51d0
(refactor) Remove api:search-api
...
Application services should not have an API, but purely act as clients
to the core services (which should always have an API).
2023-10-09 14:42:33 +02:00
Viktor Lofgren
5dd55c7cad
(refactor) Rename satellite services to application services
...
This is a better descriptor, since they now all implement different applications on top of the core services' APIs.
2023-10-09 13:45:45 +02:00
Viktor Lofgren
c0e61d4c87
(refactor) Move search service into services-satellite
2023-10-09 13:40:01 +02:00
Viktor Lofgren
97e17282ab
(query-service) Move query parsing from search-service to the new query service.
2023-10-09 13:27:44 +02:00
Viktor Lofgren
94c882af7d
(query-service) Provide delegate of IndexApi's query functionality.
...
This is an intermediate step in the process of introducing the query-service as a proxy between search and index.
2023-10-08 22:22:26 +02:00
Viktor Lofgren
89c6d85f2f
(query-service) Create new empty 'query-service' service
2023-10-08 17:31:50 +02:00
Viktor Lofgren
cf366c602f
(search) Refactor SearchQueryIndexService in preparation for feature extraction.
...
Prefer working on DecoratedSearchResultItem in favor of UrlDetails.
2023-10-08 17:15:41 +02:00
Viktor Lofgren
77ccab7d80
(index) Move linkdb to index from search.
...
This makes index complete in the sense that you can deploy an index instance and build a complete separate application on top of it, without having to go through the Marginalia-laden search service.
2023-10-08 16:48:35 +02:00
Viktor Lofgren
f51ba63742
(search) Remove dead file
2023-10-07 21:05:06 +02:00
Viktor Lofgren
9044518be5
(search) Fix broken link to git repo
2023-10-07 19:43:22 +02:00
Viktor Lofgren
9e0367eef4
(search) Filter blacklisted items in API query service as well
2023-10-07 16:16:04 +02:00
Viktor Lofgren
235bb6c1b9
(control) Administrative QOL improvement, GUI for banning spam
2023-10-07 15:45:50 +02:00
Viktor Lofgren
49344d7ea8
(control) Administrative QOL improvement, GUI for banning spam
2023-10-07 15:43:18 +02:00
Viktor Lofgren
1b418d77ff
(search) We got some new IP ranges to work with for the crawler
2023-10-07 13:41:55 +02:00
