Commit Graph

331 Commits

Author SHA1 Message Date
Viktor Lofgren
182c0cf28e (control) Add warnings about domain data contamination 2024-01-25 18:26:15 +01:00
Viktor Lofgren
0b105b5986 (converter) Update hyperlink text for new crawl spec creation.
Fix minor typo.
2024-01-25 18:05:11 +01:00
Viktor Lofgren
cae1bad274 (*) Add download-sample action, refactor file storage
This changeset adds an action for downloading a set of sample data from downloads.marginalia.nu.

It also refactors out some leaky abstractions out of FileStorageService.  allocateTemporaryStorage has been renamed allocateStorage.  The storage was never temporary in any scenario...

It also doesn't take a storage base, as there was always only one valid option for this input.  The allocateStorage method finds the appropriate base itself.
2024-01-25 13:36:30 +01:00
Viktor Lofgren
958d64720e (control) Add a view for restarting aborted processes
This will avoid having to dig in the message queue to perform this relatively common task.

The control service was also refactored to extract common timestamp formatting logic out of the data objects and into the rendering.
2024-01-24 12:47:10 +01:00
Viktor Lofgren
805afad4fe (control) New GUI for exporting crawl data samples
Not going to win any beauty pageants, but this is pretty peripheral functionality.
2024-01-23 17:08:21 +01:00
Viktor Lofgren
ee7792596d (*) Fix broken test
Probably shouldn't have tests depending on external data like this...
2024-01-23 12:03:47 +01:00
Viktor Lofgren
f15dd06473 (index) Delayed close() of SearchIndexReader
This avoids concurrent access errors.  This is especially important when using Unsafe-based LongArrays, since we have concurrent access to the underlying memory-mapped file.  If pull the rug from under the caller by closing the file, we'll get a SIGSEGV.  Even with a "safe" MemorySegment, we'll get ugly stacktraces if we close the file while a thread is still accessing it.

So we spin up a thread that sleeps for a minute before actually unmapping the file, allowing any ongoing requests to wrap up.  This is 100% a hack, but it lets us get away with doing this without adding locks to the index readers.

Since this is "just" mmapped data, and this operation happens optimistically once a month, it should be safe if the call gets lost.
2024-01-23 11:08:41 +01:00
Viktor Lofgren
dd26819d66 (actor) Try to rare data race where a finished job is considered dead. 2024-01-22 21:22:38 +01:00
Viktor Lofgren
a6d257df5b (converter) Update Stackexchange sideload instruction
The sideload instruction in the stackexchange template was updated. The instruction now states that stackexchange data will be loaded from a directory on the server and directs users to a new documentation url for more detailed information.
2024-01-22 18:29:20 +01:00
Viktor Lofgren
51cdf46645 (control) Improve accessibility in search-to-ban template
This update enhances accessibility by associating labels with the corresponding checkboxes in the search-to-ban template.
2024-01-22 15:01:00 +01:00
Viktor Lofgren
40c9d2050f (control) Fully automatic conversion
Removed the need to have to run an external tool to pre-process the data in order to load stackexchange-style data into the search engine.

Removed the tool itself.

This stirred up some issues with the dependencies, that were due to both third-party:ing xz and importing it as a dependency.  This has been fixed, and :third-party:xz was removed.
2024-01-22 13:03:24 +01:00
Viktor Lofgren
b91ea1d7ca (control) Re-add gui for sideloading dirtrees 2024-01-20 18:09:40 +01:00
Viktor Lofgren
91c7960800 (crawler) Extract additional configuration properties
This commit extracts several previously hardcoded configuration properties, and makes then available through system.properties.

The documentation is updated to reflect the change.

Dead code was also removed in the process. CrawlSpecGenerator is left feeling a bit over-engineered still, since it's built for a more general case, where all other implementations but the current one are removed, but we'll leave it like this for now as it's fairly readable still.
2024-01-20 10:36:04 +01:00
Viktor Lofgren
2079a5574b (control) Update heading in restore backup template
Changed the heading in the partial restore backup page from "Load" to "Restore Backup".
2024-01-19 21:46:53 +01:00
Viktor Lofgren
27ffb8fa8a (converter) Integrate zim->db conversion into automatic encyclopedia processing workflow
Previously, in order to load encyclopedia data into the search engine, it was necessary to use the encyclopedia.marginalia.nu converter to first create a .db-file.  This isn't very ergonomic, so parts of that code-base was lifted in as a 3rd party library, and conversion from .zim to .db is now done automatically.

The output file name is based on the original filename, plus a crc32 hash and a .db-ending, to ensure we can recycle the data on repeat loads.
2024-01-19 13:59:03 +01:00
Viktor Lofgren
964419803a Fix broken test 2024-01-18 15:42:01 +01:00
Viktor Lofgren
6271d5d544 (mq) Add relation tracking between MQ messages for easier tracking and debugging.
The change adds a new column to the MESSAGE_QUEUE table called AUDIT_RELATED_ID.  This field is populated transparently, using a dictionary mapping Thread IDs to Message IDs, populated by the inbox handlers.

The existing RELATED_ID field has too many semantics associated with them,
among other things the FSM code uses them this field in tracking state changes.

The change set also improves the consistency of inbox names.  The IndexClient was buggy and populated its outbox with a UUID.  This is fixed. All Service2Service outboxes are now prefixed with 'pp:' to make them even easier to differentiate.
2024-01-18 15:08:27 +01:00
Viktor Lofgren
175bd310f5 (control) Message queue UX improvements 2024-01-18 13:05:50 +01:00
Viktor Lofgren
67ee6f4126 (control) Clean up filtering UX in Events table 2024-01-18 12:35:39 +01:00
Viktor Lofgren
18638c62de (control) Rephrase text 2024-01-18 11:53:10 +01:00
Viktor Lofgren
753d000788 (control) Add toggle for automatic loading of processed data 2024-01-18 11:52:58 +01:00
Viktor Lofgren
19e781b104 (control) Add basic input validation to node actions
Will present a simple error message when required fields aren't populated, instead of a cryptic HTTP status error.
2024-01-18 11:52:49 +01:00
Viktor Lofgren
aa2df327db (index) Prevent index from attempting to answer queries when no index data is loaded
This improves query times, and gets rid of exceptions in the logs when one of the index nodes doesn't have any data loaded, yet is configured to answer queries.
2024-01-18 11:05:45 +01:00
Viktor Lofgren
41cdb8f71b (control) Fix broken update button in the update-domain-ranking-set form
id property was on the wrong element.
2024-01-17 18:21:09 +01:00
Viktor Lofgren
304d4c9acf (control) Fix result ordering in the file storage listing view
In some scenarios, such as when restoring storage items from json-manifest on db failure, the file storage view would present the items in a non-chronological order.  Added a sort() operation to mitigate this.
2024-01-17 10:56:30 +01:00
Viktor Lofgren
7fd4c092e3 (control) Clean up UX and accessibility for new domain ranking sets.
The change also adds basic support for error messages in the GUI.
2024-01-17 10:47:14 +01:00
Viktor Lofgren
2fe5705542 (control) GUI for ranking sets
Still missing is some polish, forms don't have proper labels, validation is inconsistent, no error messages, etc.
2024-01-16 17:10:09 +01:00
Viktor Lofgren
e968365858 (index) Use new DomainRankingSets to configure ranking algos in index svc 2024-01-16 12:43:32 +01:00
Viktor Lofgren
5a62b3058f (query-api) Make the search set identifier a string value in the API
This will free the core marginalia search engine to use arbitrary search set definitions, while the app can use its hardcoded defaults.
2024-01-16 10:55:24 +01:00
Viktor Lofgren
e162406d40 (control) New control-side actors for cleaning up stale service heartbeats and message queue entries 2024-01-15 15:44:23 +01:00
Viktor Lofgren
c41e68aaab (control) New export actions for RSS/Atom feeds and term frequency data
This commit also refactors the executor a bit, and introduces a new converter-feature called data-extractors for this class of jobs.
2024-01-15 14:54:26 +01:00
Viktor Lofgren
4665af6c42 (control) Move export data endpoint to actions controller 2024-01-15 11:06:22 +01:00
Viktor Lofgren
c0b15427fe (control) New crawl view should use radio buttons as multiple specs aren't supported 2024-01-15 11:03:47 +01:00
Viktor Lofgren
f29a9d972d (control) Move 'new crawl spec' to /node/:id/actions, out of /node/:id/storage 2024-01-15 11:02:00 +01:00
Viktor Lofgren
b192373ae7 (control) Highlight unavailable items (creating, deleting) in node actions views 2024-01-15 10:47:54 +01:00
Viktor Lofgren
c042650382 (docs) Improve query service documentation 2024-01-13 21:16:45 +01:00
Viktor Lofgren
4c62065e74 (install) Add two separate templates for the install script
One template is for the full Marginalia Search style install, and the other is for a barebones install with no Marginalia-related fluff.
2024-01-13 18:27:42 +01:00
Viktor Lofgren
c9fb45c85f (search) Fix control.hideMarginaliaApp handling 2024-01-13 17:24:15 +01:00
Viktor Lofgren
7c6e18f7a7 (*) Overhaul settings and properties
Use a system.properties file to configure the system.  This is loaded statically by MainClass or ProcessMainClass.  Update the property names to be more consistent, and update the documentations to reflect the changes.
2024-01-13 17:12:18 +01:00
Viktor Lofgren
ecd9c35233 (control) Clean up the event log
* Generate fewer uninteresting event messages.
* Display fewer irrelevant fields in the overview table.
2024-01-13 13:28:02 +01:00
Viktor Lofgren
71e32c57d9 (control) Add better timestamps for the events and message queue views
Adjust display precision based on distance into the past, full ms-accurate timestamps available via hover-action.
2024-01-13 13:04:56 +01:00
Viktor Lofgren
2fefd0e4e3 (control) Add better timestamps for the events and message queue views
Adjust display precision based on distance into the past, full ms-accurate timestamps available via hover-action.
2024-01-13 13:03:52 +01:00
Viktor Lofgren
81eaf79a25 (control) UX polish 2024-01-13 12:31:13 +01:00
Viktor Lofgren
8dea7217a6 (control) UX fixes, node GUI doesn't break when an executor service goes offline. 2024-01-13 12:17:30 +01:00
Viktor Lofgren
c0fb9e17e8 (control) Add filter dropdown to message queue table
This makes inspecting the queues for processes much easier, as it's otherwise
often these important messages are drowned out by FSM chatter.
2024-01-12 18:46:17 +01:00
Viktor Lofgren
83776a8dce (control) Wean the ExportDataActor off EC_DOMAIN_LINK
The EC_DOMAIN_LINK table is deprecated and slated for removal, use QueryClient.getAllDomainLinks() instead.

The ExportDataActor now uses the QueryClient appropriately.  The CSV format was also changed to quote the values, to prevent e.g. Excel from interpreting the comma as a decimal separator when previewing the file.

Finally the form for triggering an export was overhauled.
2024-01-12 17:09:11 +01:00
Viktor Lofgren
98c0972619 (control) Add a summary table for Actors in the Node overview 2024-01-12 16:32:15 +01:00
Viktor Lofgren
56d832d661 (control) Adjust the margins of the headings to be consistent 2024-01-12 16:16:57 +01:00
Viktor Lofgren
de3a350afe (control) Disable broken actions and mark the actions view as WIP 2024-01-12 16:16:39 +01:00
Viktor Lofgren
708a741960 (test) Clean up test usage of migrations
Several tests were manually running migrations in a large copy-paste blob of code.  This makes the test less useful as it's possible to break the code while keeping the tests green by introducing a new migration that never gets run in the tests, and it's also difficult to reason about what the tests are doing.

A new test helper library is introduced with a TestMigrationLoader that can both run Flyway migrations, or load specific migrations in the cases a specific set of migrations need to be loaded.   Existing tests are migrated to use the new code.
2024-01-12 15:55:50 +01:00