NoSQL based IO, caching, and logging adapters for Hyperlambda
This project contains alternative NoSQL file system services, implementing IFileService, IFolderService, and
IStreamService, allowing you to use it as an interchangeable "virtual file system" for cases where you want
to have 100% stateless magic instances. This is important if you're using Magic in a Kubernetes cluster or
something similar, load balancing invocations, virtually resolving files and folders towards a virtual file system.
If you take this path you'll have to configure your "appsettings.json" file such as illustrated further
down in this document. The project also contains an ILogger implementation service you can use that will create
log entries in a NoSQL based storage of your choice, in addition to an IMagicCache implementation service
allowing you to use a NoSQL based out of process cache implementation. See further down in this document for
details about how to configure these parts.
Configuration
The primary configuration for the project to apply for your "appsettings.json" file can be found below. Notice,
although you can create as many NoSQL cluster connection settings as you wish, and use these in your own code,
the IO, caching, and logging services requires you to use generic as your cluster name, and you cannot change
this.
Notice - The "credentials" parts above are optional, and can be ommitted if you don't require
authentication to connect to your database. You can also provide multiple hosts as contact points,
by separating multiple IP addresses or hosts by a comma (,). The above "port" parts is also
optional.
The above configures the adapter to use 127.0.0.1 as the host for your contact point or cluster. To configure
the adapter to store files and folders inside of its CQL based database, you can alternatively add something such
as follows to your file.
Notice - This project also includes a "mixed" service implementation, storing system files and folders
in the file system while storing dynamic files and folders in a NoSQL database of your choice. These
can be found in the same namespace, but are named "MixedXXX" instead of using the above names.
If you want to use a CQL based log implementation, you'll have to configure Magic to use the NoSQL
ILogger service such as follows.
To use the alternative CQL based file storage system you'll have to create your "magic_files" keyspace and its
"files" table as follows.
create keyspace if not exists magic_files with replication = { 'class': 'NetworkTopologyStrategy', 'replication_factor': 5 };
use magic_files;
create table if not exists files(
tenant text,
cloudlet text,
folder text,
filename text,
content blob,
primary key((tenant, cloudlet), folder, filename));
To use the alternative CQL based logging implementation you'll have to create your "magic_log" keyspace and its
"log" table as follows.
create keyspace if not exists magic_log with replication = { 'class': 'NetworkTopologyStrategy', 'replication_factor': 3 };
use magic_log;
create table if not exists log(
tenant text,
cloudlet text,
created timeuuid,
type text,
content text,
exception text,
meta frozen<map<text, text>>,
primary key((tenant, cloudlet), created)) with clustering order by (created desc);
alter table log with default_time_to_live = 1209600;
Notice - The above setting for TTL implies log items will be automatically evicted after 14 days,
since 1,209,600 seconds implies 14 days. Depending upon your needs you might want to increase or decrease this
setting. However, to avoid exhausting your hard drive space over time on your server, we suggest you do not
entirely remove the TTL setting.
To use the alternative CQL based caching implementation you'll have to create your "magic_cache" keyspace and its
"cache" table as follows.
create keyspace if not exists magic_cache with replication = { 'class': 'NetworkTopologyStrategy', 'replication_factor': 3 };
use magic_cache;
create table if not exists cache(
tenant text,
cloudlet text,
key text,
value text,
primary key((tenant, cloudlet), key));
Adding existing files into NoSQL database
The following Hyperlambda will insert all your existing files and folders into your cluster keyspace, allowing you to
play around with an existing CQL file system implementation. Notice, you'll have to change the [.tenant] and
[.cloudlet] values to resemble the absolute root folder for your Magic backend. The values in the file below is
obviously just an example of how it might look like if you've got the files on a Mac within your "Documents" folder.
The "tenant" and the "cloudlet" parts are resolved by doing a string split operation on your magic:io:root-files
root folder upon the last (/) found in your folder - Implying if you use the Docker images with the default configuration
the "tenant" part would become "magic" and the "cloudlet" part would become "files", since the Docker images
stores files within the "/magic/files/" folder.
This project is created to be a multi-tenant solution, implying multiple users, and/or cloudlets, can use the same physical
database for both files, log entries, and cache entries. However, this implies you are able to correctly resolve the
"tenant" parts and the "cloudlet" parts for all the above tables. How to achieve this is currently beyond the scope
of this document, but our suggestion is to create some sort of IRootResolver service implementation that is dynamically
created according to the Host HTTP header of requests.
Project website
The source code for this repository can be found at github.com/polterguy/magic.data.cql, and you can provide feedback, provide bug reports, etc at the same place.
