Disassembling DBMSs

Significant software systems are typically built around a database / data store / DBMS of some sort.

Using a test platform to test a comple system requires the same features as a DBMS but here we use gDS as the data handling component. If we are going to do this let's go over the features involved.

A must-have feature. You can declare and manipulate tables in SQL or, in C++, Java, assembler. Tables, and manipulating data in tables have been with us since the beginning.

The data in the various tables can be manipulated by one execution thread or multiple execution threads. Managing multiple threads is called managing concurrent access - see "Transactions" below.

Let's start with "Third-normal form":

    https://en.wikipedia.org/wiki/Third_normal_form

I have a hard time with the rigorous definition of 3NF but I know it when I see it. Better yet I know it when a collection of tables isn't in 3NF; it's cumbersome to work with the data in the tables. The code is confusing, it doesn't "work".

But if you move some columns around and maybe make a couple of new tables, operations start to become easier, cleaner, neater, simpler. This is a *huge* property!

It is helpful to split data manipulation code into "procedural" and "non-procedural" methodologies:

• Procedural code - Most mainstream languages are procedural; they access and manipulate data using "if" and "loop" operations. Procedural code can manipulate data in tables.

• Non-procedural code - SQL is one of a number of non-procedural code alternative. There are no "if" and "loop" operations supported on the data in the tables. Instead data manipulation operations are declarative statements made from a small collection of primitives (INSERT, UPDATE, DELETE, SELECT).

!!!!! Writing a complex system / application in a mixture of procedural and non-procedural code is a major effort requiring close technical management and skill. !!!!!

To appreciate the above really requires trying to do the above.

So, pick a method:

• Procedural code is familiar. A device driver uses "if"  and "loop" statements on data in tables declared in C just fine.

• I have created (one) complex system / application just using non-procedural SQL code: No procedural code was used. It was strange at first but a total eye-opener. Total. But this is not the topic under discussion in this website.

Transactions are an effort to centralize the features (Google "ACID") needed to manage concurrent access to data in tables. Transactions are invoked by calling the top of a stack of sub-features. Transactions are a challenge because it's tough to reach into the call stack when you need to do something different / complex.

An alternative to the traditional call stack is to use system locks to provide the necessary protections for concurrent access.

Given how table organization affects the development effort I believe it should be under control of the entire development staff. There are no "small fixes" or "minor updates".

Given the ability of "hyper-converged" systems to make a copy of a complete virtual machine / computer system in a minute or so, developers can then be free to get their designs going in a "sandbox" environment.

Rolling out changes to a production system then becomes, rightly, a challenge.

gDS allows Python users to create tables in global shared memory and to spawn the threads and processes needed to get the (testing) job accomplished using those shared tables. All operations are carried out using normal Python code.

There is no SQL therefore no mixing of procedural and non-procedural code. The management of concurrent access uses system locks..

I believe Python is singular in it's ability to support something like gDS and it's use in testing complex systems.