Store tables and mappers for 2.0

This page describes in detail changes needed in debea core implementations to support features mentioned in #40.

How current implementation works

All debea backends (SQLArchive, CSVArchive, XMLArchive) operate on dba::Storeable class. This class provides access to store table definitions for all child classes that derive from dba::Storeable.

Single store table contains following serialization information about class:

  • name of relation (could be NULL)
  • offset correction in bytes that express difference between object referenced as Storeable* pointer.
  • link to store table of parent class.

For every non-collection member in store table following serialization information is stored:

  • offset in bytes from object to member
  • name of serialized member
  • type of serialized member (one of Database::STRING, Database::DATE, Database::INTEGER, Database::FLOAT)
  • instance of serialization filter

For collection member following serialization information is stored:

  • offset in bytes from object to member
  • name of serialized member
  • name of foreign key (used by XMLArchive as parent element of list)
  • collection identifier (used by SQLArchive to differentiate many collections of the same objects in single class)
  • name of root relation for objects from collections.
  • instance of serialization filter

How serialization works

Archive backend needs class instance to load data from database. It gets list of store tables from that instance, constructs SQL Query that will load all member data and sends that query to database. When data is loaded, dba::StoreableFilter is used to convert data from database type to application data type in place. dba::StoreableFilter gets pointer to object member using object pointer and offset to member from store table entry for that member.

If collections need to be loaded, then after root object is completely constructed, stream examines store table members that binds collections and construct queries and load collections data recursively.

dba::Storeable class handles object id, object database state (NEW, CHANGED, DELETED), and provides access to all store tables for object.

Root relation name concept

Name of relation is set for every store table in class inheritance tree. If any class in this tree has set relation name to NULL, then it 'derives' relation name from child class. Last child in inheritance tree have to have non NULL relation name. It can be set in two ways:

  • in store table as parameter of BEGIN_STORE_TABLE macro
  • just before object is serialized or deserialized, in Stream::setRootTable()
  • in BIND_CLA macro to alter or set name of SQL table where objects from collections are serialized.

Serialization filter concept

There are two types of 'filters': Serialization filter is a class that can convert data from application type to database type. Collection filter can map database relation to application class model.

Both filter types can be user-defined, which allow library user to store complex types in database fields or add support for different container classes without changing dba internals. Example of custom serialization filter is here

Problems with current design

  • For CSVArchive and XMLArchive id of object and state is useless. Those file formats are often used for data exchange and there is no support in debea for updating already existing data in csv or xml files.
  • Meta information about serialization is spread in different places and its meaning is interpreted by backed in different ways. STR in BIND_STR defines database type, CSVArchive has API for mapping column names or column number to store table member, table member name in BIND_ macros is not used for CSVArchive and XMLArchive at all. Foreign key in BIND_CLA maps to column name in SQL table or to parent element of list name in XML file.
  • Model classes from application needs to derive from dba::Storeable. This forces to mix part of database tier with model tier and makes model classes to depend on used serialization library.
  • SQL Database tables have to define id field and make it primary key. Sometimes it is not required by application or primary key uses different name or is constructed from multiple fields.
  • User have to choose between BIND_STR/INT/DATE/FLOAT macros, but this can be deducted from conversion filter used for field. Situations where application int type is stored as varchar type in database are rare.
  • There is no way to set default filter for given application type. For dba, std::string should use dba::String by default, for wxdba wxString should use wxdba::String.
  • Getter and setter methods often have side effects - i.e. in addition to setting serialized member A other member B is updated. When debea loads object from database and fill member data directly setter and getter code is not run. Omitting this make loaded object unusable.

Proposed solution

  1. Move store tables to external mappers.
  2. Make Storeable class and object state tracking optional.
  3. Aggregate backend-related properties to one place in mappers.
  4. Add "optimal database type" to Storeable filter to automatically minimize amount of data conversions (done in 1.4)
  5. Map getter and setter methods instead of class members directly
  6. Do not use preprocessor macros if possible.

Milestones

  • Create and test external mappers
  • Make old Storeable based code compatible with new mappers
  • Implement Mapper attach to Archive and mapper selection upon serialization using new ORMSession class
  • Add aggregation support for SQL backend using property like sql.store_key=1