Skip to content

Store

The store listens to data changes. In essence IMB is a memory bases system and only changes in the data are communicated. To make the system survive a restart the store listens to data changes for all collections, key/value sets and matrices. These changes are stored and during idle time compressed to remove duplicates i.e. only the last values per object are stored. On startup the last, compressed, version of the data can be inquired as the current state. All data is stored in ProtoBuf format in flat, binary files. The format is the same as would be streamed to clients, only split in chunks to not block the network when transmitting.

The exchange of data is done in objects and collections of objects in the form of changes on the current state of the data. The objects and collections are synchronized over the clients. The store does not play an active part in this and only records this exchange of data changes. Collections are linked together in bins. Projects and scenario's are organized through those bins. Attributes of objects can have a time component to make them a time sequence of values.

  • Bin: a bin is a folder in the data hierarchy that combines a set of collections, key/value sets and/or OD matrices
  • Base bin: the base bin is a special version of a bin in respect to it's name. A bin that represents a scenario is normally the project name with a dot and then the scenario name like V1. The base bin is the same as the project name, without a scenario post-fix. It contains all non-scenario based data; data that is the same for the whole of the project.
  • Bin group or project: a bin group or project is the set of bins, including the base bin, that all start with the name of the project. All bins are separate but can visually be represented as a group
  • Collection: a collection is a set of objects. Each object is identified by a GUID/UUID. The fields of the objects are stored and communicated in Protocol Buffers format and can be address separately.
  • Key/value set: A key/value set is a set of named elements (key) with a string (value). The string value very often contains JSON.
  • OD Matrix: an OD matrix is a specialized data container where all fields consist of an array of values. The size of the fields are all the same within a matrix. The matrix itself contains 2 indices for the x and y direction that translate the index in the x and y direction to specific values; the x and y values are the same for for all fields within that matrix. The OD matrix is not implemented as a collection for simplicity and speed reasons. Values within a field can be updated as a range; because mostly updates of fields in a matrix address the whole array this is much more efficient then addressing an object in a collection per cell.

The organization of the stored date is done in directories named after the bins. There are 3 sub directories for the data:

  • collections
  • keyvaluesets
  • matrices

All data is stored in binary files that contain the same data format that is used for transmitting them over the IMB network. Collections temporary store the data changes in a file called 'deltas' that are later compressed and merged with the previous state to a new singular state of all the objects and properties in the collection. This is the state that is returned on an inquire by connected clients when they startup.