Skip to end of metadata
Go to start of metadata

STSdb 4.0 supports a wide set of user types for the keys and for the records. By default the engine generates all the needed logic. But in some cases the user needs to provide his own custom logic, for example: how to compare and store the keys, how to store the records. This can be done by providing custom comparers and custom persist implementations via the table descriptor property.

To understand how to write custom table logic we have to know how the engine builds the default table logic. For example, let us consider in details what is happening when the user opens a table with key type long and (our well known) record type Tick:

public class Tick
{
   public string Symbol { get; set; }
   public DateTime Timestamp { get; set; }
   public double Bid { get; set; }
   public double Ask { get; set; }
   public int BidSize { get; set; }
   public int AskSize { get; set; }
   public string Provider { get; set; }
}
    ITable<long, Tick> table = engine.OpenXTable<long, Tick>("table");

When the user opens (creates) a direct user table, based on the provided user types, the following things happen

  1. DataType descriptions are generated for the key type and for the record type. In our case for the types long and Tick  the generated descriptions look like this:

        DataType.Int64;
    
        DataType.Slots(DataType.String, DataType.DateTime, DataType.Double, DataType.Double, DataType.Double, DataType.Double);
    

    These values will be permanently stored respectively in KeyDataType and RecordDataType properties of the table descriptor. (They cannot be changed during the entire table life.)

  2. Comparer and equality comparer logic is generated for the key type:

        public int Compare(IData var1, IData var2)
        {
            long value1 = ((Data<long>)var1).Value;
            long value2 = ((Data<long>)var2).Value;
            if (value1 < value2)
                return -1;
            else if (value1 > value2)
                return 1;
            else
                return 0;
        }
    

    The code is invoked from a created IComparer<IData> instance, automatically assigned to the KeyComparer table descriptor property.

        public bool Equals(IData var1, IData var2)
        {
            long var3 = ((Data<long>)var1).Value;
            long var4 = ((Data<long>)var2).Value;
            return var3 == var4;
        }
        public int GetHashCode(IData var1)
        {
            long var2 = ((Data<long>)var1).Value;
            return (int)var2 ^ (int)var2 >> 32;
        }
    

    The code is invoked from a created IEqualityComparer<IData> instance, automatically assigned to the KeyEqualityComparer table descriptor property.

  3. Persist logic is generated for the keys and for the records:
    For type long:

        public void Write(BinaryWriter writer, IData data)
        {
            long dataValue = ((Data<long>)data).Value;
            writer.Write(dataValue);
        }
        public IData Read(BinaryReader reader)
        {
            return new Data<long>(reader.ReadInt64());
        }
    

    The code is invoked from a created IPersist<IData> instance, automatically assigned to the KeyPersist table descriptor property
    For type Tick:

        public void Write(BinaryWriter writer, IData idata)
        {
            Tick dataValue = ((Data<Tick>)idata).Value;
            if (dataValue.Symbol != null)
            {
                writer.Write(true);
                writer.Write(dataValue.Symbol);
            }
            else
                writer.Write(false);
            writer.Write(dataValue.Timestamp.Ticks);
            writer.Write(dataValue.Bid);
            writer.Write(dataValue.Ask);
            writer.Write(dataValue.BidSize);
            writer.Write(dataValue.AskSize);
            if (dataValue.Provider != null)
            {
                writer.Write(true);
                writer.Write(dataValue.Provider);
            }
            else
                writer.Write(false);
        }
        public IData Read(BinaryReader reader)
        {
            var var1 = new Tick();
            var1.Symbol = reader.ReadBoolean() ? reader.ReadString() : null;
            var1.Timestamp = new DateTime(reader.ReadInt64());
            var1.Bid = reader.ReadDouble();
            var1.Ask = reader.ReadDouble();
            var1.BidSize = reader.ReadInt32();
            var1.AskSize = reader.ReadInt32();
            var1.Symbol = reader.ReadBoolean() ? reader.ReadString() : null;
            return new Data<Tick>(var1);
        }
    

    The code is invoked from a created IPersist<IData> instance, automatically assigned to the RecordPersist table descriptor property.

  4. IndexerPersist logic is generated in case that key type and record type are linear:
    For type long:

        public void Store(BinaryWriter writer, Func<int, IData> values, int count)
        {
            using (MemoryStream ms = new MemoryStream())
            {
                ((Int64IndexerPersist)persists[0]).Store(new BinaryWriter(ms), (idx) => ((Data<long>)values.Invoke(idx)).Value, count);
                CountCompression.Serialize(writer, (ulong)ms.Length);
                writer.Write(ms.GetBuffer(), 0, (int)ms.Length);
            }
        }
        public void Load(BinaryReader reader, Action<int, IData> func, int count)
        {
            byte[] var1 = reader.ReadBytes((int)CountCompression.Deserialize(reader));
            using (MemoryStream var2 = new MemoryStream(var1))
            {
                ((Int64IndexerPersist)persists[0]).Load(new BinaryReader(var2), (idx, value) => { func.Invoke(idx, new Data<long>(value)); }, count);
            }
        }
    

    For type Tick:

        public void Store(BinaryWriter writer, Func<int, IData> values, int count)
        {
            Action[] actions = new Action[7];
            MemoryStream[] streams = new MemoryStream[7];
            actions[0] = () =>
            {
                streams[0] = new MemoryStream();
                ((StringIndexerPersist)persists[0]).Store(new BinaryWriter(streams[0]), (idx) => ((Data<Tick>)values.Invoke(idx)).Value.Symbol, count);
            };
            actions[1] = () =>
            {
                streams[1] = new MemoryStream();
                ((DateTimeIndexerPersist)persists[1]).Store(new BinaryWriter(streams[1]), (idx) => ((Data<Tick>)values.Invoke(idx)).Value.Timestamp, count);
            };
            actions[2] = () =>
            {
                streams[2] = new MemoryStream();
                ((DoubleIndexerPersist)persists[2]).Store(new BinaryWriter(streams[2]), (idx) => ((Data<Tick>)values.Invoke(idx)).Value.Ask, count);
            };
            actions[3] = () =>
            {
                streams[3] = new MemoryStream();
                ((DoubleIndexerPersist)persists[3]).Store(new BinaryWriter(streams[3]), (idx) => ((Data<Tick>)values.Invoke(idx)).Value.Bid, count);
            };
            actions[4] = () =>
            {
                streams[4] = new MemoryStream();
                ((Int32IndexerPersist)persists[4]).Store(new BinaryWriter(streams[4]), (idx) => ((Data<Tick>)values.Invoke(idx)).Value.BidSize, count);
            };
            actions[5] = () =>
            {
                streams[5] = new MemoryStream();
                ((Int32IndexerPersist)persists[5]).Store(new BinaryWriter(streams[5]), (idx) => ((Data<Tick>)values.Invoke(idx)).Value.AskSize, count);
            };
            actions[6] = () =>
            {
                streams[6] = new MemoryStream();
                ((StringIndexerPersist)persists[6]).Store(new BinaryWriter(streams[6]), (idx) => ((Data<Tick>)values.Invoke(idx)).Value.Provider, count);
            };
            Parallel.Invoke(actions);        
        }
            for (int i = 0; i < actions.Length; i++)
            {
                var stream = streams[i];
                using (stream)
                {
                    CountCompression.Serialize(writer, (ulong)stream.Length);
                    writer.Write(stream.GetBuffer(), 0, (int)stream.Length);
                }
            }
        }
    
    
    
        public void Load(BinaryReader reader, Action<int, IData> values, int count)
        {
            Data<Tick>[] array = new Data<Tick>[count];
            for (int i = 0; i < count; i++)
            {
                var item = new Data<Tick>();
                item.Value = new Tick();
                array[i] = item;
                values(i, item);
            }
            Action[] actions = new Action[7];
            byte[][] buffers = new byte[7][];
            for (int i = 0; i < 6; i++)
                buffers[i] = reader.ReadBytes((int)CountCompression.Deserialize(reader));
            actions[0] = () =>
            {
                using (MemoryStream ms = new MemoryStream(buffers[0]))
                    ((IIndexerPersist<String>)persists[0]).Load(new BinaryReader(ms), (idx, value) => { ((Data<Tick>)array[idx]).Value.Symbol = value; }, count);
            };
            actions[1] = () =>
            {
                using (MemoryStream ms = new MemoryStream(buffers[1]))
                    ((IIndexerPersist<DateTime>)persists[1]).Load(new BinaryReader(ms), (idx, value) => { ((Data<Tick>)array[idx]).Value.Timestamp = value; }, count);
            };
            actions[2] = () =>
            {
                using (MemoryStream ms = new MemoryStream(buffers[2]))
                    ((IIndexerPersist<Double>)persists[2]).Load(new BinaryReader(ms), (idx, value) => { ((Data<Tick>)array[idx]).Value.Bid = value; }, count);
            };
            actions[3] = () =>
            {
                using (MemoryStream ms = new MemoryStream(buffers[3]))
                    ((IIndexerPersist<Double>)persists[3]).Load(new BinaryReader(ms), (idx, value) => { ((Data<Tick>)array[idx]).Value.Ask = value; }, count);
            };
            actions[4] = () =>
            {
                using (MemoryStream ms = new MemoryStream(buffers[4]))
                    ((IIndexerPersist<int>)persists[4]).Load(new BinaryReader(ms), (idx, value) => { ((Data<Tick>)array[idx]).Value.BidSize = value; }, count);
            };
            actions[5] = () =>
            {
                using (MemoryStream ms = new MemoryStream(buffers[5]))
                    ((IIndexerPersist<int>)persists[5]).Load(new BinaryReader(ms), (idx, value) => { ((Data<Tick>)array[idx]).Value.AskSize = value; }, count);
            };
            actions[6] = () =>
            {
                using (MemoryStream ms = new MemoryStream(buffers[6]))
                    ((IIndexerPersist<String>)persists[6]).Load(new BinaryReader(ms), (idx, value) => { ((Data<Tick>)array[idx]).Value.Provider = value; }, count);
            };
            Parallel.Invoke(actions);
        }
    

    The persists instance in the code is a IIndexerPersist[] array containing already created compression engines for each of the primitive Tick members. Every IIndexerPersist implementation can compress and decompress a sequence of values from that type (for example: StringIndexerPersist – for string values, Int32IndexerPersist – for int values, etc).

    As we can see, the generated code strictly follows the concrete properties in the key and record types. The engine generated the code using vertical compressions running in tasks. The default generated compression code reduces the database size and increases the performance.

    NOTE: Parallel vertical compressions are used by the engine for all table records founded in the WaterfallTree leafs (for the internal nodes the engine uses the generated single record persist). Thus, we practically compress only the heaviest nodes in the tree - the leaf nodes.

    If the key type and the record type are linear types the engine can generate such methods. These methods are invoked from the created IIndexerPersist<IData> instances, automatically assigned respectively to KeyIndexerPersist and RecordIndexerPersist table descriptor properties. For non-linear types the engine disables the vertical compressions (KeyIndexerPersist and RecordIndexerPersist properties are null), because the needed expressions code becomes too complicated for generation and the net effect would be questionable.

    All these 4 steps are automatically performed when a table is opened.  We shall refer to all the generated code - for the keys and for the records – compare, persist etc. as “table environment code” or just “table logic”.

    Once the table logic is generated, every further work with this table or with a table with the same key and record types will not cause the engine to generate it again. While the engine is working it keeps all of the generated code in caches. And when the database is closed and reopened lately again the engine resolves the table types by their full names and gradually rebuilds this code cache.

     

Default Transformers    Go to Home    Custom comparer and persist logic

  • No labels