Skip to content

Use tablesaw in data processors

Problem to solve

As discussed with @stsc and @BZoennchen

We noticed that writing out csv data (at the end of a simulation) can be very slow in Vadere (depending on how much data is written). The main reason seems to be the design of how the csv data is written. In short, this involves collecting each row in a LinkedList and casting it to a csv-line). Currently, each DataProcessor fills a Map with objects (key and value).

We decided:

  • replace the Map and fill a tablesaw's data frames (tables)
  • each table consists of index columns and data columns, all columns have a native data format (usually int or double)
  • the DataKey (e.g. PedestrianIdKey) classes are still used and required (to be able to merge processors, in the GUI, and for information about data format of indices)
  • The value (generics V in DataProcessors) are removed and handled by the table's columns
  • the merging of DataProcessor and writing (merged) tables to csv is still handled in OutputWriter
To upload designs, you'll need to enable LFS and have an admin enable hashed storage. More information