Skip to content

Crawler updates 2022

Rajat Arunachala Chandavar requested to merge Crawler_updates_2022 into master

Edits were made to the mainly Post-processing aspects of the crawler.

Changes made:

  1. In Parsers.py, the following edit was required because we need to preserve the commas separating array variables in the string. This is necessary for json.loads() to work properly. In python, converting to numpy array (using asarray in this case) leads to removal of commas. Hence i converted it to a list and then string so that commas are preserved.

image

  1. In Postprocessors.py, for each class there are 2 arguments - arg and args. arg is what we get from the hdf5 file and args is what we provide in the arguments column in output from multiplexer. I made edits in code such that when argument is provided explicitly by user, operation is done on args. Or else it is done on arg. Also in Postprocessors.py, for return object of each class, typecasting to float was necessary as for some reason json doesn't let us dump int64.

  2. Additional minor changes in test to match with changes made above.

Merge request reports