Drive characteristics using strym

In this notebook, we will be calling the class version of the strym.meta function, which will produce a json formatted data sample for a particular drive. As that class evolves, so will this tester. The tester will have different options to call for specific files, folders, etc.

World’s smallest test of strym.meta

Import required package, then call on a specific file and print the output json

from strym import meta

# test how this will work when calling from within another
meta1 = meta(csvfile='../../PandaData/2020_05_11/2020-05-11-22-49-54_2T3Y1RFV8KC014025_CAN_Messages.csv',
             dbcfile=['../examples/newToyotacode.dbc'])
meta1.toJSON()

/home/ivory/anaconda3/envs/dbn/lib/python3.7/site-packages/statsmodels/tools/_testing.py:19: FutureWarning: pandas.util.testing is deprecated. Use the functions in the public API at pandas.testing instead.
  import pandas.util.testing as tm

Loading BokehJS ...

Reading ../../PandaData/2020_05_11/2020-05-11-22-49-54_2T3Y1RFV8KC014025_CAN_Messages.csv

{'filepath': '../../PandaData/2020_05_11/2020-05-11-22-49-54_2T3Y1RFV8KC014025_CAN_Messages.csv',
 'filename': '2020-05-11-22-49-54_2T3Y1RFV8KC014025_CAN_Messages.csv',
 'vin': '2T3Y1RFV8KC014025',
 'make': 'Toyota Canada',
 'date': 'Mon May 11 22:49:55 2020',
 'distance_km': 0.2208434659759204,
 'distance_miles': 0.13692294890507065,
 'duration_s': 40.231223821640015}

Specify Data Location

parentfolder = "../../PandaData/2020_05_11"
import glob
folderlist = glob.glob(parentfolder+"*")
csvlist = []
for datafolder in folderlist:
#     datafolder = "/Users/sprinkle/work/data/cyverse/rahulbhadani/JmscslgroupData/PandaData/2020_03_03/"
    import glob
    csvlisttmp = glob.glob(datafolder+"/*CAN_Messages.csv")
    for f in csvlisttmp:
        csvlist.append(f)

num_of_files = len(csvlist)
print("Total number of datafiles in {} is {}.".format(datafolder, num_of_files))

Total number of datafiles in ../../PandaData/2020_05_11 is 1.

Analysis

1. CSV file containing all messages

In this section, we will extract a subset of the CSV-formatted CAN Data in order to produce summary information of a specific drive.

dbcfile = ['../examples/newToyotacode.dbc']
metadata = []
metaobjs = []
for i in range(num_of_files):
    try:
        meta_i = meta(csvfile=csvlist[i], dbcfile=dbcfile)
        drive = meta_i.toJSON()
        metadata.append( drive )
        metaobjs.append( meta_i )
    except:
        print('Unable to process file...')
print(metadata)

Reading ../../PandaData/2020_05_11/2020-05-11-22-49-54_2T3Y1RFV8KC014025_CAN_Messages.csv
[{'filepath': '../../PandaData/2020_05_11/2020-05-11-22-49-54_2T3Y1RFV8KC014025_CAN_Messages.csv', 'filename': '2020-05-11-22-49-54_2T3Y1RFV8KC014025_CAN_Messages.csv', 'vin': '2T3Y1RFV8KC014025', 'make': 'Toyota Canada', 'date': 'Mon May 11 22:49:55 2020', 'distance_km': 0.2208434659759204, 'distance_miles': 0.13692294890507065, 'duration_s': 40.231223821640015}]

Get the total distance

dist=0
for d in metadata:
    dist = dist + d['distance_miles']
print(dist)

0.13692294890507065

metadata

[{'filepath': '../../PandaData/2020_05_11/2020-05-11-22-49-54_2T3Y1RFV8KC014025_CAN_Messages.csv',
  'filename': '2020-05-11-22-49-54_2T3Y1RFV8KC014025_CAN_Messages.csv',
  'vin': '2T3Y1RFV8KC014025',
  'make': 'Toyota Canada',
  'date': 'Mon May 11 22:49:55 2020',
  'distance_km': 0.2208434659759204,
  'distance_miles': 0.13692294890507065,
  'duration_s': 40.231223821640015}]

## Dump the Data file
for meta in metaobjs:
    filename = meta.write()
    print(f'Writing to file {filename}')

../../PandaData/2020_05_11/2020-05-11-22-49-54_2T3Y1RFV8KC014025__metadata.json
Writing to file ../../PandaData/2020_05_11/2020-05-11-22-49-54_2T3Y1RFV8KC014025__metadata.json

Reading in JSON files as a Dashboard

# Change this to your folder to gather all data that exists and generate the dashboard
parentfolder = "../../PandaData/"

from strym import dashboard
try:
    db = dashboard(directory=parentfolder,verbose=False)
    print(db.statistics())
    print(f'Total driving distance (miles): {db.miles()} ({db.error_count} files not parsed)')
    print(f'Total driving distance (km): {db.kilometers()} ({db.error_count} files not parsed)')

except Exception as ex:
    print(f'Exception when processing {directory} (msg={ex})')


# import strym.dashboard as dashboard

# # find all the JSON files in this directory
# parentfolder = "/Users/sprinkle/work/data/cyverse/rahulbhadani/JmscslgroupData/PandaData/"
# import glob
# folderlist = glob.glob(parentfolder+"*")
# print(folderlist)
# jsonlist = []
# for datafolder in folderlist:
# #     datafolder = "/Users/sprinkle/work/data/cyverse/rahulbhadani/JmscslgroupData/PandaData/2020_03_03/"
#     import glob
#     jsonlisttmp = glob.glob(datafolder+"/*.json")
#     print(jsonlisttmp)
#     if len(jsonlisttmp) > 0:
#         for f in jsonlisttmp:
#             jsonlist.append(f)

# print(jsonlist)
# metadata_dict = []

# for json_file_str in jsonlist:
#     try:
#         with open(json_file_str) as json_file:
#             data = json.load(json_file)
#             metadata_dict.append(data)
#     except Exception as ex:
#         print(f'Skipping {json_file_str}, continuing (ex={ex})')

# dist=0
# for d in metadata_dict:
#     try:
#         dist = dist + d['distance_miles']
#     except Exception as ex:
#         print(f'No key distance_miles in dictionary, skipping')
# print(dist)

Skipping ../../PandaData/2020_01_30  - Sprinkles drive/2020-01-27-15-22-21-800723__CAN_Message_Rav4.json, continuing (ex=Expecting value: line 1 column 1 (char 0))
Metadata entries: 628
JSON files found: 629

Total driving distance (miles): 6113.417438535157 (80 files not parsed)
Total driving distance (km): 9860.350707314767 (80 files not parsed)