Pigweed Tokenizer.pdf

1、#EMBEDDEDOSSUMMITPigweed TokenizerAl Semjonovs,GoogleEmail:Discord:asemjonovs#8232Github:asemjonovsAgendaWhat is Pigweeds tokenizer?Zephyrs Dictionary Logging ComparisonHow does tokenizing help us?How to integrate Pigweeds tokenization into your projectFuture improvements with tokenizationWhat is Pi

2、gweeds tokenizer?Not related to string parsing Replaces whole string literal with a 32-bit hash token.Why?Reduce binary size by removing string literals from binariesReduce I/O traffic,RAM,and flash usage.Reduce CPU usage by replacing snprintf calls with simple tokenization code.Zephyr Dictionary Lo

3、ggingPigweed TokenizerString mappingMaps to string addressMaps to 32-bit hash generated by string literalProbability of CollisionsStrings have a 1:1 mapping to address.Guaranteed no collisions.Database FormatJSON dictionaryCSV,Binary,Directory basedDatabase PortabilityAddresses are not guaranteed be

4、tween builds.Only works with build it was compiled against.The token hash will be the same between builds and boards using same source.Able to merge tokens from multiple boards and versions to a single database.Token DatabaseStores mapping of hash tokens to strings they representGenerated from ELF f

5、ileDatabase Format TypesCSVBinaryDirectory BasedUpdate an existing databaseCaptures removal date of unused tokensIntegrates with CMake and GN buildsDetokenizingLanguage supportPythonC/C+TypescriptWeb Console System Consolepigweed$python3-m pw_system.console-device/dev/ttyUSB0-token-databases databas

6、e.csvTokenized Logging ExampleBefore:Plain text loggingLocationLogging ContentNum bytesSource containsLOG(Battery state:%s;battery voltage:%d mV,state,voltage);Binary containsBattery state:%s;battery voltage:%d mV41Device TransmitsBattery state:CHARGING;battery voltage:3989 mV49W