Trading bots should store both raw TweetStream payloads and normalized event fields so alerts are auditable, replayable, and idempotent.
Storage layers
Trading bots use storage to debug fills, replay missed events, prove which signal the bot saw, and run analytics.
- Append-only raw event log
- Normalized tweet/account/token tables
- Short replay window for dashboard reconnects
- Idempotency keys for event processing
Retention
Keep raw payload retention long enough to debug strategy behavior, then derive compressed analytics tables for longer-term research.
Questions
Should I store raw or normalized tweets?
Store raw payloads for audit/debugging and normalized fields for queries, alerts, replay, and strategy evaluation.
How do I avoid duplicates?
Use stable event IDs or tweet IDs to make processing idempotent across reconnects and retries.