Training LLMs for Data-Driven Insight Generation

Client: Etsy Date: Dec 2023 Services: Tech Integration, Data Analysis

This client’s unique budgeting and goaling structure necessitated a nuanced approach to analyzing ad spend efficiency. The conventional process of generating insights was labor-intensive, involving extensive time spent sifting through multiple reporting dashboards. The GPT Assistant, leveraging advanced LLM techniques, successfully automated this process, significantly reducing the time and cost while maintaining the quality of deliverables.

BACKGROUND

The Client’s distinctive media budgeting strategy—unlimited ad spend contingent on meeting ROI targets for each channel and market—posed a unique challenge in generating precise and actionable insights. The client team required detailed analysis focusing on week-over-week revenue shifts. The conventional method involved manual analysis of multiple Tableau reporting dashboards, consuming over an hour twice weekly.

OBJECTIVE

This project aimed to develop an automated solution to generate executive insights reports efficiently and accurately. The goal was to maintain the quality of the insight reports (tailored to the client’s preferences and program strategy) while significantly reducing the time and resources required to produce them.

SOLUTION

The “Exec Insights GPT Assistant” was developed to automate the analysis of digital advertising performance data. This extensively trained assistant utilizes user-submitted raw reporting data (last 30 days) and current ROI targets to generate insights. Through meticulous GPT research and iterative training, a process was established to optimally format data, leverage necessary Python libraries, and create effective natural language instructions. The assistant successfully reduced the report generation time to 4 minutes per composition, costing under $0.30 per run, marking a 90% efficiency improvement.

Operational Methodology

  1. Data Preprocessing
    1. Import necessary libraries: pandas (as pd), numpy (as np), calendar.
    2. Load data: Read CSV file into a pandas DataFrame.
    3. Format and clean data:
      1. Automatic date format detection and setting ‘Date’ as index.
      2. Removal of commas, special characters; conversion to appropriate data types.
      3. Float type conversion for numerical columns.
      4. NaN value imputation with zeros.
      5. Standardization of ‘Channel’ and ‘Market’ columns for consistency.
  2. Data Aggregation and Pivot Table Creation
    1. Create pivot tables for analysis: OVERALL, CHANNEL, and MARKET views.
    2. Aggregate data by summing up key metrics (Impressions, Clicks, Orders, Revenue, Spend).
  3. Rate Metrics Calculation
    1. Define functions for calculating AOV, CPC, CTR, CVR, ROI.
    2. Apply these functions to each pivot table.
  4. Comparative Analysis
    1. Identify time periods for comparison: ‘Primary Period’ (last seven days) and ‘Comparison Period’ (preceding seven days).
    2. Filter and prepare data sets for both periods.
    3. Calculate percentage change for each metric between the two periods.
  5. Output Synthesis
    1. Construct narrative insights based on calculated data.
    2. Format output:
      1. OVERALL level analysis aggregating all channels and markets.
      2. CHANNEL level analysis per channel, aggregating across all markets.
      3. Performance Extrapolation Forecasting: Estimation of current month’s performance.
      4. Extra Credit Section: Highlighting significant findings and trends.
  6. Insights Generation
    1. Generate insights reflecting significant shifts in KPIs (Key Performance Indicators).
    2. Articulate the relationship between primary KPI shifts and changes in peripheral KPIs.
    3. Provide context and explanations for observed changes.

CONCLUSION

Creating the Exec Insights GPT Assistant for this particular client account represents a significant leap in the efficiency and effectiveness of data analysis in digital advertising. This custom GPT tool not only streamlines the process of generating insights but also opens avenues for further exploration in the field of LLM-driven data analysis for other clients. This project demonstrates the immense potential of AI in transforming complex data analysis tasks into efficient, automated processes. All while still tailored to client preferences and the strategy underlying our paid media programs.

APPENDICES / REFERENCES