dancefinder/PROJECTIONS.md

# Projections

This document describes the projection system used to maintain read models from domain events.

## Overview

The DanceFinder application uses CQRS (Command Query Responsibility Segregation) with event sourcing. Domain events are stored in the event store, and projections update read model tables that are optimized for queries.

## Architecture

```
Domain Events → Event Store → Projection Handlers → Read Model Tables
                    ↓
                AMQP/RabbitMQ
```

1. **Commands** update aggregates and produce domain events
2. **Events** are stored in the event store
3. **Events** are published to AMQP
4. **Projection handlers** consume events and update read model tables
5. **GraphQL queries** read from the optimized read model tables

## Projections

### Band Projection

**Projection Name**: `band`

**Handles Events**:
- `BandCreated` - Creates a new band in the read model

**Updates Table**: `band`

**Reset Handler**: Deletes all bands from the read model

### DanceHall Projection

**Projection Name**: `dancehall`

**Handles Events**:
- `DanceHallCreated` - Creates a new dance hall in the read model
- `DanceHallGeocoded` - Updates dance hall coordinates

**Updates Table**: `dance_hall`

**Reset Handler**: Deletes all dance halls from the read model

### Event Projection

**Projection Name**: `event`

**Handles Events**:
- `EventCreated` - Creates a new event in the read model
- `EventUpdated` - Updates event details
- `EventDeleted` - Removes an event from the read model

**Updates Table**: `event`

**Reset Handler**: Deletes all events from the read model

### UserPreferences Projection

**Projection Name**: `userpreferences`

**Handles Events**:
- `OriginAdded` / `OriginRemoved` - Manage user's home locations
- `BandIgnored` / `BandUnignored` - Manage ignored bands
- `DanceHallIgnored` / `DanceHallUnignored` - Manage ignored venues
- `CityIgnored` / `CityUnignored` - Manage ignored cities
- `MunicipalityIgnored` / `MunicipalityUnignored` - Manage ignored municipalities
- `StateIgnored` / `StateUnignored` - Manage ignored states

**Updates Tables**:
- `origin`
- `ignored_band`
- `ignored_dance_hall`
- `ignored_city`
- `ignored_municipality`
- `ignored_state`

**Reset Handler**: Deletes all user preference data from all related tables

## Projection State Management

Each projection maintains its position in the event stream using the `readviews` table:

```sql
CREATE TABLE readviews (
    name TEXT PRIMARY KEY,
    position BIGINT NOT NULL
);
```

The position tracks the last processed event timestamp, ensuring projections:
- Process events exactly once
- Can resume from their last position after restart
- Don't miss events

## Rebuilding Projections

Projections can be rebuilt from scratch using the reset functionality. This is useful when:
- Schema changes require rebuilding the read model
- Data corruption in read models needs to be fixed
- Projection logic changes and historical data needs reprocessing

### Using Reset Handlers

Reset handlers are automatically called by the `readview` framework when a projection needs to be rebuilt.

**Manual Reset** (requires application code):

```go
// Reset a specific projection
err := projectionManager.ResetProjection(ctx, "band")

// This will:
// 1. Call resetBand() which truncates the band table (with CASCADE)
// 2. Reset the projection position to 0
// 3. Replay all BandCreated events from the event store
// 4. Update the band table with current data
```

**What happens during reset**:

1. **Reset handler executes** in a transaction:
   - Truncates projection tables (fast and efficient)
   - For `band`: `TRUNCATE TABLE band CASCADE`
   - For `dancehall`: `TRUNCATE TABLE dance_hall CASCADE`
   - For `event`: `TRUNCATE TABLE event`
   - For `userpreferences`: Truncates all 6 related tables

2. **Position is reset** to 0

3. **Events are replayed**:
   - All relevant events are read from the event store
   - Events are processed in order
   - Read model is rebuilt from scratch

4. **Projection catches up** to current state

### TRUNCATE vs DELETE

The reset handlers use `TRUNCATE TABLE CASCADE` instead of `DELETE` for better performance:

**Advantages**:
- Much faster for clearing entire tables
- Reclaims disk space immediately
- Less transaction log overhead
- More efficient for large tables

**CASCADE Usage**:
- Handles any remaining foreign key dependencies
- Required even though we remove FK constraints in migrations
- Some constraint names may be auto-generated and differ from expected names
- Ensures reset succeeds even if FK constraints remain

### No Foreign Key Constraints

**Important**: Read view tables have **no foreign key constraints** between them.

**Rationale**:
- Event store is the single source of truth
- Read models are just projections that can be rebuilt
- Foreign keys add unnecessary coupling between projections
- Simplifies independent projection resets
- No CASCADE side effects
- Better flexibility for projection evolution

**What this means**:
- Each projection can be reset independently without affecting others
- No need for CASCADE on TRUNCATE operations
- Reset order doesn't matter (though you may want to reset in a specific order for logical reasons)
- Referential integrity is maintained at the write side (domain aggregates), not read side

### Reset Safety

Reset operations are:
- **Transactional**: If reset fails, no data is lost
- **Coordinated**: The readview framework manages the process
- **Automatic**: Once initiated, the framework handles replay
- **Independent**: No foreign key constraints means no side effects on other projections
- **Safe**: Event store is the source of truth, read models can always be rebuilt

### Reset Scenarios

**Scenario 1: Reset Single Projection**
```go
// Reset any single projection independently
// No side effects, no CASCADE, completely isolated
manager.ResetProjection(ctx, "band")
// Only bands are cleared and rebuilt
// Events, dance halls, and user prefs are unaffected
```

**Scenario 2: Reset Multiple Related Projections**
```go
// If you have dangling references (e.g., events referencing non-existent bands),
// reset the dependent projections too
manager.ResetProjection(ctx, "band")      // Reset bands
manager.ResetProjection(ctx, "event")     // Reset events (they reference bands)

// This ensures referential consistency in your read models
```

**Scenario 3: Reset All Projections**
```go
// Complete rebuild of all read models
// Order doesn't matter, but logical order is clearer:
manager.ResetProjection(ctx, "band")
manager.ResetProjection(ctx, "dancehall")
manager.ResetProjection(ctx, "event")
manager.ResetProjection(ctx, "userpreferences")

// All projections are rebuilt from their respective events
```

**Scenario 4: Selective Reset for Debugging**
```go
// Reset only what you need to test/fix
manager.ResetProjection(ctx, "event")
// Test event projection logic without affecting other projections
```

## Event Handler Patterns

### Insert with Conflict Resolution

Most handlers use `ON CONFLICT` to handle idempotency:

```go
query := `
    INSERT INTO band (id, name, created)
    VALUES ($1, $2, $3)
    ON CONFLICT (name) DO UPDATE
    SET id = EXCLUDED.id,
        created = EXCLUDED.created
`
```

This ensures:
- Events can be replayed safely
- Duplicate event processing doesn't cause errors
- Latest event data wins in case of conflicts

### Update by ID

Update handlers use the aggregate ID to find records:

```go
query := `
    UPDATE dance_hall
    SET latitude = $1, longitude = $2
    WHERE id = $3
`
```

### Delete Operations

Delete handlers remove specific records:

```go
query := `DELETE FROM event WHERE id = $1`
```

### Foreign Key Lookups

Some handlers need to look up related entities:

```go
// Find band ID by name
var bandID string
err := tx.QueryRowContext(ctx, `SELECT id FROM band WHERE name = $1`, event.BandName).Scan(&bandID)

// Use ID in ignored_band table
query := `INSERT INTO ignored_band (band_id, subject, created) VALUES ($1, $2, $3)`
```

## Monitoring Projections

### Check Projection Status

Query the readviews table to see projection positions:

```sql
SELECT name, position,
       (SELECT MAX(tstamp) FROM events) as latest_event
FROM readviews;
```

### Check Lag

Compare projection position with latest event:

```sql
SELECT
    rv.name,
    rv.position as projection_position,
    MAX(e.tstamp) as latest_event_time,
    MAX(e.tstamp) - rv.position as lag_seconds
FROM readviews rv
CROSS JOIN events e
GROUP BY rv.name, rv.position;
```

### Verify Data Consistency

Count records in both event store and read models:

```sql
-- Count events in event store by type
SELECT name, COUNT(*)
FROM events
WHERE aggregate_name = 'domain.Band'
GROUP BY name;

-- Count bands in read model
SELECT COUNT(*) FROM band;
```

## Error Handling

Projection errors are logged and the projection manager will:
1. Log the error with event details
2. Stop processing for that projection
3. Preserve the position (won't advance past the error)
4. Allow other projections to continue

To recover from projection errors:
1. Fix the underlying issue (schema, logic, etc.)
2. Restart the service
3. The projection will resume from its last position
4. If the error persists, reset the projection

## Performance Considerations

### Indexes

All read model tables have appropriate indexes:
- Primary keys on `id` columns (UUID)
- Foreign key indexes
- Unique constraints where needed
- Additional indexes for common queries

### Batch Processing

Projections process events one at a time in transactions:
- Ensures consistency
- Prevents partial updates
- Allows rollback on errors

### Concurrent Processing

Different projections run independently:
- `band`, `dancehall`, `event`, and `userpreferences` projections can process concurrently
- Each maintains its own position
- No coordination required between projections

## Testing Projections

### Unit Testing Event Handlers

```go
func TestBandCreatedHandler(t *testing.T) {
    // Create test database and transaction
    db := setupTestDB(t)
    tx, _ := db.Begin()
    defer tx.Rollback()

    // Create test event
    event := &domain.BandCreated{
        BaseEvent: eventsourced.BaseEvent{
            EventAggregateId: "test-uuid",
            EventTime:        time.Now(),
        },
        Name: "Test Band",
    }

    // Execute handler
    err := handleBandCreated(context.Background(), tx, event)
    require.NoError(t, err)

    // Verify result
    var count int
    tx.QueryRow("SELECT COUNT(*) FROM band WHERE name = $1", "Test Band").Scan(&count)
    assert.Equal(t, 1, count)
}
```

### Integration Testing

```go
func TestProjectionReplay(t *testing.T) {
    // Store events in event store
    // Reset projection
    // Verify read model matches expected state
}
```

## Best Practices

1. **Idempotent Handlers**: Always use `ON CONFLICT` or check for existence
2. **Transactional Updates**: All projection updates in a single transaction
3. **Error Handling**: Return meaningful errors for debugging
4. **Schema Alignment**: Keep projection tables aligned with domain model
5. **Foreign Keys**: Use proper foreign key constraints for data integrity
6. **Reset Testing**: Test reset handlers regularly
7. **Monitor Lag**: Watch for projections falling behind
8. **Index Optimization**: Add indexes for common query patterns

## Troubleshooting

### Projection is Stuck

**Symptoms**: Position not advancing, data not updating

**Solutions**:
1. Check logs for errors
2. Verify event store contains events
3. Check for database locks
4. Restart the service

### Projection is Behind

**Symptoms**: High lag, old data in read models

**Solutions**:
1. Check for slow queries in handlers
2. Add missing indexes
3. Optimize handler logic
4. Consider increasing resources

### Data Inconsistency

**Symptoms**: Read model doesn't match event store

**Solutions**:
1. Reset the affected projection
2. Verify handler logic
3. Check for missing event handlers
4. Review ON CONFLICT logic

### Foreign Key Errors

**Symptoms**: Handler fails with foreign key violation

**Solutions**:
1. Ensure dependent records exist
2. Check projection order (bands/halls before events)
3. Verify reset order
4. Add missing event handlers