Extension Patterns: Advanced Automation

When Extension Patterns Matter

Basic scripts work for simple tasks. Extension patterns become essential when:

• Performance bottlenecks emerge - Scripts take minutes instead of seconds
• Complex workflows develop - Multi-step processes with dependencies
• API costs increase - Redundant calls drain budgets
• Reliability becomes critical - Need graceful error handling
• Scale demands optimization - Processing hundreds of files or requests

Extension patterns transform single-purpose scripts into production-grade automation systems.

Pattern 1: Script Chaining

The Concept

Script chaining connects multiple specialized scripts into complex workflows. Each script does one thing well, then passes results to the next stage.

Basic Sequential Chaining

# 3-stage pipeline
./download.sh | ./process.sh | ./analyze.sh > results.txt

Each script:

• Reads from stdin (previous stage output)
• Writes to stdout (next stage input)
• Handles errors independently

Conditional Chaining

# Success-dependent execution
./download.sh && ./process.sh && ./report.sh

# Error fallback
./primary.sh || ./backup.sh || ./emergency.sh

Operators explained:

• && - Run next only if previous succeeds (exit code 0)
• || - Run next only if previous fails (non-zero exit code)
• | - Always pass output regardless of success

Domain-Specific Applications

# Download → Extract → Synthesize
./arxiv-search.sh "platform economics" | \
  ./extract-citations.sh | \
  ./generate-bibliography.sh > literature.bib

# Test → Build → Deploy (only if tests pass)
./run-tests.sh && ./build-docker.sh && ./deploy-staging.sh

# Scrape → Clean → Analyze → Visualize
./scrape-competitors.sh | \
  ./clean-data.sh | \
  ./analyze-trends.sh | \
  ./generate-report.sh

Pattern 2: Parallel Execution

The Concept

Parallel execution runs independent scripts simultaneously, reducing total workflow time from sum of durations to maximum of individual durations.

Basic Parallel Pattern

# Launch 3 scripts in background, wait for all
./task1.sh & ./task2.sh & ./task3.sh &
wait

Syntax breakdown:

• & - Run script in background (returns control immediately)
• wait - Block until all background jobs complete

Real-World Performance Gains

Example: Multi-source data collection

When to Use Parallel Execution

Domain-Specific Parallel Examples

Economics Research:

# Download papers from 3 databases simultaneously
./arxiv-fetch.sh & ./jstor-fetch.sh & ./ssrn-fetch.sh &
wait

Software Testing:

# Run test suites in parallel
./unit-tests.sh & ./integration-tests.sh & ./e2e-tests.sh &
wait

Business Monitoring:

# Monitor competitors concurrently
./track-competitor-a.sh & ./track-competitor-b.sh & ./track-competitor-c.sh &
wait

Pattern 3: Caching for Speed

The Concept

Caching stores expensive operation results—API calls, computations, file downloads—to avoid redundant work. First run pays full cost, subsequent runs return instantly.

Simple Cache Pattern

# Check cache first, fetch only if missing
[ -f cache.json ] && cat cache.json && exit
curl "$API_ENDPOINT" | tee cache.json

Pattern breakdown:

• [ -f cache.json ] - Test if cache file exists
• cat cache.json && exit - Return cached data if found
• tee cache.json - Save API response to cache while outputting

Cache Invalidation Strategies

# Delete cache older than 1 hour
find cache.json -mmin +60 -delete 2>/dev/null
curl "$API_ENDPOINT" | tee cache.json

# Recompute if input file is newer than cache
[ cache.json -nt input.csv ] || ./expensive-analysis.sh < input.csv > cache.json
cat cache.json

# Support --refresh flag
[ "$1" = "--refresh" ] && rm -f cache.json
[ -f cache.json ] && cat cache.json && exit
curl "$API_ENDPOINT" | tee cache.json

Caching Benefits

Speed comparison:

Cost reduction:

• Development: 100 test runs × $0.01 per API call = $1.00 without cache, $0.01 with cache
• Production: 1000 daily requests → 100 unique requests cached = 90% cost reduction

Offline capability:

Cached data enables scripts to function without network connectivity, critical for:

• Development on unstable connections
• Demo environments without API access
• Disaster recovery scenarios

Combining Patterns

Advanced Research Pipeline

Real-world workflows benefit from multiple patterns working together:

# Fetch from 3 sources simultaneously, cache each
./econ-papers.sh & ./cs-papers.sh & ./stats-papers.sh &
wait

# Deduplicate → Extract → Analyze → Report
cat *.json | ./deduplicate.sh | ./extract-data.sh | ./analyze.sh | ./report.sh

Pattern Selection Guide

Advanced Considerations

Error Handling in Chains

Stop on first error:

set -e  # Exit immediately if any command fails
./stage1.sh | ./stage2.sh | ./stage3.sh

Fallback behavior:

# Try primary, fall back to secondary
./primary.sh || ./secondary.sh || echo "All methods failed"

Resource Management in Parallel

Limit concurrent processes:

• Prevent system overload by capping parallel executions
• Use GNU parallel for sophisticated job control
• Monitor CPU, memory, and network bandwidth

Respect API rate limits:

• Add delays between parallel requests
• Implement token bucket algorithms
• Queue requests to stay within limits

Cache Strategy Evolution

Time-to-Live (TTL) expiration:

• Set appropriate cache lifetimes based on data volatility
• Stock prices: 1 minute
• Weather forecasts: 1 hour
• Research papers: 1 week

Least Recently Used (LRU) eviction:

• Limit cache size to prevent disk exhaustion
• Remove oldest unused entries first
• Balance between hit rate and storage

Versioning for data changes:

• Include version identifiers in cache keys
• Invalidate automatically when schemas change
• Support migration between cache versions

Next Steps

Extension patterns transform basic scripts into production automation:

• Start simple: Add one pattern to existing scripts
• Measure impact: Compare performance before and after
• Iterate gradually: Combine patterns as complexity grows
• Monitor resources: Watch for bottlenecks and optimize

The next chapter covers troubleshooting common script failures and debugging techniques.