Skip to content

Commit 9fc3cba

Browse files
kaustubh-khavnekar-quantiphikaustubh-khavnekar-quantiphi
authored
Merge branch 'elephant-xyz:main' into main
2 parents 6543a31 + 4a23b47 commit 9fc3cba

17 files changed

Lines changed: 2203 additions & 541 deletions

‎.gitignore‎

Lines changed: 5 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -369,3 +369,8 @@ elephant-cli*.log
369369
prepare/layers/
370370
workflow/lambdas/post/transforms.zip
371371

372+
set_alachua_envs.sh
373+
set_sarasota_envs.sh
374+
set_dixie_envs.sh
375+
set_multi_counties_envs.sh
376+
unset_envs.sh

‎README.md‎

Lines changed: 118 additions & 22 deletions
Original file line numberDiff line numberDiff line change
@@ -34,6 +34,9 @@ export ELEPHANT_PREPARE_USE_BROWSER=false # Force browser mode
3434
export ELEPHANT_PREPARE_NO_FAST=false # Disable fast mode
3535
export ELEPHANT_PREPARE_NO_CONTINUE=false # Disable continue mode
3636

37+
# Optional (Continue button selector)
38+
export ELEPHANT_PREPARE_CONTINUE_BUTTON="" # CSS selector for continue button
39+
3740
# Optional (Browser flow template - both must be provided together)
3841
export ELEPHANT_PREPARE_BROWSER_FLOW_TEMPLATE="" # Browser flow template name
3942
export ELEPHANT_PREPARE_BROWSER_FLOW_PARAMETERS="" # Browser flow parameters as JSON string
@@ -43,6 +46,9 @@ export UPDATER_SCHEDULE_RATE="1 minute" # How often updater runs (default: "1
4346
# For sub-minute intervals, use cron expressions:
4447
# export UPDATER_SCHEDULE_RATE="cron(*/1 * * * ? *)" # Every minute
4548
# export UPDATER_SCHEDULE_RATE="cron(0/30 * * * ? *)" # Every 30 seconds (at :00 and :30)
49+
50+
# Optional (Proxy rotation - for automatic proxy rotation support)
51+
export PROXY_FILE=/path/to/proxies.txt # File containing proxy URLs (one per line: username:password@ip:port)
4652
```
4753

4854
#### Option B: Keystore Mode (using encrypted private key)
@@ -73,6 +79,9 @@ export ELEPHANT_PREPARE_USE_BROWSER=false # Force browser mode
7379
export ELEPHANT_PREPARE_NO_FAST=false # Disable fast mode
7480
export ELEPHANT_PREPARE_NO_CONTINUE=false # Disable continue mode
7581

82+
# Optional (Continue button selector)
83+
export ELEPHANT_PREPARE_CONTINUE_BUTTON="" # CSS selector for continue button
84+
7685
# Optional (Browser flow template - both must be provided together)
7786
export ELEPHANT_PREPARE_BROWSER_FLOW_TEMPLATE="" # Browser flow template name
7887
export ELEPHANT_PREPARE_BROWSER_FLOW_PARAMETERS="" # Browser flow parameters as JSON string
@@ -100,12 +109,13 @@ This creates the VPC, S3 buckets, SQS queues, Lambdas, and the Express Step Func
100109

101110
The `DownloaderFunction` uses the `prepare` command from `@elephant-xyz/cli` to fetch and process data. You can control its behavior using environment variables that map to CLI flags:
102111

103-
| Environment Variable | Default | CLI Flag | Description |
104-
| ------------------------------ | ------------ | --------------- | ------------------------------------------------------------- |
105-
| `ELEPHANT_PREPARE_USE_BROWSER` | `false` | `--use-browser` | Force browser mode for fetching |
106-
| `ELEPHANT_PREPARE_NO_FAST` | `false` | `--no-fast` | Disable fast mode |
107-
| `ELEPHANT_PREPARE_NO_CONTINUE` | `false` | `--no-continue` | Disable continue mode |
108-
| `UPDATER_SCHEDULE_RATE` | `"1 minute"` | N/A | Updater frequency (e.g., "5 minutes", "cron(_/1 _ \* _ ? _)") |
112+
| Environment Variable | Default | CLI Flag | Description |
113+
| ---------------------------------- | ------------ | --------------- | ------------------------------------------------------------- |
114+
| `ELEPHANT_PREPARE_USE_BROWSER` | `false` | `--use-browser` | Force browser mode for fetching |
115+
| `ELEPHANT_PREPARE_NO_FAST` | `false` | `--no-fast` | Disable fast mode |
116+
| `ELEPHANT_PREPARE_NO_CONTINUE` | `false` | `--no-continue` | Disable continue mode |
117+
| `ELEPHANT_PREPARE_CONTINUE_BUTTON` | `""` | N/A | CSS selector for continue button |
118+
| `UPDATER_SCHEDULE_RATE` | `"1 minute"` | N/A | Updater frequency (e.g., "5 minutes", "cron(_/1 _ \* _ ? _)") |
109119

110120
#### County-Specific Configuration
111121

@@ -130,8 +140,9 @@ export ELEPHANT_PREPARE_NO_CONTINUE=false
130140

131141
# Alachua County - needs browser mode with specific selectors
132142
export ELEPHANT_PREPARE_USE_BROWSER_Alachua=true
143+
export ELEPHANT_PREPARE_CONTINUE_BUTTON_Alachua=".btn.btn-primary.button-1"
133144
export ELEPHANT_PREPARE_BROWSER_FLOW_TEMPLATE_Alachua="SEARCH_BY_PARCEL_ID"
134-
export ELEPHANT_PREPARE_BROWSER_FLOW_PARAMETERS_Alachua='{"continue_button_selector": ".btn.btn-primary.button-1", "search_form_selector": "#ctlBodyPane_ctl03_ctl01_txtParcelID", "search_result_selector": "#ctlBodyPane_ctl10_ctl01_lstBuildings_ctl00_dynamicBuildingDataRightColumn_divSummary"}'
145+
export ELEPHANT_PREPARE_BROWSER_FLOW_PARAMETERS_Alachua='{"search_form_selector": "#ctlBodyPane_ctl03_ctl01_txtParcelID", "search_result_selector": "#ctlBodyPane_ctl10_ctl01_lstBuildings_ctl00_dynamicBuildingDataRightColumn_divSummary"}'
135146

136147
# Sarasota County - different template and slower processing
137148
export ELEPHANT_PREPARE_USE_BROWSER_Sarasota=true
@@ -142,6 +153,15 @@ export ELEPHANT_PREPARE_BROWSER_FLOW_PARAMETERS_Sarasota='{"timeout": 60000, "se
142153
# Charlotte County - simple browser mode, no template needed
143154
export ELEPHANT_PREPARE_USE_BROWSER_Charlotte=true
144155

156+
# Santa Rosa County - note the underscore for the space
157+
export ELEPHANT_PREPARE_USE_BROWSER_Santa_Rosa=true
158+
export ELEPHANT_PREPARE_BROWSER_FLOW_TEMPLATE_Santa_Rosa="SANTA_ROSA_FLOW"
159+
export ELEPHANT_PREPARE_BROWSER_FLOW_PARAMETERS_Santa_Rosa='{"timeout": 45000}'
160+
161+
# Palm Beach County - another example with space in name
162+
export ELEPHANT_PREPARE_USE_BROWSER_Palm_Beach=true
163+
export ELEPHANT_PREPARE_NO_FAST_Palm_Beach=true
164+
145165
# Broward County - uses general settings (no browser mode)
146166
# No county-specific settings needed
147167

@@ -154,6 +174,8 @@ When the Lambda processes data:
154174
- Alachua inputs → Uses browser mode with SEARCH_BY_PARCEL_ID template
155175
- Sarasota inputs → Uses browser mode with CUSTOM_SEARCH template and no-fast mode
156176
- Charlotte inputs → Uses simple browser mode
177+
- Santa Rosa inputs → Uses browser mode with SANTA_ROSA_FLOW template (county with space in name)
178+
- Palm Beach inputs → Uses browser mode with no-fast flag (county with space in name)
157179
- Broward inputs → Uses general settings (no browser mode)
158180
- Any other county → Uses general settings
159181

@@ -175,10 +197,19 @@ export ELEPHANT_PREPARE_BROWSER_FLOW_TEMPLATE_Charlotte="CHARLOTTE_FLOW"
175197
- `ELEPHANT_PREPARE_USE_BROWSER_<CountyName>`
176198
- `ELEPHANT_PREPARE_NO_FAST_<CountyName>`
177199
- `ELEPHANT_PREPARE_NO_CONTINUE_<CountyName>`
200+
- `ELEPHANT_PREPARE_CONTINUE_BUTTON_<CountyName>`
178201
- `ELEPHANT_PREPARE_BROWSER_FLOW_TEMPLATE_<CountyName>`
179202
- `ELEPHANT_PREPARE_BROWSER_FLOW_PARAMETERS_<CountyName>`
180203

181-
**Note:** County names in environment variables should match exactly as they appear in `county_jurisdiction` field (case-sensitive). For example, if the JSON contains `"county_jurisdiction": "Alachua"`, use `_Alachua` suffix.
204+
**Important naming convention for counties with spaces:**
205+
206+
For counties with spaces in their names, replace spaces with underscores in the environment variable name:
207+
208+
- `"Santa Rosa"` → `ELEPHANT_PREPARE_USE_BROWSER_Santa_Rosa`
209+
- `"Palm Beach"` → `ELEPHANT_PREPARE_BROWSER_FLOW_TEMPLATE_Palm_Beach`
210+
- `"San Diego"` → `ELEPHANT_PREPARE_NO_FAST_San_Diego`
211+
212+
The Lambda automatically handles this conversion when matching county names from the data.
182213

183214
#### Browser Flow Template Configuration
184215

@@ -194,9 +225,10 @@ For advanced browser automation scenarios, you can provide custom browser flow t
194225
**General configuration example:**
195226

196227
```bash
197-
# Set browser flow template configuration for all counties
228+
# Set browser flow template configuration and continue button selector for all counties
229+
export ELEPHANT_PREPARE_CONTINUE_BUTTON=".btn.btn-primary.button-1"
198230
export ELEPHANT_PREPARE_BROWSER_FLOW_TEMPLATE="SEARCH_BY_PARCEL_ID"
199-
export ELEPHANT_PREPARE_BROWSER_FLOW_PARAMETERS='{"continue_button_selector": ".btn.btn-primary.button-1", "search_form_selector": "#ctlBodyPane_ctl03_ctl01_txtParcelID", "search_result_selector": "#ctlBodyPane_ctl10_ctl01_lstBuildings_ctl00_dynamicBuildingDataRightColumn_divSummary"}'
231+
export ELEPHANT_PREPARE_BROWSER_FLOW_PARAMETERS='{"search_form_selector": "#ctlBodyPane_ctl03_ctl01_txtParcelID", "search_result_selector": "#ctlBodyPane_ctl10_ctl01_lstBuildings_ctl00_dynamicBuildingDataRightColumn_divSummary"}'
200232

201233
# Deploy with the configuration
202234
./scripts/deploy-infra.sh
@@ -205,12 +237,14 @@ export ELEPHANT_PREPARE_BROWSER_FLOW_PARAMETERS='{"continue_button_selector": ".
205237
**County-specific browser flow example:**
206238

207239
```bash
208-
# Different counties may need different browser flow templates and selectors
240+
# Different counties may need different browser flow templates, continue button selectors, and parameters
241+
export ELEPHANT_PREPARE_CONTINUE_BUTTON_Alachua=".btn.btn-primary.button-1"
209242
export ELEPHANT_PREPARE_BROWSER_FLOW_TEMPLATE_Alachua="SEARCH_BY_PARCEL_ID"
210-
export ELEPHANT_PREPARE_BROWSER_FLOW_PARAMETERS_Alachua='{"continue_button_selector": ".btn.btn-primary.button-1", "search_form_selector": "#ctlBodyPane_ctl03_ctl01_txtParcelID", "search_result_selector": "#ctlBodyPane_ctl10_ctl01_lstBuildings_ctl00_dynamicBuildingDataRightColumn_divSummary"}'
243+
export ELEPHANT_PREPARE_BROWSER_FLOW_PARAMETERS_Alachua='{"search_form_selector": "#ctlBodyPane_ctl03_ctl01_txtParcelID", "search_result_selector": "#ctlBodyPane_ctl10_ctl01_lstBuildings_ctl00_dynamicBuildingDataRightColumn_divSummary"}'
211244

245+
export ELEPHANT_PREPARE_CONTINUE_BUTTON_Sarasota="#submit"
212246
export ELEPHANT_PREPARE_BROWSER_FLOW_TEMPLATE_Sarasota="CUSTOM_SEARCH"
213-
export ELEPHANT_PREPARE_BROWSER_FLOW_PARAMETERS_Sarasota='{"timeout": 60000, "selector": "#search-box", "submit_button": "#submit"}'
247+
export ELEPHANT_PREPARE_BROWSER_FLOW_PARAMETERS_Sarasota='{"timeout": 60000, "selector": "#search-box"}'
214248

215249
# Deploy - each county will use its specific configuration
216250
./scripts/deploy-infra.sh
@@ -247,24 +281,22 @@ The Lambda logs will show exactly which configuration is being used for each cou
247281

248282
```
249283
📍 Extracting county information from input...
250-
✅ Detected county: Alachua
284+
✅ Detected county: Santa Rosa
251285
Building prepare options...
252286
Event browser setting: undefined (using: true)
253287
Checking environment variables for prepare flags:
254-
🏛️ Looking for county-specific configurations for: Alachua
255-
Using county-specific: ELEPHANT_PREPARE_USE_BROWSER_Alachua='true'
288+
🏛️ Looking for county-specific configurations for: Santa Rosa
289+
Using county-specific: ELEPHANT_PREPARE_USE_BROWSER_Santa_Rosa='true'
256290
✓ Setting useBrowser: true (Force browser mode)
257291
Using general: ELEPHANT_PREPARE_NO_FAST='false'
258292
✗ Not setting noFast flag (Disable fast mode)
259-
Using county-specific: ELEPHANT_PREPARE_BROWSER_FLOW_TEMPLATE_Alachua='SEARCH_BY_PARCEL_ID'
293+
Using county-specific: ELEPHANT_PREPARE_BROWSER_FLOW_TEMPLATE_Santa_Rosa='SANTA_ROSA_FLOW'
260294
Browser flow template configuration detected:
261-
✓ ELEPHANT_PREPARE_BROWSER_FLOW_TEMPLATE='SEARCH_BY_PARCEL_ID'
262-
Using county-specific: ELEPHANT_PREPARE_BROWSER_FLOW_PARAMETERS_Alachua='continue_button_selector:.btn.btn-primary.button-1,search_form_selector:#ctlBodyPane_ctl03_ctl01_txtParcelID,search_result_selector:#ctlBodyPane_ctl10_ctl01_lstBuildings_ctl00_dynamicBuildingDataRightColumn_divSummary'
295+
✓ ELEPHANT_PREPARE_BROWSER_FLOW_TEMPLATE='SANTA_ROSA_FLOW'
296+
Using county-specific: ELEPHANT_PREPARE_BROWSER_FLOW_PARAMETERS_Santa_Rosa='timeout:45000'
263297
✓ ELEPHANT_PREPARE_BROWSER_FLOW_PARAMETERS parsed successfully:
264298
{
265-
"continue_button_selector": ".btn.btn-primary.button-1",
266-
"search_form_selector": "#ctlBodyPane_ctl03_ctl01_txtParcelID",
267-
"search_result_selector": "#ctlBodyPane_ctl10_ctl01_lstBuildings_ctl00_dynamicBuildingDataRightColumn_divSummary"
299+
"timeout": 45000
268300
}
269301
Calling prepare() with these options...
270302
```
@@ -446,6 +478,70 @@ Options:
446478

447479
This query tool uses CloudWatch Logs Insights to efficiently analyze large volumes of log data and provides actionable metrics for monitoring post-processing performance across different counties.
448480

481+
## Proxy Rotation
482+
483+
The system supports automatic proxy rotation for the prepare function. This helps distribute load across multiple proxies and prevents rate limiting or IP blocking.
484+
485+
### Setup Proxies
486+
487+
**1. Create a proxy file:**
488+
489+
Create a text file with one proxy per line in format: `username:password@ip:port`
490+
491+
```bash
492+
# Create proxies.txt
493+
cat > proxies.txt <<EOF
494+
user1:password123@192.168.1.100:8080
495+
user2:password456@192.168.1.101:8080
496+
user3:password789@192.168.1.102:8080
497+
EOF
498+
```
499+
500+
You can use `proxies.example.txt` as a template.
501+
502+
**2. Deploy with proxies:**
503+
504+
```bash
505+
# Initial deployment with proxies
506+
export PROXY_FILE=proxies.txt
507+
./scripts/deploy-infra.sh
508+
```
509+
510+
**3. Update proxies later (without full deployment):**
511+
512+
```bash
513+
# Update or add proxies anytime
514+
./scripts/update-proxies.sh proxies.txt
515+
```
516+
517+
### How It Works
518+
519+
- Each Lambda invocation automatically selects the **least recently used** proxy
520+
- Proxies are rotated automatically based on usage timestamps
521+
- Failed proxies are tracked but remain available for retry
522+
- No configuration needed - just provide the proxy file
523+
524+
### Verify Proxies
525+
526+
Check which proxies are configured:
527+
528+
```bash
529+
./scripts/update-proxies.sh --list
530+
```
531+
532+
### Remove All Proxies
533+
534+
```bash
535+
./scripts/update-proxies.sh --clear
536+
```
537+
538+
### Notes
539+
540+
- Empty lines and lines starting with `#` are ignored (comments)
541+
- Proxy format must be: `username:password@ip:port`
542+
- Port must be numeric
543+
- If no proxies are configured, the system works normally without them
544+
449545
That's it — set env vars, deploy, start, monitor, and tune concurrency.
450546

451547
## Dead Letter Queue (DLQ) Management

0 commit comments

Comments
 (0)