Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
34 commits
Select commit Hold shift + click to select a range
375fdbd
adding token dispenser to get harmony client
DavidVWoodSr Apr 15, 2026
6474a30
pull back 1 rev to test develop base in CICD
DavidVWoodSr Apr 15, 2026
2290c22
add current region to boto3.client call
DavidVWoodSr Apr 15, 2026
50ba51c
add definition for region
DavidVWoodSr Apr 15, 2026
1d0f862
add additional region definitions
DavidVWoodSr Apr 15, 2026
4c56094
adding mock for boto3.client tests
DavidVWoodSr Apr 15, 2026
4936481
adjusting mock for boto3.client
DavidVWoodSr Apr 15, 2026
923eb3c
adjusting mock for boto3.client
DavidVWoodSr Apr 15, 2026
ba6696b
addinging boto mock for testing
DavidVWoodSr Apr 16, 2026
901e069
remove cassette
DavidVWoodSr Apr 16, 2026
c732c72
changing boto mocking for tests
DavidVWoodSr Apr 16, 2026
796de8b
working on UAT mock
DavidVWoodSr Apr 16, 2026
0822c64
working on UAT remove boto mock
DavidVWoodSr Apr 16, 2026
c2d2f80
working on UAT remove boto mock
DavidVWoodSr Apr 16, 2026
442d752
changing lambda return to json
DavidVWoodSr Apr 16, 2026
67084c0
adding access-token as lambda return
DavidVWoodSr Apr 16, 2026
79d1f94
change mock for no_data
DavidVWoodSr Apr 16, 2026
bae6133
move imports
DavidVWoodSr Apr 16, 2026
5aee785
add missing import
DavidVWoodSr Apr 16, 2026
0c53125
copied uat 2 prod,sit tests
DavidVWoodSr Apr 16, 2026
31f8512
working with diff mock usage
DavidVWoodSr Apr 16, 2026
453b6c2
working with diff mock usage-2
DavidVWoodSr Apr 16, 2026
218e97f
working with diff mock usage-3
DavidVWoodSr Apr 16, 2026
b1f6d84
pushing region to os
DavidVWoodSr Apr 27, 2026
da1c8ff
fixing payload for lambda response
DavidVWoodSr Apr 27, 2026
6b3512d
proper lambda response mocked
DavidVWoodSr Apr 27, 2026
402bad5
changing all lambda mock response to the same values
DavidVWoodSr Apr 27, 2026
dd6a6aa
removing doule json.loads
DavidVWoodSr Apr 27, 2026
d36268e
working out getting access-token
DavidVWoodSr Apr 27, 2026
728068d
working out getting access-token
DavidVWoodSr Apr 27, 2026
956d55a
json debugging
DavidVWoodSr Apr 27, 2026
3c99c1e
sync up fake token generation
DavidVWoodSr Apr 27, 2026
1e1b267
fixed harmony client tests
DavidVWoodSr May 7, 2026
98c9a8b
fixed test merge conflicts
DavidVWoodSr May 20, 2026
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
43 changes: 42 additions & 1 deletion bignbit/utils.py
Original file line number Diff line number Diff line change
Expand Up @@ -4,6 +4,7 @@
import os
import pathlib
import re
import base64
from datetime import datetime, timedelta

from typing import Any
Expand Down Expand Up @@ -345,9 +346,49 @@ def get_harmony_client(environment_str: str) -> Client:
HARMONY_CLIENT = None

if not HARMONY_CLIENT:
# Create a boto3 Lambda client for token retrieval
session = boto3.session.Session()
region = (
os.environ.get("AWS_REGION") or
os.environ.get("AWS_DEFAULT_REGION") or
session.region_name or
"us-west-2"
)

lambda_client = boto3.client('lambda', region_name=region)

# Retrieve EDL credentials (username/password)
edl_user, edl_pass = get_edl_creds()

# Prepare payload to request an access token from the Lambda "token-dispenser"
payload = {
"action": "edl",
"edl_user": edl_user,
"edl_pass": edl_pass,
"edl_env": environment_str,
"minimum_alive_secs": 300 # keep token valid for at least 5 minutes
}
encoded_payload = base64.b64encode(
json.dumps(payload).encode("utf-8")
).decode("utf-8")

# Invoke the Lambda synchronously and get the token
response = lambda_client.invoke(
FunctionName='sndbx-launchpad_token_dispenser',

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is this always the name for the token dispenser lambda? The sndbx designation seems suspicious. Should this be configured in terraform?

InvocationType='RequestResponse', # wait for response
Payload=encoded_payload
)

# Read the payload from Lambda response and parse JSON
response_payload = json.loads(response["Payload"].read())
print(type(response_payload))
print(response_payload)
token = response_payload["access-token"]

# Instantiate Harmony client with retrieved token
HARMONY_CLIENT = Client(
env=harmony_environ,
auth=get_edl_creds(),
token=token,
should_validate_auth=HARMONY_SHOULD_VALIDATE_AUTH
)

Expand Down
57 changes: 57 additions & 0 deletions cassettes/test_handle_big_result/test_process_harmony_results.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,57 @@
interactions:
- request:
body: null
headers:
Accept:
- '*/*'
Accept-Encoding:
- gzip, deflate
Connection:
- keep-alive
User-Agent:
- python-requests/2.32.5
method: GET
uri: https://harmony.uat.earthdata.nasa.gov/jobs/3d276f84-56e2-4f0a-acb2-35b9fcaaa317?linktype=https
response:
body:
string: '{"code":"harmony.UnauthorizedError","description":"Error: You are not
authorized to access the requested resource"}'
headers:
Access-Control-Allow-Origin:
- '*'
Connection:
- keep-alive
Content-Length:
- '115'
Content-Type:
- application/json; charset=utf-8
Date:
- Wed, 29 Apr 2026 15:50:03 GMT
ETag:
- W/"73-nItWkl68x6IlPn6tx7bg/IcouGY"
Server:
- CloudFront
Strict-Transport-Security:
- max-age=31536000; includeSubDomains; preload
Via:
- 1.1 22922394e3534f85930c1603d51f2156.cloudfront.net (CloudFront)
X-Amz-Cf-Id:
- 8h3Stmve4aRgESrEJpwVOaf5co7Gx6FbA5eRgrpChi2FoepoNyHhPg==
X-Amz-Cf-Pop:
- SFO53-P10
X-Cache:
- Error from cloudfront
X-Content-Type-Options:
- nosniff
X-Frame-Options:
- SAMEORIGIN
X-Powered-By:
- Express
X-Request-Id:
- 8h3Stmve4aRgESrEJpwVOaf5co7Gx6FbA5eRgrpChi2FoepoNyHhPg==
X-XSS-Protection:
- 1; mode=block
status:
code: 401
message: Unauthorized
version: 1
267 changes: 267 additions & 0 deletions cassettes/test_process_harmony_results.yaml

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is there a reason why the two cassettes in the cassettes directory are not in tests/cassettes? You should move them into the tests directory

Original file line number Diff line number Diff line change
@@ -0,0 +1,267 @@
interactions:
- request:
body: <CreateBucketConfiguration xmlns="http://s3.amazonaws.com/doc/2006-03-01/"><LocationConstraint>us-west-2</LocationConstraint></CreateBucketConfiguration>
headers:
Content-Length:
- '153'
User-Agent:
- !!binary |
Qm90bzMvMS4yNi4xNjUgUHl0aG9uLzMuMTAuMTMgRGFyd2luLzI0LjYuMCBCb3RvY29yZS8xLjI5
LjE2NQ==
X-Amz-Content-SHA256:
- !!binary |
YTRhM2QyNjMxNzliZmNiYWY4ODc1NWIxNGVhOWRiMzA0OGU0MTc5YzAzYmJjOTRjY2YwZjEzYzhj
MmQzZGY2YQ==
X-Amz-Date:
- !!binary |
MjAyNjA0MjhUMTIxNzI0Wg==
amz-sdk-invocation-id:
- !!binary |
MjlhZTA0ODctMGE3NC00YzViLWJmNDctNzc1ZDQ4NTA2NDVj
amz-sdk-request:
- !!binary |
YXR0ZW1wdD0x
method: PUT
uri: https://harmony-uat-staging.s3.us-west-2.amazonaws.com/
response:
body:
string: '<?xml version="1.0" encoding="UTF-8"?>

<Error><Code>BucketAlreadyExists</Code><Message>The requested bucket name
is not available. The bucket namespace is shared by all users of the system.
Please select a different name and try again.</Message><BucketName>harmony-uat-staging</BucketName><RequestId>3KDANE7WF976V34N</RequestId><HostId>SjY5MpO95JO7GJNLWoMrvbLjgpvS0/fKYzAYt2d4IlZlHDxSzPVp+Pg2yHdinX9fMG/fxeBRhs8=</HostId></Error>'
headers:
Content-Type:
- application/xml
Date:
- Tue, 28 Apr 2026 12:17:24 GMT
Server:
- AmazonS3
Transfer-Encoding:
- chunked
x-amz-bucket-region:
- us-west-2
x-amz-id-2:
- SjY5MpO95JO7GJNLWoMrvbLjgpvS0/fKYzAYt2d4IlZlHDxSzPVp+Pg2yHdinX9fMG/fxeBRhs8=
x-amz-request-id:
- 3KDANE7WF976V34N
status:
code: 409
message: Conflict
- request:
body: <CreateBucketConfiguration xmlns="http://s3.amazonaws.com/doc/2006-03-01/"><LocationConstraint>us-west-2</LocationConstraint></CreateBucketConfiguration>
headers:
Content-Length:
- '153'
User-Agent:
- !!binary |
Qm90bzMvMS4yNi4xNjUgUHl0aG9uLzMuMTAuMTMgRGFyd2luLzI0LjYuMCBCb3RvY29yZS8xLjI5
LjE2NQ==
X-Amz-Content-SHA256:
- !!binary |
YTRhM2QyNjMxNzliZmNiYWY4ODc1NWIxNGVhOWRiMzA0OGU0MTc5YzAzYmJjOTRjY2YwZjEzYzhj
MmQzZGY2YQ==
X-Amz-Date:
- !!binary |
MjAyNjA0MjhUMTM1NDA5Wg==
amz-sdk-invocation-id:
- !!binary |
NGM2YTI3OGUtYWI2ZS00ZDY2LWE2MWMtOGQ0ODY5ZDUyODkx
amz-sdk-request:
- !!binary |
YXR0ZW1wdD0x
method: PUT
uri: https://harmony-uat-staging.s3.us-west-2.amazonaws.com/
response:
body:
string: '<?xml version="1.0" encoding="UTF-8"?>

<Error><Code>BucketAlreadyExists</Code><Message>The requested bucket name
is not available. The bucket namespace is shared by all users of the system.
Please select a different name and try again.</Message><BucketName>harmony-uat-staging</BucketName><RequestId>XZ4WJ0RP7BAVS2S2</RequestId><HostId>R/NYuETRTfQoJN9u01s056AkIFtbPMBuwRZ6B001HZxo3Cy2n+h2KrN21yxcQvTKjLPfkg3jg3qbz/JsH4v3GA==</HostId></Error>'
headers:
Content-Type:
- application/xml
Date:
- Tue, 28 Apr 2026 13:54:09 GMT
Server:
- AmazonS3
Transfer-Encoding:
- chunked
x-amz-bucket-region:
- us-west-2
x-amz-id-2:
- R/NYuETRTfQoJN9u01s056AkIFtbPMBuwRZ6B001HZxo3Cy2n+h2KrN21yxcQvTKjLPfkg3jg3qbz/JsH4v3GA==
x-amz-request-id:
- XZ4WJ0RP7BAVS2S2
status:
code: 409
message: Conflict
- request:
body: !!python/object/new:_io.BytesIO
state: !!python/tuple
- !!binary |
ZmFrZSBpbWFnZSBkYXRhIGZvciB0ZXN0aW5n
- 0
- null
headers:
Content-Length:
- '27'
Content-MD5:
- !!binary |
dTlXQ0gvREpaZ1RqQnVYbXBVOUE0UT09
Expect:
- !!binary |
MTAwLWNvbnRpbnVl
User-Agent:
- !!binary |
Qm90bzMvMS4yNi4xNjUgUHl0aG9uLzMuMTAuMTMgRGFyd2luLzI0LjYuMCBCb3RvY29yZS8xLjI5
LjE2NQ==
X-Amz-Content-SHA256:
- !!binary |
VU5TSUdORUQtUEFZTE9BRA==
X-Amz-Date:
- !!binary |
MjAyNjA0MjhUMTM1NDEwWg==
amz-sdk-invocation-id:
- !!binary |
ZDM3MzZjMGItNmMyZi00YjM2LWIyNGEtMWMzNGZmZjA4MTc0
amz-sdk-request:
- !!binary |
YXR0ZW1wdD0x
method: PUT
uri: https://harmony-uat-staging.s3.us-west-2.amazonaws.com/public/3d276f84-56e2-4f0a-acb2-35b9fcaaa317/9202859/PREFIRE_SAT2_2B-FLX_S07_R00_20210721013413_03040.nc.G00.png
response:
body:
string: '<?xml version="1.0" encoding="UTF-8"?>

<Error><Code>AccessDenied</Code><Message>User: arn:aws:iam::536711851782:user/NGAPShApplicationDeveloper-bcwong1-580

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Account number and ARN for a user are exposed, this should be removed from the cassette file.

is not authorized to perform: s3:PutObject on resource: "arn:aws:s3:::harmony-uat-staging/public/3d276f84-56e2-4f0a-acb2-35b9fcaaa317/9202859/PREFIRE_SAT2_2B-FLX_S07_R00_20210721013413_03040.nc.G00.png"
because no resource-based policy allows the s3:PutObject action</Message><RequestId>EQNCR3A24D1J8J7B</RequestId><HostId>fLXDRiPEiJDpPvoYwRr2ODSjX6WKk8+3FR0PmA0fyr053y4YnthlfNV7DZSxjiwU8J4RPm1yssvW8bdfn4NTXw==</HostId></Error>'
headers:
Connection:
- close
Content-Type:
- application/xml
Date:
- Tue, 28 Apr 2026 13:54:09 GMT
Server:
- AmazonS3
Transfer-Encoding:
- chunked
x-amz-id-2:
- fLXDRiPEiJDpPvoYwRr2ODSjX6WKk8+3FR0PmA0fyr053y4YnthlfNV7DZSxjiwU8J4RPm1yssvW8bdfn4NTXw==
x-amz-request-id:
- EQNCR3A24D1J8J7B
status:
code: 403
message: Forbidden
- request:
body: <CreateBucketConfiguration xmlns="http://s3.amazonaws.com/doc/2006-03-01/"><LocationConstraint>us-west-2</LocationConstraint></CreateBucketConfiguration>
headers:
Content-Length:
- '153'
User-Agent:
- !!binary |
Qm90bzMvMS4yNi4xNjUgUHl0aG9uLzMuMTAuMTMgRGFyd2luLzI0LjYuMCBCb3RvY29yZS8xLjI5
LjE2NQ==
X-Amz-Content-SHA256:
- !!binary |
YTRhM2QyNjMxNzliZmNiYWY4ODc1NWIxNGVhOWRiMzA0OGU0MTc5YzAzYmJjOTRjY2YwZjEzYzhj
MmQzZGY2YQ==
X-Amz-Date:
- !!binary |
MjAyNjA0MjhUMTM1NjQ1Wg==
amz-sdk-invocation-id:
- !!binary |
N2Q1ZWIzZjEtZWEwNi00N2U1LWEzYTItN2Q3N2JhZjM5NmNj
amz-sdk-request:
- !!binary |
YXR0ZW1wdD0x
method: PUT
uri: https://harmony-uat-staging.s3.us-west-2.amazonaws.com/
response:
body:
string: '<?xml version="1.0" encoding="UTF-8"?>

<Error><Code>BucketAlreadyExists</Code><Message>The requested bucket name
is not available. The bucket namespace is shared by all users of the system.
Please select a different name and try again.</Message><BucketName>harmony-uat-staging</BucketName><RequestId>GX93N1KJ872KXDC7</RequestId><HostId>LKQ8fp13yzWAVRjOAcanxIYz73LuMyti0FSQMzgf3hguKCc/2WPppBxM4OYDWN/pLrBQehzPPSk=</HostId></Error>'
headers:
Content-Type:
- application/xml
Date:
- Tue, 28 Apr 2026 13:56:46 GMT
Server:
- AmazonS3
Transfer-Encoding:
- chunked
x-amz-bucket-region:
- us-west-2
x-amz-id-2:
- LKQ8fp13yzWAVRjOAcanxIYz73LuMyti0FSQMzgf3hguKCc/2WPppBxM4OYDWN/pLrBQehzPPSk=
x-amz-request-id:
- GX93N1KJ872KXDC7
status:
code: 409
message: Conflict
- request:
body: !!python/object/new:_io.BytesIO
state: !!python/tuple
- !!binary |
ZmFrZSBpbWFnZSBkYXRhIGZvciB0ZXN0aW5n
- 0
- null
headers:
Content-Length:
- '27'
Content-MD5:
- !!binary |
dTlXQ0gvREpaZ1RqQnVYbXBVOUE0UT09
Expect:
- !!binary |
MTAwLWNvbnRpbnVl
User-Agent:
- !!binary |
Qm90bzMvMS4yNi4xNjUgUHl0aG9uLzMuMTAuMTMgRGFyd2luLzI0LjYuMCBCb3RvY29yZS8xLjI5
LjE2NQ==
X-Amz-Content-SHA256:
- !!binary |
VU5TSUdORUQtUEFZTE9BRA==
X-Amz-Date:
- !!binary |
MjAyNjA0MjhUMTM1NjQ2Wg==
amz-sdk-invocation-id:
- !!binary |
M2Y4YWQzNGQtYmQzZS00M2ViLWFiYjAtNTlhYjMxZWQxNjgx
amz-sdk-request:
- !!binary |
YXR0ZW1wdD0x
method: PUT
uri: https://harmony-uat-staging.s3.us-west-2.amazonaws.com/public/3d276f84-56e2-4f0a-acb2-35b9fcaaa317/9202859/PREFIRE_SAT2_2B-FLX_S07_R00_20210721013413_03040.nc.G00.png
response:
body:
string: '<?xml version="1.0" encoding="UTF-8"?>

<Error><Code>AccessDenied</Code><Message>User: arn:aws:iam::536711851782:user/NGAPShApplicationDeveloper-bcwong1-580
is not authorized to perform: s3:PutObject on resource: "arn:aws:s3:::harmony-uat-staging/public/3d276f84-56e2-4f0a-acb2-35b9fcaaa317/9202859/PREFIRE_SAT2_2B-FLX_S07_R00_20210721013413_03040.nc.G00.png"
because no resource-based policy allows the s3:PutObject action</Message><RequestId>GX9C5QGV8VCN50B3</RequestId><HostId>iq7RXLPY4W/n09vtZ7afhBtw97J5kjFmBJ97/hA2Z7K62SScb5c9vjlFAleCZkvsGoFcKUDxOGI=</HostId></Error>'
headers:
Connection:
- close
Content-Type:
- application/xml
Date:
- Tue, 28 Apr 2026 13:56:46 GMT
Server:
- AmazonS3
Transfer-Encoding:
- chunked
x-amz-id-2:
- iq7RXLPY4W/n09vtZ7afhBtw97J5kjFmBJ97/hA2Z7K62SScb5c9vjlFAleCZkvsGoFcKUDxOGI=
x-amz-request-id:
- GX9C5QGV8VCN50B3
status:
code: 403
message: Forbidden
version: 1
Loading
Loading