Skip to main content Link Menu Expand (external link) Document Search Copy Copied

EP Metrics Test Cases

Test Case ID: CMN/EP/EPMETRICS/METRICS/BV/01: Verify Endpoint has corresponding metrics but not invalid metrics enabled

Field Value
Case ID C103619324
Executable by Customer No
Test Steps
Step Description Expected Result
1 Initialize stack and start:
sid init 1 / sid start (BLE only)
sid init 2 / sid start (FSK only)
sid init 3 / sid start (LoRa only)
sid init 4 / sid start (BLE+LoRa)
sid init 5 / sid start (BLE+FSK)
sid init 6 / sid start (FSK+LoRa)
sid init 7 / sid start (BLE+FSK+LoRa)
CMD: ERR: 0
2 For non-LoRa stacks (BLE, FSK, BLE+LoRa, BLE+FSK, FSK+LoRa, BLE+FSK+LoRa), initialize SBDT:
sbdt init
CMD: ERR: 0
3 Verify expected metrics categories exist, unexpected ones do not:
print metrics 0x36
Integration Type / Present / Not Present:
BLE only: BLE_LINK, SBDT / LORA_MAC, FSK_MAC, NWK_SYNC, GWD
FSK only: FSK_MAC, FSK_LINK, NWK_SYNC, GWD, FFN, SBDT / LORA_MAC, BLE_LINK
LoRa only: LORA_MAC, LORA_LINK, NWK_SYNC / FSK_MAC, BLE_LINK, GWD, SBDT
BLE + LoRa: LORA_MAC, LORA_LINK, BLE_LINK, NWK_SYNC, SBDT / FSK_MAC, GWD
BLE + FSK: FSK_MAC, FSK_LINK, BLE_LINK, NWK_SYNC, GWD, FFN, SBDT / LORA_MAC
FSK + LoRa: LORA_MAC, LORA_LINK, FSK_MAC, FSK_LINK, NWK_SYNC, GWD, FFN, SBDT / BLE_LINK
BLE + FSK + LoRa: All categories present / None

Test Case ID: BLE/EP/EPMETRICS/REPORT/BV/01: Verify Endpoint sends Endpoint Metrics to Cloud over BLE in a specific time interval

Field Value
Case ID C103619325
Executable by Customer No
Test Steps
Step Description Expected Result
1 Initialize BLE stack and wait for time sync:
sid init 1 (or 4, 5, 7 - any stack with BLE enabled)
sid start
EVENT SID STATUS: State: 0, Reg: 0, Time: 0, Link_Mask: 1
Key check: Time synced (Time: 0)
2 Print all metrics:
print metrics 0x36
EN shows metrics data
3 Wait uploading interval for periodic reporting or trigger by command:
metrics send
Default uploading interval is 24 hours
Check current interval: print get_cap 3 -> metrics periodicity
ENC: M:15 [B:E:2]
TX CRID: <crid>
Key check: EN creates BLE connection and sends metrics to cloud
4 Check on Splunk by TX CRID:
index=halo_operations* <sidewalkID> methodIdentifier = "ENDPOINT_METRICS_PERIODIC"
Metrics data uploaded to cloud

Test Case ID: BLE/EP/EPMETRICS/REPORT/BV/02: Verify Endpoint sends remaining Endpoint Metrics to Cloud over BLE via piggybacking on a periodic reporting

Field Value
Case ID C103619326
Executable by Customer No
Test Steps
Step Description Expected Result
1 Initialize BLE+FSK+LoRa stack and wait for time sync:
sid init 7
sid start
EVENT SID STATUS: State: 0, Reg: 0, Time: 0, Link_Mask: 1
Key check: Time synced (Time: 0)
2 Initialize SBDT:
sbdt init
CMD: ERR: 0
3 Set all metrics to boundary value to exceed BLE MTU (255). Set all metrics categories (LoRa MAC, LoRa Link, FSK MAC, FSK Link, BLE Link, Time Sync, Network Sync, Link Message Manager, Security State Machine, Gateway Discovery and Tracking, Registration and Key Refresh, FFS/FFN, SBDT, Multi-Link Manager, Autoconnect Manager, Gateway Scanning and Selection) to their maximum boundary values using:
metrics set <category> <metric_id> <max_value>
CMD: ERR: 0 for each command
4 Print all metrics:
print metrics 0x36
EN shows all metrics data
5 Establish BLE connection and trigger metrics send or wait for periodic reporting:
sid conn_req 1
metrics send
ENC: M:15 [B:E:2]
TX CRID: <crid>
MET: NDB5 st:0
Key check: EN sends metrics via BLE, remaining metrics via piggybacking (NDB)
6 Send ULs until no more metrics in NDB (all metrics uploaded):
sid send TEST (repeat until MET: NDB5 st:0 no longer appears)
While metrics remain: MET: NDB5 st:0 (metrics piggybacked on UL)
When complete: No MET: NDB5 st:0 after sid send
Key check: All metrics uploaded via piggybacking
7 Check on Splunk by TX CRID:
index=halo_operations* <sidewalkID> methodIdentifier = "ENDPOINT_METRICS_PERIODIC"
Metrics data uploaded to cloud

Test Case ID: BLE/EP/EPMETRICS/REPORT/BV/03: Verify Endpoint Metrics are uploaded via Piggybacking over BLE if the explicit reporting is disabled

Field Value
Case ID C103619327
Executable by Customer No
Test Steps
Step Description Expected Result
1 Initialize BLE stack and wait for time sync:
sid init 1
sid start
EVENT SID STATUS: State: 0, Reg: 0, Time: 0, Link_Mask: 1
Key check: Time synced (Time: 0)
2 Disable metrics reporting via API:
{"deviceId": "{<DEVICE_ID>}", "enableMetricsReport": false}
Received write capability and config message:
[Rx][0:2:1:1] err:0
ECPFG(8) st 10->2
ECPFG(2) st 2->5
Response write capability and config message:
[Tx][0:2:3:1] st:0
ECPFG(5) st 5->10
Key check: EN received configuration and sent response
3 Print all metrics:
print metrics 0x36
EN shows all metrics data
4 Establish BLE connection and trigger metrics send, verify piggybacking:
sid conn_req 1
metrics send
sid send TEST (repeat until MET: NDB5 st:0 no longer appears)
MET: NDB5 st:0 (remaining metrics piggybacked on UL)
When complete: No MET: NDB5 st:0 after sid send
Key check: All metrics uploaded via piggybacking even though explicit reporting is disabled

Field Value
Case ID C103619328
Executable by Customer No
Test Steps
Step Description Expected Result
1 Initialize BLE stack, wait for time sync, establish BLE connection, clear metrics:
sid init 1
sid start
sid conn_req 1
clear metrics
EVENT SID STATUS: State: 0, Reg: 0, Time: 0, Link_Mask: 1
EVENT SID STATUS LINK MODE: LORA: 0, FSK: 0, BLE: 1
CMD: ERR: 0
2 Send 3 ULs and verify ul_msg counter increments after each:
sid send TEST (1st UL)
print metrics 0x05 (check BLE_LINK: ul_msg = 1)
sid send TEST (2nd UL)
print metrics 0x05 (check BLE_LINK: ul_msg = 2)
sid send TEST (3rd UL)
print metrics 0x05 (check BLE_LINK: ul_msg = 3)
After 1st UL: BLE_LINK: ul_msg = 1
After 2nd UL: BLE_LINK: ul_msg = 2
After 3rd UL: BLE_LINK: ul_msg = 3
Key check: ul_msg counter increments by 1 after each UL
3 Upload to Cloud to check via S360:
https://360.sidewalk.amazon/jobs/create
Job Type: Perform Action For Endpoint Metrics
Action: Get Metrics
Metrics Category: BLE LINK
Individual Metrics: Uplink Messages Per Day
ENC: M:15 [B:E:3]
TX CRID: <crid>
Key check: Metrics uploaded, note the TX CRID
4 Check on Splunk by TX CRID The metrics is in the event

Field Value
Case ID C103619329
Executable by Customer No
Test Steps
Step Description Expected Result
1 Initialize BLE stack, wait for time sync, clear metrics:
sid init 1 (or 4, 5, 7)
sid start
print clear_metrics 0x36
EVENT SID STATUS: State: 0, Reg: 0, Time: 0, Link_Mask: 1
BLE_LINK: dl_msg = 0
2 Send 3 DLs from AWS IoT and verify dl_msg counter increments after each:
Send DL (seq=0) -> print metrics 0x05
Send DL (seq=1) -> print metrics 0x05
Send DL (seq=2) -> print metrics 0x05
After 1st DL: EVENT SID RECEIVED: ... LINK: 1 ... / BLE_LINK: dl_msg = 1
After 2nd DL: EVENT SID RECEIVED: ... LINK: 1 ... / BLE_LINK: dl_msg = 2
After 3rd DL: EVENT SID RECEIVED: ... LINK: 1 ... / BLE_LINK: dl_msg = 3
Key check: dl_msg counter increments by 1 after each DL
3 Upload metrics to Cloud via S360:
https://gamma.360.sidewalk.amazon/jobs/create
Job Type: Periodic Metrics Upload
ENC: M:15 [B:E:3]
TX CRID: <crid>
Key check: Metrics uploaded, note the TX CRID
4 Check on Splunk by TX CRID:
index=halo_operations* <sidewalkID> methodIdentifier = "ENDPOINT_METRICS_PERIODIC"
Metrics data found on Splunk with correct dl_msg value

Test Case ID: BLE/EP/EPMETRICS/CONN-BV-01: Verify Connection attempts Metrics

Field Value
Case ID C103619330
Executable by Customer No
Test Steps
Step Description Expected Result
1 Initialize BLE stack, wait for time sync, wait for BLE disconnected, clear metrics:
sid init 1
sid start
Wait for BLE disconnected
print clear_metrics 0x36
EVENT SID STATUS: State: 0, Reg: 0, Time: 0, Link_Mask: 1
EVENT SID STATUS LINK MODE: LORA: 0, FSK: 0, BLE: 0
BLE_LINK: conn_attempts = 0
2 Establish BLE connection 2 times and verify conn_attempts increments:
sid conn_req 1
print metrics 0x05 (check BLE_LINK: conn_attempts = 1)
sid conn_req 1
print metrics 0x05 (check BLE_LINK: conn_attempts = 2)
After 1st connection: BLE_LINK: conn_attempts = 1
After 2nd connection: BLE_LINK: conn_attempts = 2
Key check: conn_attempts counter increments by 1 after each connection attempt
3 Upload to Cloud to check via S360:
https://gamma.360.sidewalk.amazon/jobs/create
Job Type: Perform Action For Endpoint Metrics
Action: Get Metrics
Metrics Category: BLE LINK
Individual Metrics: Connection Attempts Per Day
ENC: M:15 [B:E:3]
TX CRID: <crid>
Key check: Metrics uploaded, note the TX CRID
4 Check on Splunk by TX CRID The metrics is in the event

Test Case ID: FSK/EP/EPMETRICS/REPORT/BV/01: Verify Endpoint sends Endpoint Metrics to Cloud over FSK in a specific time interval

Field Value
Case ID C103619332
Executable by Customer No
Test Steps
Step Description Expected Result
1 Initialize FSK stack and wait for time sync and join:
sid init 2 (or 5, 6, 7)
sid start
EVENT SID STATUS: State: 0, Reg: 0, Time: 0, Link_Mask: 2
Njoin(1, 2) st 2->6
Key check: Time synced and joined
2 Print all metrics:
print metrics 0x36
EN shows metrics data
3 Wait uploading interval for periodic reporting or trigger by command:
metrics send
Default uploading interval is 24 hours
Check current interval: print get_cap 3 -> metrics periodicity
ENC: M:15 [B:E:2]
TX CRID: <crid>
Key check: EN sends metrics to cloud via FSK
4 Check on Splunk by TX CRID:
index=halo_operations* <sidewalkID> methodIdentifier = "ENDPOINT_METRICS_PERIODIC"
Metrics data found on Splunk

Test Case ID: FSK/EP/EPMETRICS/REPORT/BV/02: Verify Endpoint sends remaining Endpoint Metrics to Cloud over FSK via piggybacking on a periodic reporting

Field Value
Case ID C103619333
Executable by Customer No
Test Steps
Step Description Expected Result
1 Initialize BLE+FSK+LoRa stack and wait for time sync and join:
sid init 7
sid start
EVENT SID STATUS: State: 0, Reg: 0, Time: 0, Link_Mask: 2
Njoin(1, 2) st 2->6
2 Initialize SBDT:
sbdt init
CMD: ERR: 0
3 Set all metrics to boundary value to exceed FSK MTU (200). Set all metrics categories (FSK MAC, FSK Link, Time Sync, Network Sync, Link Message Manager, Security State Machine, Gateway Discovery and Tracking, Registration and Key Refresh, FFS/FFN, SBDT, Multi-Link Manager, Autoconnect Manager, Gateway Scanning and Selection) to their maximum boundary values using:
metrics set <category> <metric_id> <max_value>
CMD: ERR: 0 for each command
4 Print all metrics:
print metrics 0x36
EN shows all metrics data
5 Trigger metrics send:
metrics send
sid send TEST (repeat until MET: NDB5 st:0 no longer appears)
ENC: M:15 [B:E:2]
TX CRID: <crid>
MET: NDB5 st:0 (remaining metrics piggybacked on UL)
When complete: No MET: NDB5 st:0 after sid send
Key check: EN sends metrics via FSK, remaining via piggybacking
6 Check on Splunk by TX CRID:
index=halo_operations* <sidewalkID> methodIdentifier = "ENDPOINT_METRICS_PERIODIC"
Metrics data found on Splunk

Test Case ID: FSK/EP/EPMETRICS/REPORT/BV/03: Verify Endpoint Metrics are uploaded via Piggybacking over FSK if the explicit reporting is disabled

Field Value
Case ID C103619334
Executable by Customer No
Test Steps
Step Description Expected Result
1 Initialize FSK stack and wait for time sync and join:
sid init 2 (or 5, 6, 7)
sid start
EVENT SID STATUS: State: 0, Reg: 0, Time: 0, Link_Mask: 2
Njoin(1, 2) st 2->6
2 Disable metrics reporting via API:
{"deviceId": "{<DEVICE_ID>}", "enableMetricsReport": false}
Received write capability and config message:
[Rx][0:2:1:2] err:0
ECPFG(8) st 10->2
ECPFG(2) st 2->5
Response write capability and config message:
[Tx][0:2:3:2] st:0
ECPFG(5) st 5->10
Key check: EN received configuration and sent response
3 Print all metrics:
print metrics 0x36
EN shows all metrics data
4 Trigger metrics send and verify piggybacking:
metrics send
sid send TEST (repeat until MET: NDB5 st:0 no longer appears)
MET: NDB5 st:0 (remaining metrics piggybacked on UL)
When complete: No MET: NDB5 st:0 after sid send
Key check: All metrics uploaded via piggybacking even though explicit reporting is disabled
5 Recover metrics reporting setting:
print clear_cap 1
CMD: ERR: 0
6 Check on Splunk by TX CRID:
index=halo_operations* <sidewalkID> methodIdentifier = "ENDPOINT_METRICS_PERIODIC"
Metrics data found on Splunk

Field Value
Case ID C103619335
Executable by Customer No
Test Steps
Step Description Expected Result
1 Initialize FSK stack, wait for time sync and join, clear metrics:
sid init 2 (or 5, 6, 7)
sid start
print clear_metrics 0x36
EVENT SID STATUS: State: 0, Reg: 0, Time: 0, Link_Mask: 2
Njoin(1, 2) st 2->6
FSK_MAC: fsk_tx_pkts = 0
2 Send UL and verify fsk_tx_pkts increments:
sid send TEST
print metrics 0x03 (check FSK_MAC: fsk_tx_pkts = 1)
FskTx:TxDone sqn:<sqn> dst_id:0x<id> ch:<ch> pktlen:<len> tx_st:0
FSK_MAC: fsk_tx_pkts = 1
Key check: fsk_tx_pkts increments by 1 after each TX
3 Upload metrics to Cloud via S360:
https://360.sidewalk.amazon/jobs/create
Job Type: Periodic Metrics Upload
ENC: M:15 [B:E:3]
TX CRID: <crid>
Key check: Metrics uploaded, note the TX CRID
4 Check on Splunk by TX CRID:
index=halo_operations* <sidewalkID> methodIdentifier = "ENDPOINT_METRICS_PERIODIC"
Metrics data found on Splunk

Field Value
Case ID C103619336
Executable by Customer No
Test Steps
Step Description Expected Result
1 Initialize FSK stack, wait for time sync and join, clear metrics:
sid init 2 (or 5, 6, 7)
sid start
print clear_metrics 0x36
EVENT SID STATUS: State: 0, Reg: 0, Time: 0, Link_Mask: 2
Njoin(1, 2) st 2->6
FSK_MAC: fsk_rx_pkts = 0
2 Send DL from AWS IoT and verify fsk_rx_pkts increments:
Send DL from AWS IoT
print metrics 0x03 (check FSK_MAC: fsk_rx_pkts = 1)
FskRx:RxDone sqn:<sqn> src_id:0x<id> dst_id:0x<id> ch:<ch> pktlen:<len> rx_st:0
FSK_MAC: fsk_rx_pkts = 1
Key check: fsk_rx_pkts increments by 1 after each RX
3 Upload metrics to Cloud via S360:
https://360.sidewalk.amazon/jobs/create
Job Type: Periodic Metrics Upload
ENC: M:15 [B:E:3]
TX CRID: <crid>
Key check: Metrics uploaded, note the TX CRID
4 Check on Splunk by TX CRID:
index=halo_operations* <sidewalkID> methodIdentifier = "ENDPOINT_METRICS_PERIODIC"
Metrics data found on Splunk

Test Case ID: LORA/EP/EPMETRICS/REPORT/BV/01: Verify Endpoint sends Endpoint Metrics to Cloud over LoRa in a specific time interval

Field Value
Case ID C103619339
Executable by Customer No
Test Steps
Step Description Expected Result
1 Initialize LoRa stack and wait for time sync and join:
sid init 3 (or 4, 6, 7)
sid start
EVENT SID STATUS: State: 0, Reg: 0, Time: 0, Link_Mask: 4
Njoin(1, 2) st 2->6
Key check: Time synced and joined
2 Print all metrics:
print metrics 0x36
EN shows metrics data
3 Wait uploading interval for periodic reporting or trigger by command:
metrics send
Default uploading interval is 24 hours
Check current interval: print get_cap 3 -> metrics periodicity
ENC: M:15 [B:E:2]
TX CRID: <crid>
Key check: EN sends metrics to cloud via LoRa
4 Check on Splunk by TX CRID:
index=halo_operations* <sidewalkID> methodIdentifier = "ENDPOINT_METRICS_PERIODIC"
Metrics data found on Splunk

Test Case ID: LORA/EP/EPMETRICS/REPORT/BV/02: Verify Endpoint sends remaining Endpoint Metrics to Cloud over LoRa via piggybacking on a periodic reporting

Field Value
Case ID C103619340
Executable by Customer No
Test Steps
Step Description Expected Result
1 Initialize BLE+FSK+LoRa stack and wait for time sync and join:
sid init 7
sid start
EVENT SID STATUS: State: 0, Reg: 0, Time: 0, Link_Mask: 4
Njoin(1, 2) st 2->6
2 Set all metrics to boundary value (see BLE/EP/EPMETRICS/REPORT/BV/02 step 3 for full command list) CMD: ERR: 0 for each command
3 Print all metrics:
print metrics 0x36
EN shows all metrics data
4 Trigger metrics send:
metrics send
sid send TEST (repeat until MET: NDB5 st:0 no longer appears)
ENC: M:15 [B:E:2]
TX CRID: <crid>
MET: NDB5 st:0 (remaining metrics piggybacked on UL)
When complete: No MET: NDB5 st:0 after sid send
Key check: EN sends metrics via LoRa, remaining via piggybacking
5 Check on Splunk by TX CRIDs:
index=halo_operations* <sidewalkID> methodIdentifier = "ENDPOINT_METRICS_PERIODIC"
index=halo_operations* <sidewalkID> methodIdentifier = "ENDPOINT_METRICS_PIGGYBACK"
Metrics data found on Splunk (both periodic and piggybacked)

Test Case ID: LORA/EP/EPMETRICS/REPORT/BV/03: Verify Endpoint Metrics are uploaded via Piggybacking over LoRa if the explicit reporting is disabled

Field Value
Case ID C103619341
Executable by Customer No
Test Steps
Step Description Expected Result
1 Initialize LoRa stack and wait for time sync and join:
sid init 3 (or 4, 6, 7)
sid start
EVENT SID STATUS: State: 0, Reg: 0, Time: 0, Link_Mask: 4
Njoin(1, 2) st 2->6
2 Disable metrics reporting via API:
{"deviceId": "{<DEVICE_ID>}", "enableMetricsReport": false}
Received write capability and config message:
[Rx][0:2:1:4] err:0
ECPFG(8) st 10->2
ECPFG(2) st 2->5
Response write capability and config message:
[Tx][0:2:3:4] st:0
ECPFG(5) st 5->10
Key check: EN received configuration and sent response
3 Verify metrics reporting is disabled:
print get_cap 1
metrics enabled 0
4 Print all metrics:
print metrics 0x36
EN shows all metrics data
5 Trigger metrics send and verify piggybacking:
metrics send
sid send TEST (repeat until MET: NDB5 st:0 no longer appears)
MET: NDB5 st:0 (remaining metrics piggybacked on UL)
When complete: No MET: NDB5 st:0 after sid send
Key check: All metrics uploaded via piggybacking even though explicit reporting is disabled
6 Recover metrics reporting setting:
print clear_cap 1
CMD: ERR: 0
7 Check on Splunk by TX CRIDs:
index=halo_operations* <sidewalkID> methodIdentifier = "ENDPOINT_METRICS_PIGGYBACK"
Metrics data found on Splunk (piggybacked only, no periodic)

Field Value
Case ID C103619342
Executable by Customer No
Test Steps
Step Description Expected Result
1 Initialize LoRa stack, wait for time sync and join, clear metrics:
sid init 3 (or 4, 6, 7)
sid start
print clear_metrics 0x36
EVENT SID STATUS: State: 0, Reg: 0, Time: 0, Link_Mask: 4
Njoin(1, 2) st 2->6
LORA_MAC: lora_tx_normal_pkts = 0
2 Send UL and verify lora_tx_normal_pkts increments:
sid send TEST
print metrics 0x01 (check LORA_MAC: lora_tx_normal_pkts = 1)
LoraTx:TxDone sqn:<sqn> dst_id:0x<id> ch:<ch> pktlen:<len> tx_st:0
LORA_MAC: lora_tx_normal_pkts = 1
Key check: lora_tx_normal_pkts increments by 1 after each TX
3 Upload metrics to Cloud via S360:
https://gamma.360.sidewalk.amazon/jobs/create
Job Type: Periodic Metrics Upload
ENC: M:15 [B:E:3]
TX CRID: <crid>
Key check: Metrics uploaded, note the TX CRID
4 Check on Splunk by TX CRID:
index=halo_operations* <sidewalkID> methodIdentifier = "ENDPOINT_METRICS_PERIODIC"
Metrics data found on Splunk

Field Value
Case ID C103619343
Executable by Customer No
Test Steps
Step Description Expected Result
1 Initialize LoRa stack, wait for time sync and join, clear metrics:
sid init 3 (or 4, 6, 7)
sid start
print clear_metrics 0x36
EVENT SID STATUS: State: 0, Reg: 0, Time: 0, Link_Mask: 4
Njoin(1, 2) st 2->6
LORA_MAC: lora_rx_pkts = 0
2 Send DL from AWS IoT and verify lora_rx_pkts increments:
Send DL from AWS IoT
print metrics 0x01 (check LORA_MAC: lora_rx_pkts = 1)
LoraRx:RxDone sqn:<sqn> src_id:0x<id> dst_id:0x<id> ch:<ch> pktlen:<len> rx_st:0
LORA_MAC: lora_rx_pkts = 1
Key check: lora_rx_pkts increments by 1 after each RX
3 Upload metrics to Cloud via S360:
https://gamma.360.sidewalk.amazon/jobs/create
Job Type: Periodic Metrics Upload
ENC: M:15 [B:E:3]
TX CRID: <crid>
Key check: Metrics uploaded, note the TX CRID
4 Check on Splunk by TX CRID:
index=halo_operations* <sidewalkID> methodIdentifier = "ENDPOINT_METRICS_PERIODIC"
Metrics data found on Splunk

Back to top

©2023 Amazon.com, Inc. or its affiliates (collectively, “Amazon”). All Rights Reserved.