Skip to content

Ministack

9641f520cebd 5 recorded runs

65.1%
correct, across 674 of 684 operations · 10 unsupported
Tier 1 80.9%
Tier 2 73.7%
Tier 3 35.0%

Fully supports

Operation areas this target implements completely — every test passes, nothing skipped. Areas it only partly implements, or gets wrong, are in “Where it falls short” below.

  • deleteTable tier 1
  • describeTable tier 1
  • listTables tier 1
  • kinesis tier 2
  • streams tier 2

Conformance over time

55% 78% 100% 62.4% 27 Apr 68.4% 23 May 66.1% 24 May 65.5% 25 May 65.1% 26 May

Where it falls short

Operation groups with gaps in the latest run, biggest first. Unsupported means the target doesn't implement that feature at all, often by design. Open one for the exact tests, or see the conformance suite.

  • error-messages Tier 3 63 failing
    View these tests in the suite →
    • BatchGetItem — exact error messages empty RequestItems: full required-parameter error
    • BatchGetItem — exact error messages > 100 keys across all tables: interpolated full error
    • BatchGetItem — exact error messages non-existent table: full ResourceNotFoundException message
    • BatchGetItem — exact error messages duplicate keys in one Keys array: full duplicate-keys error
    • BatchWriteItem — exact error messages empty RequestItems: full required-parameter error
    • BatchWriteItem — exact error messages > 25 requests: anchored regex on the constraint phrase
    • BatchWriteItem — exact error messages same key Put and Delete in one batch: full duplicate-keys error
    • CreateTable — exact error messages missing TableName: full required-parameter error
    • CreateTable — exact error messages short table name (2 chars): minimum length 3 error
    • CreateTable — exact error messages duplicate attribute in KeySchema
    • CreateTable — exact error messages more than 2 KeySchema elements
    • CreateTable — exact error messages invalid KeyType
    • CreateTable — exact error messages invalid AttributeType
    • CreateTable — exact error messages LSI without range key on base table
    • CreateTable — exact error messages duplicate index names
    • DeleteItem — exact error messages non-existent table: full ResourceNotFoundException message
    • GetItem — exact error messages non-existent table: full ResourceNotFoundException message
    • GetItem — exact error messages invalid ProjectionExpression syntax: full parser error
    • PutItem — exact error messages missing table name: full validation error string
    • PutItem — exact error messages empty table name: minimum length 1 error
    • PutItem — exact error messages table name too long (256 chars): maximum length 255 error
    • PutItem — exact error messages table name with invalid chars: regex pattern error
    • PutItem — exact error messages empty string set: full parameter-values-invalid error
    • PutItem — exact error messages empty number set: full parameter-values-invalid error
    • PutItem — exact error messages duplicate values in SS: full duplicates error
    • PutItem — exact error messages NULL attr with false: full null-attribute error
    • PutItem — exact error messages ExpressionAttributeValues without expression: full unused-EAV error
    • PutItem — exact error messages redundant parentheses in ConditionExpression: full error string
    • PutItem — exact error messages contains() with duplicate path and operand: full distinct-operand error
    • Query — exact error messages missing hash key in KeyConditionExpression
    • Query — exact error messages non-key attribute in KeyConditionExpression
    • Query — exact error messages unused ExpressionAttributeNames
    • Query — exact error messages invalid Select value
    • Query — exact error messages ConsistentRead on GSI
    • Query — exact error messages Limit of 0
    • Query — exact error messages empty KeyConditionExpression
    • Query — exact error messages filter references undefined ExpressionAttributeNames
    • Query — exact error messages redundant parentheses in KeyConditionExpression: full error string
    • Query — exact error messages malformed ExclusiveStartKey: short invalid-starting-key error
    • Scan — exact error messages Segment without TotalSegments: full required-parameter error
    • Scan — exact error messages Segment >= TotalSegments: full out-of-range error
    • Scan — exact error messages TotalSegments without Segment: full required-parameter error
    • Scan — exact error messages Limit of 0: full minimum-value error
    • Scan — exact error messages non-existent table: full ResourceNotFoundException message
    • Scan — exact error messages begins_with with non-string operand: full operand-type error
    • Scan — exact error messages redundant parentheses in FilterExpression: full error string
    • Scan — exact error messages malformed ExclusiveStartKey: long schema-mismatch error
    • TransactGetItems — exact error messages empty TransactItems: full minimum-length error
    • TransactGetItems — exact error messages > 100 gets: anchored regex on the constraint phrase
    • TransactGetItems — exact error messages non-existent table: full ResourceNotFoundException message
    • TransactGetItems — exact error messages invalid ProjectionExpression syntax: full parser error
    • TransactGetItems — exact error messages missing key attribute: action-level ValidationError surfaces as TransactionCanceledException
    • TransactWriteItems — exact error messages empty TransactItems: full minimum-length error
    • TransactWriteItems — exact error messages > 100 actions: anchored regex on the constraint phrase
    • TransactWriteItems — exact error messages duplicate target keys in same transaction: full multi-op error
    • TransactWriteItems — exact error messages non-existent table: full ResourceNotFoundException message
    • UpdateItem — exact error messages cannot update hash key attribute
    • UpdateItem — exact error messages invalid UpdateExpression syntax
    • UpdateItem — exact error messages unused ExpressionAttributeNames
    • UpdateItem — exact error messages unused ExpressionAttributeValues
    • UpdateItem — exact error messages missing ExpressionAttributeValues reference
    • UpdateItem — exact error messages empty UpdateExpression
    • UpdateItem — exact error messages cannot update range key attribute on composite table
  • limits Tier 3 36 failing
    View these tests in the suite →
    • BatchWriteItem limits BatchWriteItem with 26 items fails with ValidationException
    • BatchGetItem limits BatchGetItem with 101 keys fails with ValidationException
    • Empty values — strings, binary, and sets empty string as hash key value is rejected
    • Empty values — strings, binary, and sets empty string as sort key value is rejected
    • Empty values — strings, binary, and sets empty string set (SS) is rejected
    • Empty values — strings, binary, and sets empty number set (NS) is rejected
    • Empty values — strings, binary, and sets empty binary set (BS) is rejected
    • Empty values — strings, binary, and sets empty set nested inside a Map is rejected
    • Empty values — strings, binary, and sets empty set nested inside a List is rejected
    • Item size limit (400KB) item over 400KB fails with ValidationException
    • Item size limit (400KB) size calculation includes attribute names
    • Item size limit (400KB) size includes nested map attribute names and values
    • Item size limit (400KB) size includes set elements
    • Item size limit (400KB) size includes list elements
    • Item size limit (400KB) large binary attribute over limit fails
    • Item size limit — number sizing a number-dominated item over 400KB fails (21000 numbers)
    • Number precision — DynamoDB number limits and edge cases rejects a number with 39 significant digits
    • Number precision — DynamoDB number limits and edge cases rejects number just over maximum positive: 1E+126
    • Number precision — DynamoDB number limits and edge cases rejects number just over maximum negative: -1E+126
    • Number precision — DynamoDB number limits and edge cases rejects number below minimum positive: 1E-131
    • Number precision — DynamoDB number limits and edge cases leading zeros are not counted as significant digits: 00042 treated as 42
    • Number precision — DynamoDB number limits and edge cases negative zero: -0 behavior
    • Number precision — DynamoDB number limits and edge cases leading zeros are stripped on round-trip
    • Number precision — DynamoDB number limits and edge cases trailing decimal zeros are stripped on round-trip
    • Number precision — DynamoDB number limits and edge cases scientific notation is normalized on round-trip
    • Number precision — DynamoDB number limits and edge cases rejects arithmetic that overflows the supported number magnitude
    • Reserved words — ExpressionAttributeNames handling reserved word without # in ConditionExpression fails
    • Reserved words — ExpressionAttributeNames handling reserved word without # in UpdateExpression fails
    • Reserved words — ExpressionAttributeNames handling reserved word without # in FilterExpression fails
    • Reserved words — ExpressionAttributeNames handling reserved word without # in ProjectionExpression fails
    • Reserved words — ExpressionAttributeNames handling multiple reserved words in same expression all need aliases
    • Response size limit (1MB) Query returns LastEvaluatedKey when response would exceed 1MB
    • Response size limit (1MB) Scan returns LastEvaluatedKey when response would exceed 1MB
    • TransactWriteItems limits TransactWriteItems with 101 actions fails with ValidationException
    • TransactWriteItems limits TransactWriteItems total item size over 4MB fails with ValidationException
    • TransactGetItems limits TransactGetItems with 101 items fails with ValidationException
  • validation-ordering Tier 3 31 failing
    View these tests in the suite →
    • Batch operations — validation ordering BatchWriteItem rejects empty RequestItems
    • Batch operations — validation ordering BatchGetItem rejects empty RequestItems
    • Batch operations — validation ordering BatchWriteItem rejects more than 25 items with exact count in message
    • Batch operations — validation ordering BatchGetItem rejects more than 100 keys with exact count in message
    • CreateTable — validation ordering empty TableName reports only tableName constraint
    • CreateTable — validation ordering invalid table name pattern reports only tableName
    • CreateTable — validation ordering reports invalid BillingMode and invalid KeySchema element together
    • CreateTable — validation ordering reports missing ProvisionedThroughput and invalid key type together
    • CreateTable — validation ordering rejects a duplicate attribute in KeySchema
    • CreateTable — validation ordering rejects more than two KeySchema elements
    • CreateTable — validation ordering rejects an invalid BillingMode on its own
    • DeleteItem — validation ordering empty TableName reports only tableName constraint
    • DeleteItem — validation ordering reports invalid ReturnValues and invalid ReturnConsumedCapacity together
    • DeleteItem — validation ordering invalid table name pattern reports only tableName
    • Non-existent index — error surface Query on a non-existent index reports ValidationException
    • Non-existent index — error surface Scan on a non-existent index reports ValidationException
    • Non-existent index — error surface UpdateTable deleting a non-existent GSI reports ResourceNotFoundException
    • PutItem — validation ordering empty TableName reports tableName constraint (stops early)
    • PutItem — validation ordering empty TableName with invalid ReturnValues reports only tableName
    • PutItem — validation ordering reports invalid ReturnConsumedCapacity, ReturnItemCollectionMetrics, and ReturnValues together
    • PutItem — validation ordering reports invalid table name pattern
    • Query — validation ordering reports invalid TableName pattern
    • Query — validation ordering reports invalid Limit value
    • Query — validation ordering reports invalid ReturnConsumedCapacity
    • Scan — validation ordering rejects Segment without TotalSegments
    • Scan — validation ordering rejects Segment >= TotalSegments
    • Scan — validation ordering rejects TotalSegments without Segment
    • Scan — validation ordering rejects a negative Segment
    • UpdateItem — validation ordering empty TableName reports only tableName constraint
    • UpdateItem — validation ordering reports invalid ReturnValues and invalid ReturnConsumedCapacity together
    • UpdateItem — validation ordering invalid table name pattern reports only tableName
  • partiql Tier 2 14 failing
    View these tests in the suite →
    • BatchExecuteStatement — PartiQL batch of multiple SELECT statements
    • BatchExecuteStatement — PartiQL batch of INSERT and UPDATE statements
    • BatchExecuteStatement — PartiQL partial failure — one valid and one invalid statement
    • BatchExecuteStatement — PartiQL rejects an empty Statements array
    • ExecuteStatement — PartiQL rejects INSERT on an existing item (INSERT is not upsert)
    • ExecuteStatement — PartiQL SELECT with nested map path
    • ExecuteStatement — PartiQL SELECT with begins_with in WHERE clause
    • ExecuteStatement — PartiQL PartiQL UPDATE with REMOVE
    • ExecuteStatement — PartiQL returns a populated ConsumedCapacity block when requested
    • ExecuteStatement — PartiQL evaluates negated predicates (NOT begins_with, IS NOT MISSING)
    • ExecuteTransaction — PartiQL transactional INSERT and UPDATE both succeed atomically
    • ExecuteTransaction — PartiQL transaction rolls back on duplicate key INSERT
    • ExecuteTransaction — PartiQL multiple INSERTs in one transaction
    • ExecuteTransaction — PartiQL rejects empty TransactStatements
  • putItem Tier 1 13 failing
    View these tests in the suite →
    • PutItem — ConditionExpression parens rejects redundant parentheses in ConditionExpression
    • PutItem — ConditionExpression parens rejects contains() with a duplicate path and operand
    • ReturnItemCollectionMetrics PutItem with SIZE returns ItemCollectionMetrics
    • ReturnItemCollectionMetrics DeleteItem with SIZE returns ItemCollectionMetrics
    • ReturnItemCollectionMetrics UpdateItem with SIZE returns ItemCollectionMetrics
    • PutItem — validation rejects item missing the hash key
    • PutItem — validation rejects empty string set
    • PutItem — validation rejects empty number set
    • PutItem — validation rejects empty binary set
    • PutItem — validation rejects duplicate values in string set
    • PutItem — validation rejects duplicate values in number set
    • PutItem — validation rejects duplicate values in binary set
    • PutItem — validation rejects invalid ReturnValues
  • query Tier 1 11 failing
    View these tests in the suite →
    • Query — Limit + FilterExpression interaction rejects ExclusiveStartKey that does not match the table schema
    • Query — Limit + FilterExpression interaction rejects ExclusiveStartKey missing the index key on a GSI query
    • Query — binary sort key returns items in byte-wise ascending order
    • Query — binary sort key returns items in descending order with ScanIndexForward=false
    • Query — binary sort key supports > comparison on binary sort key (byte-wise)
    • Query — binary sort key supports begins_with on binary sort key
    • Query — size() string encoding counts UTF-16 code units (3), not UTF-8 bytes (6)
    • Query — KeyConditionExpression with parentheses rejects redundant parentheses in KeyConditionExpression
    • Query — LSI INCLUDE projection returns only specified non-key attributes plus keys
    • Query — Select modes on indexes Select ALL_PROJECTED_ATTRIBUTES on GSI returns only projected attributes
    • Query — Select SPECIFIC_ATTRIBUTES Select SPECIFIC_ATTRIBUTES without ProjectionExpression is rejected
  • createTable Tier 1 10 failing
    View these tests in the suite →
    • CreateTable — validation rejects a table name shorter than 3 characters
    • CreateTable — validation rejects a table name longer than 255 characters
    • CreateTable — validation rejects a table name with invalid characters
    • CreateTable — validation rejects missing KeySchema
    • CreateTable — validation rejects unused AttributeDefinitions
    • CreateTable — configuration parameters TableClass STANDARD_INFREQUENT_ACCESS round-trips
    • CreateTable — configuration parameters SSESpecification with the AWS-managed key round-trips
    • CreateTable — configuration parameters OnDemandThroughput round-trips on a PAY_PER_REQUEST table
    • CreateTable — LSI rejects LSI on a hash-only table
    • CreateTable — LSI rejects LSI with a hash key different from the base table
  • transactions Tier 2 10 failing
    View these tests in the suite →
    • TransactGetItems - validation rejects empty TransactItems
    • TransactGetItems - validation rejects get on non-existent table
    • TransactGetItems - validation rejects duplicate keys in same transaction
    • TransactGetItems — ConsumedCapacity charges 2 read capacity units per item, including a missing item
    • TransactGetItems — ConsumedCapacity INDEXES breakdown includes the table read capacity units
    • TransactGetItems — projection matching nothing omits Item when the projection matches no attribute on a present item
    • TransactWriteItems - basic functionality rejects same ClientRequestToken with different payload
    • TransactWriteItems - validation rejects duplicate target items in same transaction
    • TransactWriteItems - validation rejects empty TransactItems
    • TransactWriteItems — ConsumedCapacity charges 2 write capacity units per item
  • legacy-api Tier 3 9 failing
    View these tests in the suite →
    • Legacy API — AttributeUpdates (legacy UpdateExpression) AttributeUpdates cannot modify key attributes
    • Legacy API — AttributesToGet (legacy ProjectionExpression) GetItem with AttributesToGet returns only specified attributes
    • Legacy API — AttributesToGet (legacy ProjectionExpression) GetItem with AttributesToGet does NOT auto-include key attributes
    • Legacy API — AttributesToGet (legacy ProjectionExpression) Query with AttributesToGet returns only specified attributes
    • Legacy API — AttributesToGet (legacy ProjectionExpression) Scan with AttributesToGet returns only specified attributes
    • Legacy API — AttributesToGet (legacy ProjectionExpression) BatchGetItem with AttributesToGet returns only specified attributes
    • Legacy API — AttributesToGet (legacy ProjectionExpression) Mixing AttributesToGet with ProjectionExpression throws ValidationException
    • Legacy API — AttributesToGet (legacy ProjectionExpression) AttributesToGet with single attribute
    • Legacy API — ScanFilter (legacy FilterExpression) Mixing ScanFilter with FilterExpression throws ValidationException
  • scan Tier 1 8 failing
    View these tests in the suite →
    • Scan — ProjectionExpression rejects ExclusiveStartKey that does not match the table schema
    • Scan — binary attribute comparisons Binary BEGINS_WITH
    • Scan — filter expression validation rejects begins_with with a non-string/binary operand
    • Scan — filter expression validation rejects redundant parentheses in FilterExpression
    • Scan — GSI scans a GSI with INCLUDE projection and returns only projected attributes
    • Scan — GSI pagination across tied sort keys walks every item once across a paged GSI scan with tied sort keys
    • Scan — LSI supports ConsistentRead on LSI scan
    • Scan — parallel parallel scan with segments returns disjoint results
  • updateTable Tier 1 6 failing
    View these tests in the suite →
    • UpdateTable — basic rejects PROVISIONED to PROVISIONED no-op with same throughput values
    • UpdateTable — validation rejects invalid throughput values (0 or negative)
    • UpdateTable — validation rejects PAY_PER_REQUEST with ProvisionedThroughput specified
    • UpdateTable — configuration parameters UpdateTable changes TableClass
    • UpdateTable — configuration parameters UpdateTable changes OnDemandThroughput
    • DeleteTable — deletion protection enforcement rejects DeleteTable while deletion protection is enabled
  • getItem Tier 1 5 failing
    View these tests in the suite →
    • Nested attribute projection GetItem ProjectionExpression with nested map path returns only the nested value
    • Nested attribute projection GetItem ProjectionExpression with list index returns the element
    • Nested attribute projection Query ProjectionExpression with nested path and list index
    • Nested attribute projection BatchGetItem ProjectionExpression with nested path
    • Nested attribute projection GetItem ProjectionExpression with multiple sibling paths in one map keeps all of them
  • updateItem Tier 1 5 failing
    View these tests in the suite →
    • UpdateItem — SET evaluation semantics a second SET clause reads the pre-update value of another attribute
    • UpdateItem — ReturnValues granularity UPDATED_NEW on a nested SET returns only the changed fragment
    • UpdateItem — ReturnValues granularity REMOVE with UPDATED_NEW omits Attributes (nothing was set to a new value)
    • UpdateItem — nested path semantics SET fails when intermediate map path does not exist
    • UpdateItem — validation rejects update that modifies the hash key
  • batchGetItem Tier 1 4 failing
    View these tests in the suite →
    • BatchGetItem — validation rejects more than 100 keys
    • BatchGetItem — validation rejects empty RequestItems
    • BatchGetItem — validation rejects reads from a non-existent table
    • BatchGetItem — validation rejects duplicate keys in the same batch
  • batchWriteItem Tier 1 3 failing
    View these tests in the suite →
    • BatchWriteItem — validation rejects more than 25 items
    • BatchWriteItem — validation rejects empty RequestItems
    • BatchWriteItem — validation rejects duplicate keys in the same table batch
  • backups Tier 2 3 skipped
    View these tests in the suite →
    • On-demand backups — lifecycle and restore CreateBackup → DescribeBackup → ListBackups → DeleteBackup
    • On-demand backups — lifecycle and restore RestoreTableFromBackup initiates a restore into a new table
    • On-demand backups — lifecycle and restore DescribeBackup on a deleted backup throws BackupNotFoundException
  • updateTable Tier 2 3 failing
    View these tests in the suite →
    • UpdateTable — GSI validation rejects adding a GSI with a name that already exists
    • UpdateTable — GSI validation rejects adding a GSI with an attribute not in AttributeDefinitions
    • UpdateTable — GSI validation rejects removing a non-existent GSI
  • contributorInsights Tier 2 unsupported
    View these tests in the suite →
    • Contributor insights — enable/describe/list reports DISABLED by default
    • Contributor insights — enable/describe/list enabling transitions the status and lists the table
  • export Tier 2 unsupported
    View these tests in the suite →
    • Export and import — S3 ExportTableToPointInTime initiates an export and reports it
    • Export and import — S3 ImportTable ingests S3 data into a new table
  • resourcePolicy Tier 2 unsupported
    View these tests in the suite →
    • Resource policies — Put/Get/Delete GetResourcePolicy on a table with no policy throws PolicyNotFoundException
    • Resource policies — Put/Get/Delete Put then Get round-trips the policy, and Delete removes it
  • tags Tier 2 2 failing
    View these tests in the suite →
    • Tags — validation rejects TagResource with an invalid ARN format
    • Tags — validation rejects ListTagsOfResource with a non-existent ARN
  • deleteItem Tier 1 1 failing
    View these tests in the suite →
    • DeleteItem — ConditionExpression parens rejects redundant parentheses in ConditionExpression
  • account Tier 2 1 skipped
    View these tests in the suite →
    • Account reads — DescribeLimits, DescribeEndpoints DescribeLimits returns positive account and table capacity limits
  • ttl Tier 2 1 failing
    View these tests in the suite →
    • TTL — validation rejects empty attribute name

Run history

Run Total Movement

Suite on

Grew to 684 tests with a control-plane and table-configuration sweep: the CreateTable/UpdateTable config parameters and the secondary control-plane operations (limits, backups and PITR, exports and imports, Kinesis, resource policies, contributor insights), each characterised against real AWS and probe-skipped where a target doesn't implement it.

The published percentage changed with it. It now measures correctness over implemented operations, Pass / (Pass + Fail), so skips no longer count against the score. A skip is honest scope; a fail is a bug.

 		65.1% -0.4pp fell 0.4 percentage points
65.5% -0.6pp fell 0.6 percentage points

Suite on

Grew to 625 tests, up 24 on the previous run: eleven more in Tier 1 and thirteen more in Tier 3, tightening coverage of core operations and the strict edge cases.

 		66.1% -2.3pp fell 2.3 percentage points

Suite on

ExtendDB joined the run as a target. The suite itself held steady at 601 tests.

 		68.4% +6.0pp rose 6.0 percentage points

Suite on

Grew to 601 tests, up 29, and every new test landed in Tier 3: a broader strict surface spanning validation ordering, exact error messages, service limits, and the legacy request shapes. Floci and Ministack were added to the run as targets the same day.

 		62.4% first run