Python · AWS SDK · Object storage reference

boto3 · s3

boto3 1.43.x  /  botocore  /  s3transfer  —  verified July 2026

S3 is a flat key→bytes store behind an HTTP API. boto3 gives you three doors into it: a Session that resolves credentials, a Client that maps 1:1 onto the S3 API, and a Resource that wraps the same calls in objects. On top sits a transfer manager that quietly turns one upload_file into many parallel part uploads. Everything below is one of those four things.

setup · session · config buckets & listing objects — the core transfers & multipart access · presign · crypto advanced & interop gotcha / changed ★most common

Verified against boto3 1.43.52 developer guide & S3 customization reference · Amazon S3 User Guide · boto/boto3 issue #4392 (S3 default integrity) · boto3 discussions #3563 (resources) & #4789 (2026 retries) · PyPI boto3.
Conventions — s3 = boto3.client('s3'). Bucket names use the AWS documentation placeholder style. Sizes are binary (1 MB = 1048576 B).

Mental model — where a call goes, and what it lands on
BAND A — ONE CALL, END TO END credential chain 1  explicit kwargs 2  env AWS_* 3  ~/.aws/ + SSO 4  IMDS / task role first hit wins Session creds + region + profile client('s3') 1:1 with the API · use this resource('s3') frozen — no new features botocore serialize → SigV4 sign checksum → retry parse → dict botocore, not boto3, does the work HTTPS Amazon S3 REST over HTTP · regional strongly read-after-write every op is a separate request transfer manager: upload_file / download_file / copy → fan out into N parallel part requests BAND B — WHAT IT LANDS ON Bucket globally unique name lives in ONE region policy · versioning lifecycle · BPA Key — one flat string "logs/2026/07/app.log" the slashes are just characters there are no directories Object Body (bytes) · ContentType Metadata · Tags · ETag StorageClass · VersionId immutable — a PUT replaces it the one query that fakes folders list_objects_v2(Bucket=b,   Prefix='logs/', Delimiter='/') → Contents  = keys at this level → CommonPrefixes = "subfolders"
Quickstart — the twelve lines you actually retype
import boto3
from botocore.exceptions import ClientError

s3 = boto3.client("s3", region_name="ap-south-1")          # one client, reuse it — thread-safe

s3.upload_file("report.pdf", "my-bucket", "docs/report.pdf")  # auto-multipart above 8 MB
s3.put_object(Bucket="my-bucket", Key="hi.txt", Body=b"hello")   # in-memory bytes

body = s3.get_object(Bucket="my-bucket", Key="hi.txt")["Body"]   # StreamingBody
with body as f: data = f.read()                              # read once, then close

for page in s3.get_paginator("list_objects_v2").paginate(       # NEVER call list_objects_v2 raw
        Bucket="my-bucket", Prefix="docs/"):
    for obj in page.get("Contents", []):                        # .get() — key is ABSENT if empty
        print(obj["Key"], obj["Size"])
Part I Getting a client credentials → session → client · get this wrong and nothing else matters
01Install & import

boto3 pulls botocore (the engine) and s3transfer (the transfer manager) with it.

  • pip install boto3Ships botocore + s3transfer + jmespath. Python 3.10+ as of 2026 — 3.9 support ended 29 Apr 2026.
  • pip install "boto3[crt]"Adds awscrt, the C-based transfer client. Faster multipart on fat pipes; opted into automatically (see card 15).
  • pip install boto3-stubs[s3] mypy-boto3-s3Type hints & editor completion. boto3 itself is dynamically generated, so IDEs are blind without these.
  • import boto3 from botocore.exceptions import ClientError from botocore.config import ConfigThe three imports every real script needs. Exceptions and config live in botocore, not boto3.
  • boto3.__version__; boto3.set_stream_logger('botocore')Log every request/response for debugging. Leaks credentials & payloads — never in production.
02Credentials — the provider chain ★

Never hardcode keys. boto3 searches these in order and stops at the first hit.

  • boto3.client('s3', aws_access_key_id=..., aws_secret_access_key=...)1 — explicit kwargs. Highest priority. Use only for tests and third-party endpoints.
  • AWS_ACCESS_KEY_ID / AWS_SECRET_ACCESS_KEY / AWS_SESSION_TOKEN2 — environment. Also AWS_DEFAULT_REGION, AWS_PROFILE, AWS_ENDPOINT_URL_S3.
  • ~/.aws/credentials  ·  ~/.aws/config3 — shared files, written by aws configure / aws sso login. Profiles, role_arn, source_profile and SSO all resolve here.
  • # nothing set → container / instance metadata4 — IAM role from ECS task metadata, EKS web identity, or EC2 IMDSv2. This is the right answer in production — rotating, no secrets on disk.
  • boto3.Session().get_credentials().get_frozen_credentials()Which identity did I actually get? Pair with boto3.client('sts').get_caller_identity() for a definitive answer.
03Session — profiles & assumed roles

A Session holds credentials + region; clients are cheap views onto it.

  • sess = boto3.Session(profile_name='prod', region_name='ap-south-1') s3 = sess.client('s3')Explicit sessions beat the implicit default one. Required for multi-account or multi-region scripts.
  • boto3.setup_default_session(region_name='eu-west-1')Sets the module-level default used by bare boto3.client(...).
  • c = boto3.client('sts').assume_role(RoleArn=arn, RoleSessionName='x')['Credentials'] s3 = boto3.client('s3', aws_access_key_id=c['AccessKeyId'], aws_secret_access_key=c['SecretAccessKey'], aws_session_token=c['SessionToken'])Cross-account access. These credentials expire (default 1 h) — long jobs must refresh.
  • sess.get_available_regions('s3')  ·  sess.region_nameIntrospection without a network call.
  • Threading: client objects are thread-safe and should be shared. Sessions and resources are not — build one Session per thread.trap
04Client vs Resource — pick client ★

Both do the same HTTP. Only one is still getting features.

  • s3 = boto3.client('s3') s3.put_object(Bucket=b, Key=k, Body=data)Client — one method per S3 API operation, CapitalCase kwargs, plain dicts back. Complete and current.
  • r = boto3.resource('s3') r.Object(b, k).put(Body=data) r.Bucket(b).objects.filter(Prefix='logs/').delete()Resource — nicer for iteration and bulk delete, but AWS states it will get no new features; it stays functional for boto3's lifetime.frozen
  • r.meta.client.generate_presigned_url(...)Escape hatch: every resource carries the underlying client at .meta.client.
  • Rule of thumb — write client code; reach for a resource only for bucket.objects.filter(...).delete(), which has no one-line client equivalent.
  • Same transfer methods on both: upload_file/download_file are injected into Client, Bucket and Object identically. No performance difference.
05Client config — endpoints, retries, timeouts

Everything below is botocore.config.Config, passed as config=.

  • cfg = Config(region_name='ap-south-1', signature_version='s3v4', s3={'addressing_style': 'virtual'})AWS's recommended baseline for presigned URLs. 'path' style is what most S3-compatible stores need instead.
  • boto3.client('s3', endpoint_url='https://minio.local:9000')Point at MinIO, Ceph, R2, LocalStack, GCS. Usually pair with path addressing and see card 27.
  • Config(retries={'max_attempts': 10, 'mode': 'adaptive'})Modes: legacy (today's default) · standard · adaptive (client-side rate limiting). max_attempts counts the first try.
  • # AWS_NEW_RETRIES_2026=true → standard mode + retry quotaAWS is switching the default from legacy to standard, no sooner than Nov 2026. Shorter backoff, fewer attempts, a quota that fails fast during outages. Pin 'mode': 'legacy' to opt out.2026
  • Config(connect_timeout=5, read_timeout=60, max_pool_connections=50)Default pool is 10 — raise it when running transfers with high max_concurrency or you will serialise on the pool.
  • Config(user_agent_extra='my-app/1.2')Shows up in CloudTrail — cheap request attribution.
Part II Buckets, keys and objects the flat namespace · list → put → get → copy → delete
06Buckets — create, list, delete

Names are globally unique, DNS-shaped, and permanently tied to one region.

  • s3.list_buckets()['Buckets']Returns Name + CreationDate for every bucket in the account, regardless of region.
  • s3.create_bucket(Bucket='amzn-s3-demo-bucket', CreateBucketConfiguration={'LocationConstraint': 'ap-south-1'})Every region except us-east-1 requires this block.
  • s3.create_bucket(Bucket='amzn-s3-demo-bucket') # us-east-1 ONLYPassing LocationConstraint='us-east-1' raises InvalidLocationConstraint. The asymmetry is historical and permanent.trap
  • s3.get_bucket_location(Bucket=b)['LocationConstraint']Returns None for us-east-1 — same asymmetry, mirrored.
  • s3.head_bucket(Bucket=b)Exists-and-I-can-reach-it check. 404 = missing, 403 = exists but not yours, 301 = wrong region.
  • s3.delete_bucket(Bucket=b)Fails unless completely empty — including every noncurrent version, delete marker and unfinished multipart upload.
07Listing objects — always paginate ★

The single most common boto3 bug lives in this card.

  • pages = s3.get_paginator('list_objects_v2').paginate( Bucket=b, Prefix='logs/2026/') for page in pages: for o in page.get('Contents', []): print(o['Key'], o['Size'])The canonical loop. Handles continuation tokens for you and has no key limit.
  • resp = s3.list_objects_v2(Bucket=b) # max 1000 keys, silentlyA raw call caps at 1000 and sets IsTruncated=True. Code that ignores it works in dev and loses data in prod.trap
  • resp.get('Contents', [])When nothing matches, 'Contents' is absent entirely — not an empty list. resp['Contents'] raises KeyError.trap
  • .paginate(Bucket=b, PaginationConfig={'MaxItems': 500, 'PageSize': 100})MaxItems stops the whole iteration; PageSize only sizes each request. StartingToken resumes.
  • s3.list_object_versions(Bucket=b, Prefix=p)Versioned buckets: returns Versions and DeleteMarkers. Plain listing hides both.
  • for o in boto3.resource('s3').Bucket(b).objects.filter(Prefix=p):The resource collection auto-paginates too, and is the tersest form. Add .limit(n) or .page_size(n).
  • Cost note: listing is billed per request, ~1000 keys each. Scanning a 10 M-key bucket is 10 000 LIST calls — prefer S3 Inventory for recurring full scans.
08Prefixes, delimiters & fake folders

S3 has no directories. Delimiter is the illusion generator.

  • Prefix='logs/2026/'Plain string match on the key. Recursive by nature — returns everything below, at every depth.
  • Prefix='logs/', Delimiter='/'Groups everything past the next / into CommonPrefixes. This is exactly what the console shows as folders.
  • [p['Prefix'] for p in page.get('CommonPrefixes', [])]→ ['logs/2025/', 'logs/2026/']. Same absent-key caveat as Contents.
  • paginator.paginate(Bucket=b, Prefix=p, Delimiter='/').search('CommonPrefixes[].Prefix')JMESPath straight off the paginator — a flat generator of strings.
  • "Create a folder" = put a zero-byte object whose key ends in /. Purely cosmetic; nothing requires it.
  • "Delete a folder" = list every key under the prefix and delete them all (card 13). There is no recursive delete API.trap
  • Performance: S3 scales per prefix. Random or hashed leading characters spread load; 2026-07-21-... keys concentrate it.
09Put an object ★

For bytes you already hold. For files on disk, use card 14 instead.

  • s3.put_object(Bucket=b, Key='hi.txt', Body=b'hello')Body takes bytes, str, or any file-like object. Single request, 5 GB hard ceiling.
  • s3.put_object(..., Body=json.dumps(d).encode(), ContentType='application/json')S3 never sniffs content type — unset means binary/octet-stream, and browsers will download instead of render.trap
  • Metadata={'author': 'kalyan', 'run-id': '42'}User metadata: ASCII string values only, 2 KB total, keys come back lowercased. Immutable without a copy.
  • ContentEncoding='gzip'  CacheControl='max-age=3600'  ContentDisposition='attachment'Standard HTTP headers stored with the object and replayed on GET.
  • StorageClass='INTELLIGENT_TIERING'Also STANDARD · STANDARD_IA · ONEZONE_IA · GLACIER_IR · GLACIER · DEEP_ARCHIVE · EXPRESS_ONEZONE.
  • IfNoneMatch='*'Conditional write: fails with PreconditionFailed if the key exists. The atomic way to avoid overwrites — a plain PUT always clobbers.
  • Tagging='env=prod&team=data'URL-encoded on put; a dict-of-dicts everywhere else. Up to 10 tags, and they drive lifecycle rules and IAM conditions.
10Get an object ★

Body is a live network stream, not a buffer.

  • data = s3.get_object(Bucket=b, Key=k)['Body'].read()The one-liner. Fine below a few hundred MB; it materialises everything in RAM.
  • with s3.get_object(Bucket=b, Key=k)['Body'] as body: for chunk in body.iter_chunks(1024*1024): process(chunk)StreamingBody is a context manager. Not closing it leaks a pooled connection, and enough leaks will hang your client.trap
  • body.iter_lines()  ·  body.read(amt)Line-wise for text logs; sized reads for framed formats. The stream is one-shot — you cannot seek or re-read it.
  • s3.get_object(Bucket=b, Key=k, Range='bytes=0-1023')Byte-range GET. Reads a Parquet footer or a file header without paying for the whole object.
  • VersionId='3sL4kqtJl...'Reads a specific version. Omit it and you get the current one.
  • IfNoneMatch=etag  ·  IfModifiedSince=dtConditional GET → 304 Not Modified as a ClientError. Cheap cache revalidation.
  • r = s3.get_object(...); r['ContentLength'], r['ETag'], r['Metadata']The response dict carries all headers alongside Body.
11Head, exists & metadata

A HEAD costs one request and transfers no body.

  • m = s3.head_object(Bucket=b, Key=k) m['ContentLength'], m['LastModified'], m['ETag'], m['StorageClass']Size, mtime and metadata without downloading.
  • try: s3.head_object(Bucket=b, Key=k); exists = True except ClientError as e: if e.response['Error']['Code'] == '404': exists = False else: raiseThe idiomatic exists-check. head_object raises code '404' while get_object raises 'NoSuchKey' — HEAD has no response body to carry the real code.trap
  • A 403 here can also mean the object exists but you lack s3:GetObject. Absence and denial are indistinguishable without s3:ListBucket.
  • m['ETag'].strip('"')ETag arrives quoted. It equals the MD5 only for single-part uploads; multipart ETags end in -N.trap
  • s3.get_object_attributes(Bucket=b, Key=k, ObjectAttributes=['ObjectSize','Checksum','ObjectParts'])The modern replacement: real checksums and per-part sizes, no ETag guesswork.
  • s3.get_waiter('object_exists').wait(Bucket=b, Key=k)Polls HEAD until it appears (20 × 5 s default). Also object_not_exists, bucket_exists.
12Copy, move & rename

S3 has no rename. Every move is copy-then-delete, server-side.

  • s3.copy_object(Bucket=dst, Key=dk, CopySource={'Bucket': src, 'Key': sk})Server-side — no bytes cross your machine. 5 GB limit per call.
  • s3.copy(CopySource={'Bucket': src, 'Key': sk}, Bucket=dst, Key=dk)The managed copy: transparently switches to multipart copy above the threshold, so it handles objects up to 5 TB. Prefer this.use this
  • CopySource={..., 'VersionId': v}Copying an old version forward is how you "restore" it in a versioned bucket.
  • MetadataDirective='REPLACE', Metadata={...}, ContentType='text/csv'Default is COPY — metadata is inherited and your new values are silently ignored. This is the only way to edit metadata in place: copy onto itself.trap
  • TaggingDirective='REPLACE'  ·  StorageClass='GLACIER_IR'Same pattern for tags; copy-onto-itself is also the way to change storage class outside lifecycle rules.
  • s3.copy(...); s3.delete_object(Bucket=src, Key=sk)Move = these two lines. Not atomic — verify the copy landed before deleting.
13Delete — single & batch

Deletes are cheap, fast, and quietly partial.

  • s3.delete_object(Bucket=b, Key=k)Succeeds with 204 even if the key never existed — deletes are idempotent, so this is not an existence test.
  • s3.delete_objects(Bucket=b, Delete={ 'Objects': [{'Key': k} for k in batch], 'Quiet': True})Batch delete, max 1000 keys per call. Roughly 1000× cheaper than a loop.
  • resp.get('Errors', [])A 200 does not mean everything was deleted. Per-key failures come back in Errors and are trivially missed.trap
  • boto3.resource('s3').Bucket(b).objects.filter(Prefix='tmp/').delete()Delete a whole prefix in one expression — it pages and batches internally. The one place the resource API clearly wins.
  • s3.delete_object(Bucket=b, Key=k, VersionId=v)On a versioned bucket, a delete without VersionId only writes a delete marker; the data (and its bill) stays. With VersionId it is permanent.trap
  • bkt.object_versions.all().delete()Empties a versioned bucket for real — versions and delete markers. Required before delete_bucket.
Part III Transfers at scale s3transfer · where one call quietly becomes many
14upload_file & download_file ★

Managed transfers: retries, threading and multipart, handled for you.

  • s3.upload_file('report.pdf', b, 'docs/report.pdf')Positional order is (Filename, Bucket, Key) — the odd one out; every other method is keyword-first.
  • s3.download_file(b, 'docs/report.pdf', 'report.pdf')Order flips to (Bucket, Key, Filename). Writes to a temp file and renames on success.
  • with open('f.bin', 'rb') as f: s3.upload_fileobj(f, b, k) with open('f.bin', 'wb') as f: s3.download_fileobj(b, k, f)Any file-like object — io.BytesIO, a socket, a pipe. Must be opened in binary mode.
  • ExtraArgs={'ContentType': 'application/pdf', 'ServerSideEncryption': 'aws:kms', 'Metadata': {'run': '42'}}The only way to set headers on a managed transfer. Allowed keys are fixed by S3Transfer.ALLOWED_UPLOAD_ARGS — anything else raises ValueError.
  • S3Transfer.ALLOWED_DOWNLOAD_ARGSMuch shorter: ChecksumMode, VersionId, the three SSECustomer*, RequestPayer, ExpectedBucketOwner.
  • Why prefer these over put_object: no 5 GB ceiling, parallel parts, resumable retries per part, and constant memory regardless of file size.
  • Identical methods exist on Bucket and Object resources. No behavioural difference — only the argument capitalisation changes.
15TransferConfig — the real defaults

Verified from boto3.s3.transfer.TransferConfig.DEFAULTS.

  • from boto3.s3.transfer import TransferConfig cfg = TransferConfig(multipart_threshold=64*1024**2) s3.upload_file(fn, b, k, Config=cfg)Passed as Config= (capital C) to any managed transfer.
  • multipart_threshold = 8388608 # 8 MB multipart_chunksize = 8388608 # 8 MBAbove the threshold the transfer becomes multipart. Raising the chunk size is the main knob for big files — S3 allows at most 10 000 parts, so 8 MB parts cap you around 80 GB.
  • max_concurrency = 10Worker threads per transfer. Raise for fat pipes, lower to be polite — and raise max_pool_connections to match, or you will just queue.
  • use_threads = TrueSet False in Lambda or any single-core, memory-tight context; max_concurrency is then ignored.
  • max_bandwidth = None # bytes/sec, e.g. 5*1024**2Throttle so a backup job does not starve everything else on the box.
  • num_download_attempts = 5 · max_io_queue = 100 · io_chunksize = 262144Download-side retry count and the write-queue depth. Memory ceiling ≈ max_io_queue × io_chunksize (≈ 25 MB).
  • preferred_transfer_client = 'auto' # 'classic' | 'crt'auto upgrades to the C-based CRT transfer manager when awscrt is installed and the environment supports it. CRT ignores use_threads, max_bandwidth, io_chunksize, max_io_queue and num_download_attempts — pin 'classic' if you depend on those.watch
16Progress callbacks

A callable invoked with bytes-transferred-since-last-call.

  • def cb(n): print(n) # n = DELTA, not cumulative s3.upload_file(fn, b, k, Callback=cb)Accumulate yourself. Misreading the delta as a total is the classic bug.trap
  • class Progress: def __init__(self, total): self.total, self.seen = total, 0 def __call__(self, n): self.seen += n print(f"{self.seen/self.total:.1%}", end="\r")The stateful pattern. Get total from os.path.getsize() on upload, or head_object's ContentLength on download.
  • bar = tqdm(total=size, unit='B', unit_scale=True) s3.download_file(b, k, fn, Callback=bar.update)tqdm.update already takes a delta — it plugs straight in with no wrapper.
  • Callbacks fire from worker threads and out of order. Guard shared state with a Lock; never do slow I/O inside one.
  • Available on all four managed methods plus copy(). Not available on put_object/get_object — wrap the stream yourself there.
17Manual multipart API

Only when you need control the transfer manager will not give you.

  • u = s3.create_multipart_upload(Bucket=b, Key=k) uid = u['UploadId']Opens the upload. Nothing is visible at Key until you complete it.
  • p = s3.upload_part(Bucket=b, Key=k, UploadId=uid, PartNumber=i, Body=chunk) parts.append({'PartNumber': i, 'ETag': p['ETag']})Parts are 1-indexed. Minimum 5 MB each except the last; maximum 10 000 parts, 5 GB per part.
  • s3.complete_multipart_upload(Bucket=b, Key=k, UploadId=uid, MultipartUpload={'Parts': sorted(parts, key=lambda p: p['PartNumber'])})Parts must be in ascending order or S3 rejects the manifest.
  • s3.abort_multipart_upload(Bucket=b, Key=k, UploadId=uid)Always in a try/finally. Orphaned parts are invisible to listing and billed forever.costs money
  • s3.list_multipart_uploads(Bucket=b)Find the orphans. Then add a lifecycle rule with AbortIncompleteMultipartUpload: {DaysAfterInitiation: 7} to every bucket — the single highest-value S3 hygiene rule.
  • s3.upload_part_copy(..., CopySource=..., CopySourceRange='bytes=0-...')Server-side multipart copy — how s3.copy() beats the 5 GB copy_object limit internally.
Part IV Access, safety and the edges presign · encrypt · govern · iterate · and the things that break
18Presigned URLs ★

Hand out time-limited access without handing out credentials.

  • url = s3.generate_presigned_url('get_object', Params={'Bucket': b, 'Key': k}, ExpiresIn=3600)Pure local computation — no network call, and it does not check that the object exists.
  • s3 = boto3.client('s3', region_name=r, config=Config( signature_version='s3v4', s3={'addressing_style': 'virtual'}))AWS's documented recommendation. Signing with the wrong region produces a URL that 400s on use.trap
  • .generate_presigned_url('put_object', Params={'Bucket': b, 'Key': k, 'ContentType': 'image/png'}, ExpiresIn=900)Browser then does a plain PUT. Every signed Params entry becomes a required header on the request.
  • Expiry is a min, not a max. SigV4 caps at 7 days, and the URL also dies the moment the signing credentials expire — a Lambda role signing a 24 h URL yields one valid for minutes.trap
  • The URL inherits the signer's permissions, and anyone holding it can use it. Treat it as a bearer token: short TTL, HTTPS only, never in a log or a query-string referrer.
  • .generate_presigned_url('head_object'|'delete_object'|'list_objects_v2', ...)Most client methods can be presigned, not just GET/PUT.
19Presigned POST — browser uploads

A form POST with a signed policy. Constrainable in ways a PUT URL is not.

  • r = s3.generate_presigned_post(b, k, ExpiresIn=3600) r['url'], r['fields']Returns a form action plus hidden fields (key, Policy, X-Amz-Signature…). Post them all, with file last.
  • Conditions=[['content-length-range', 0, 5*1024**2], {'Content-Type': 'image/png'}]The reason to choose POST: you can cap the upload size server-side. A presigned PUT cannot — a client may send 5 TB.
  • Fields={'acl': 'private', 'Content-Type': 'image/png'}Prefilled values. Anything in Fields must also appear in Conditions, or S3 rejects the policy.trap
  • Key='uploads/${filename}' + ['starts-with', '$key', 'uploads/']Let the client pick the filename inside a prefix you control.
  • requests.post(r['url'], data=r['fields'], files={'file': fh})Success is HTTP 204 with an empty body, not 200. Checking for 200 will look like failure.trap
20Encryption — SSE-S3, KMS, SSE-C

Server-side encryption is on by default; the choice is who holds the key.

  • Default since 2023: every new object is encrypted with SSE-S3 (AES256) at no cost. You only act to upgrade or to override.
  • ServerSideEncryption='AES256'SSE-S3 explicitly. AWS holds and rotates the key; nothing to configure.
  • ServerSideEncryption='aws:kms', SSEKMSKeyId='arn:aws:kms:...'SSE-KMS — auditable in CloudTrail, revocable via key policy. Readers need kms:Decrypt as well as s3:GetObject; a missing KMS grant surfaces as a confusing AccessDenied.trap
  • BucketKeyEnabled=TrueCuts KMS request cost by up to 99% on high-volume buckets. Effectively free; enable it at the bucket level.
  • SSECustomerAlgorithm='AES256', SSECustomerKey=key_bytesSSE-C — you supply the key on every request, put and get. Lose it and the data is gone; AWS stores only an HMAC.
  • s3.put_bucket_encryption(Bucket=b, ServerSideEncryptionConfiguration={...})Set the bucket default once instead of tagging every call — and add a bucket policy denying puts without it.
21Access control & public exposure

Modern S3: policies for access, ACLs disabled, public access blocked.

  • s3.put_public_access_block(Bucket=b, PublicAccessBlockConfiguration={ 'BlockPublicAcls': True, 'IgnorePublicAcls': True, 'BlockPublicPolicy': True, 'RestrictPublicBuckets': True})All four are ON by default for buckets created since Apr 2023. This is why your "public" bucket 403s.
  • s3.put_bucket_policy(Bucket=b, Policy=json.dumps(policy))Resource ARNs: arn:aws:s3:::bucket for bucket-level actions (ListBucket), arn:aws:s3:::bucket/* for object-level. Mixing them up is the #1 policy bug.trap
  • s3.get_bucket_policy_status(Bucket=b)['PolicyStatus']['IsPublic']A definitive yes/no on whether the bucket is publicly readable.
  • ACL='public-read'Usually a no-op now. New buckets default to BucketOwnerEnforced object ownership, which disables ACLs entirely; passing one raises AccessControlListNotSupported.legacy
  • ExpectedBucketOwner='123456789012'Fails the call if the bucket changed hands. Cheap defence against bucket-sniping and typo'd names.
  • s3.put_bucket_cors(...)  ·  s3.put_bucket_website(...)CORS is required before any browser fetch/XHR against presigned URLs, including uploads.
22Versioning, lifecycle & tags

Bucket-level policy that quietly governs cost and recoverability.

  • s3.put_bucket_versioning(Bucket=b, VersioningConfiguration={'Status': 'Enabled'})Once enabled it can only be suspended, never removed. Existing objects get VersionId='null'.
  • s3.put_bucket_lifecycle_configuration(Bucket=b, LifecycleConfiguration={'Rules': [...]})Whole-config replace, not a merge — always get, mutate, then put.trap
  • 'AbortIncompleteMultipartUpload': {'DaysAfterInitiation': 7}The rule every bucket should have. Sweeps up orphaned parts you cannot even see.
  • 'Transitions': [{'Days': 30, 'StorageClass': 'STANDARD_IA'}] 'NoncurrentVersionExpiration': {'NoncurrentDays': 90}Tiering plus version pruning — where versioned buckets stop growing without bound.
  • s3.put_object_tagging(Bucket=b, Key=k, Tagging={'TagSet': [{'Key': 'env', 'Value': 'prod'}]})Also a full replace. Tags drive lifecycle filters, cost allocation and IAM conditions.
  • s3.put_object_lock_configuration(...)  ·  ObjectLockMode='COMPLIANCE'WORM retention. COMPLIANCE cannot be shortened by anyone, including the root account — test with GOVERNANCE first.irreversible
23Paginators, waiters & JMESPath

Three generic botocore facilities that make S3 code short.

  • s3.can_paginate('list_objects_v2')Paginators exist for list_objects_v2, list_object_versions, list_multipart_uploads, list_parts.
  • it = paginator.paginate(Bucket=b, Prefix=p) keys = it.search('Contents[].Key').search() applies JMESPath across pages and yields a flat generator — no nested loop, no .get() dance.
  • it.search("Contents[?Size > `1048576`].[Key,Size]")Server returns everything; JMESPath filters client-side. Backticks are JMESPath literals.
  • it.search("sort_by(Contents, &LastModified)[-1].Key")Newest key in a prefix. Note S3 returns keys in lexicographic order, never by date — sorting is always yours to do.
  • s3.get_waiter('bucket_exists').wait(Bucket=b, WaiterConfig={'Delay': 2, 'MaxAttempts': 30})Four S3 waiters: bucket_exists, bucket_not_exists, object_exists, object_not_exists. They raise WaiterError on timeout.
  • s3.meta.events.register('before-send.s3', hook)The extensibility hook — inject headers, log, or stub requests in tests without monkeypatching.
24Interop — pandas, s3fs, wrangler

Most data work should not call boto3 directly at all.

  • df = pd.read_parquet('s3://bucket/path/part-*.parquet') df.to_csv('s3://bucket/out.csv', index=False)pandas speaks s3:// natively via s3fs. Credentials resolve through the same boto3 chain.
  • storage_options={'profile': 'prod', 'client_kwargs': {'endpoint_url': 'https://minio:9000'}}How to pass profile, region or a custom endpoint through pandas / Dask / pyarrow.
  • import s3fs; fs = s3fs.S3FileSystem() fs.ls('bucket/prefix'); fs.glob('bucket/**/*.json') with fs.open('bucket/k.txt', 'rb') as f: f.read()A filesystem facade with glob and caching. Beware: it caches directory listings, so a fresh key may be invisible — call fs.invalidate_cache().trap
  • import awswrangler as wr wr.s3.to_parquet(df, path='s3://b/t/', dataset=True, partition_cols=['dt'], database='db', table='t')AWS SDK for pandas — partitioned datasets plus Glue catalog registration in one call.
  • smart_open.open('s3://b/big.gz', 'rt')Streams and transparently decompresses at constant memory. Ideal for line-by-line log processing.
  • from moto import mock_awsThe standard way to unit-test S3 code with no network and no bucket.
25The wider S3 API family

's3' is one of six S3 clients boto3 exposes.

  • boto3.client('s3')General purpose buckets and objects — everything above.
  • boto3.client('s3control')Account-level: S3 Batch Operations jobs, Access Points, Storage Lens, account-wide public access block.
  • s3.create_bucket(Bucket='my-data--use1-az4--x-s3', CreateBucketConfiguration={ 'Location': {'Type': 'AvailabilityZone', 'Name': 'use1-az4'}, 'Bucket': {'Type': 'Directory', 'DataRedundancy': 'SingleAvailabilityZone'}})S3 Express One Zone directory bucket — single-digit-ms latency, one AZ. Made on the ordinary 's3' client; the name suffix --<zone-id>--x-s3 is mandatory.express
  • s3.create_session(Bucket=dirbucket, SessionMode='ReadWrite')Directory buckets authenticate once per session rather than per request — that is where the latency win comes from. boto3 handles it for you; you need s3express:CreateSession.
  • boto3.client('s3tables')  ·  boto3.client('s3vectors')Managed Apache Iceberg tables, and native vector storage/query for embeddings. Separate services with their own APIs.new
  • Also 's3outposts' and 's3files'. Directory buckets support a reduced API surface — no ACLs, no versioning, no lifecycle transitions.
26Errors & exceptions

Almost everything is a ClientError carrying a string code.

  • except ClientError as e: code = e.response['Error']['Code'] status = e.response['ResponseMetadata']['HTTPStatusCode']The universal handler. e.response also holds RequestId and HostId — both required if you open an AWS support case.
  • except s3.exceptions.NoSuchKey:Modelled exceptions hang off the client instance, not the module: NoSuchBucket, NoSuchKey, BucketAlreadyOwnedByYou, InvalidObjectState.
  • NoSuchKey vs '404': get_object raises the named exception; head_object can only raise a bare ClientError with code '404', because HEAD has no body to carry the code. Handle both.trap
  • 'AccessDenied' · 'InvalidAccessKeyId' · 'SignatureDoesNotMatch' · 'ExpiredToken'SignatureDoesNotMatch usually means clock skew or a wrong region, not a wrong secret.
  • 'SlowDown' (503) · 'RequestTimeout' · 'InternalError' (500)Retryable. botocore already backs these off — do not add your own retry loop on top, or you will multiply the load.
  • from botocore.exceptions import EndpointConnectionError, \ NoCredentialsError, ParamValidationErrorClient-side failures that never reach AWS — these are not ClientError subclasses and a bare except ClientError will miss them.trap
  • 'PermanentRedirect' / 301Right bucket, wrong region client. Recreate the client with the region from get_bucket_location.
27Data integrity & the checksum change

The change that broke a lot of S3-compatible code in 2025.

  • Since boto3 1.36.0 (Jan 2025) the SDK computes an extra CRC32 checksum on every upload and validates checksums on download, by default.changed
  • Where it bites: operations that previously sent Content-MD5 — notably delete_objects — now send CRC32 instead. MinIO, GCS, Ceph and older emulators that only accept MD5 start failing.trap
  • AWS_REQUEST_CHECKSUM_CALCULATION=when_required AWS_RESPONSE_CHECKSUM_VALIDATION=when_requiredThe escape hatch, as environment variables. Same names in ~/.aws/config, or on the client:
  • cfg = Config(request_checksum_calculation='when_required', response_checksum_validation='when_required')Scope it to the one client talking to a non-AWS endpoint — keep full protection against real S3.
  • ChecksumAlgorithm='SHA256'  ·  ChecksumMode='ENABLED'Opt into a stronger algorithm on upload; ask for validation on download. Available: CRC32, CRC32C, CRC64NVME, SHA1, SHA256.
  • s3.get_object_attributes(..., ObjectAttributes=['Checksum'])Read the stored whole-object checksum back — the correct integrity check, and the reason ETag is now obsolete for this purpose.
  • AWS's own framing: the SDKs are built for AWS services, and new defaults may ship before third-party implementations support them. Pin your boto3 version if you target an emulator.

Four things worth drawing

1 · There are no foldershow Delimiter='/' manufactures a directory tree
WHAT IS ACTUALLY STORED logs/2025/dec.log logs/2026/jul/app.log logs/2026/jul/err.log logs/README.md data/train.parquet five independent strings — no hierarchy anywhere list_objects_v2(   Prefix='logs/', Delimiter='/') the delimiter does all the work WHAT COMES BACK CommonPrefixes logs/2025/ logs/2026/ the console draws these as folder icons Contents logs/README.md only keys at this level — nothing deeper data/train.parquet is absent it never matched Prefix='logs/' Drop the Delimiter and you get all four logs/ keys, flat. Drop the Prefix and you get the whole bucket. Deleting "a folder" = deleting every key under the prefix.
2 · The multipart lifecyclewhat upload_file is doing above 8 MB
SIZE DECIDES, NOT YOU < 8 MB → one PUT multipart_threshold = 8 MB ≥ 8 MB → split into 8 MB parts, uploaded in parallel create_multipart → UploadId upload_part × N PartNumber=1 → ETag PartNumber=2 → ETag PartNumber=3 → ETag max_concurrency=10 threads min 5 MB each · max 10 000 parts each part retried independently complete_multipart object appears — atomically abort_multipart on any failure — use try/finally Neither completed nor aborted? The parts stay in the bucket, invisible to list_objects_v2, and you are billed for them forever. Find them: list_multipart_uploads(Bucket=b) Prevent them: lifecycle rule AbortIncompleteMultipartUpload → DaysAfterInitiation: 7
3 · Anatomy of a presigned URLeverything is in the query string — and it is a bearer token
https://my-bucket.s3.ap-south-1.amazonaws.com/docs/report.pdf host encodes bucket + region — virtual-hosted addressing ? then six signed parameters: X-Amz-Algorithm AWS4-HMAC-SHA256 — SigV4 X-Amz-Credential AKIA.../20260721/ap-south-1/s3/aws4_request X-Amz-Date 20260721T073000Z — clock skew kills it X-Amz-Expires 3600 — seconds, max 604800 (7 days) X-Amz-SignedHeaders host — each one becomes REQUIRED on use X-Amz-Signature HMAC over all of the above The URL expires at whichever comes FIRST: X-Amz-Expires  OR  the moment the signing credentials expire. A Lambda role signing a 7-day URL produces one valid for minutes. presigned URL (GET or PUT) one call, one method, no size limit cannot constrain what is uploaded presigned POST (form) url + fields dict, browser-native CAN cap size & type via Conditions
4 · Which method moves the bytessource and size pick the call, every time
WHERE ARE THE BYTES? in memory bytes / str / dict put_object(Body=...) one request · 5 GB ceiling a file on disk any size upload_file / download_file auto-multipart · no ceiling ★ a stream BytesIO, socket, pipe upload_fileobj binary mode only already in S3 copy or move copy( ) — managed copy_object caps at 5 GB a dataframe parquet / csv df.to_parquet('s3://...') via s3fs — skip boto3 ceilings to remember single PUT      5 GB one object      5 TB copy_object    5 GB parts per upload 10 000 min part size   5 MB keys per LIST   1 000 keys per DELETE 1 000 user metadata   2 KB presign lifetime 7 days 8 MB default parts × 10 000 ≈ 80 GB — past that, raise multipart_chunksize. Every managed method also takes ExtraArgs=, Callback= and Config=. put_object / get_object take none of the three.
Worth memorising — the fourteen that cost people a day
list_objects_v2 → 1000 keys
A raw call silently truncates and sets IsTruncated. Always get_paginator('list_objects_v2'). This bug passes every dev test and loses data in production.
resp.get('Contents', [])
When nothing matches, the key is absent, not empty. resp['Contents'] raises KeyError. Same for CommonPrefixes and Errors.
prefix ≠ folder
Keys are flat strings. There is no recursive delete, no rename, no mkdir. Deleting a "folder" means listing and deleting every key beneath it.
Body is a one-shot stream
StreamingBody can be read once and must be closed, or you leak a pooled connection. Use it as a context manager.
head 404 vs get NoSuchKey
get_object raises the named exception; head_object can only raise ClientError with code '404' — HEAD has no body to carry the real code. And a 403 may mean "exists, denied".
delete_objects 200 ≠ success
Max 1000 keys per call, and per-key failures come back in Errors. Check it, or you will report deletions that never happened.
boto3 ≥ 1.36 sends CRC32
Default integrity protections replaced Content-MD5 on some calls. Breaks MinIO / GCS / older emulators → set request_checksum_calculation and response_checksum_validation to when_required.
clients yes, resources no
Client objects are thread-safe and meant to be shared. Sessions and resources are not — build one per thread.
us-east-1 is the odd one
Creating a bucket there must omit CreateBucketConfiguration; every other region must include it. And get_bucket_location returns None for it.
ETag is not a checksum
It equals the MD5 only for single-part uploads. Anything multipart ends in -N. Use get_object_attributes(ObjectAttributes=['Checksum']) instead.
presign: first expiry wins
SigV4 caps at 7 days, but the URL also dies with the signing credentials. Signing with the wrong region produces a URL that 400s on use.
MetadataDirective='REPLACE'
Without it, copy_object inherits the source metadata and silently ignores yours. Copy-onto-itself is the only way to edit metadata in place.
aborted parts bill forever
Incomplete multipart uploads are invisible to listing and never expire on their own. Put AbortIncompleteMultipartUpload: 7 days on every bucket you own.
the resource API is frozen
AWS has said it will get no new features; it stays working for boto3's lifetime. Write client code, and keep resources for bucket.objects.filter(...).delete().