pip install tensorflow★CPU + GPU on Linux in one wheel since 2.16.pip install tensorflow[and-cuda]Pulls the matching CUDA libraries with it.tf.__version__ · keras.__version__★Check both. TF 2.16+ ships Keras 3; a stray Keras 2 in the env is the root of most import errors.tf.config.list_physical_devices("GPU")★Empty list means you are on CPU, whatever the wheel name says.pip install tensorboard2.21No longer a dependency as of 2.21 — install it yourself.Windows native GPUsince 2.11Dropped after 2.10. Use WSL2, or accept CPU.Python 3.92.21Support removed in 2.21. Floor is 3.10.
import tensorflow as tf★The one import you always need.import keras★Prefer this totf.keras. Same object on TF 2.16+, but it keeps you honest about which Keras you are on.from keras import layers, models, callbacksThe three you reach for constantly.tf.data · tf.image · tf.stringsInput pipeline, image ops, string ops.import tensorflow_hub as hubPart II. Still installable, frozen since Jan 2024.import keras_hubcurrentPart III. The successor, and where new presets land.tf.compat.v1The TF1 attic:Session,placeholder,get_variable. Read-only history.tf.estimator2.16Removed in 2.16. Any tutorial built on Estimators is dead code.
tf.constant([[1, 2], [3, 4]])★Immutable. Every op returns a new tensor.tf.zeros((2, 3)) · ones · fill(shape, v)Shape goes in as a tuple or list.tf.random.normal((2, 3)) · uniformAddseed=for repeatability.tf.range(0, 10, 2) · tf.linspace(0, 1, 5)Sequences.tf.convert_to_tensor(arr)★NumPy, lists, scalars — all become tensors.tf.eye(3) · tf.one_hot(idx, depth)Identity; index vector to one-hot matrix.tf.zeros_like(x) · ones_like(x)Match an existing shape and dtype.tf.random.set_seed(0)Global seed. Also seed Python and NumPy separately.
x.shape · x.dtype · x.ndim★The three questions. Most errors are one of them disagreeing.tf.shape(x)★The dynamic shape, as a tensor. Use this inside@tf.functionwherex.shapemay holdNone.tf.cast(x, tf.float32)★TF will not promote for you. Mixing int32 and float32 raises rather than silently casting.tf.float32 · tf.float16 · tf.bfloat16bfloat16 keeps float32 range — the safer half precision.tf.int32 · tf.int64 · tf.bool · tf.stringYes, string is a real dtype and tensors of them work.x.numpy()Out to NumPy. Fails inside a tracedtf.function.tf.experimental.numpy · x[..., tf.newaxis]NumPy-style API and axis insertion.InvalidArgumentError: cannot compute ... as input #1(zero-based) was expected to be a floatThe dtype-mismatch message. It means: you forgot atf.cast.
tf.reshape(x, (2, -1))★-1means "work it out". Only one per call.tf.transpose(x, perm=[2, 0, 1])Reorder axes.permlists the new order.tf.expand_dims(x, axis=0) · tf.squeeze(x)★Add or drop a length-1 axis — the batch dimension, usually.tf.concat([a, b], axis=0)★Joins along an existing axis. Rank stays the same.tf.stack([a, b], axis=0)Adds an axis. This is the pair people confuse.tf.split(x, 3, axis=1) · tf.unstack(x)Break apart into a list.tf.tile(x, [2, 1]) · tf.repeat(x, 3)Real copies, unlike broadcasting.keras.layers.Flatten() · Reshape(shape)The layer forms, for inside a model.
shapes align from the right★Trailing dims must match, or be 1, or be absent. That one rule explains every broadcast.(3,1) + (1,4) → (3,4)Length-1 axes stretch. Nothing is copied until it must be.tf.broadcast_to(x, shape)Make it explicit when you want the full tensor.tf.reduce_sum(x, axis=1, keepdims=True)Keeps rank so the result broadcasts back against the original.pred (N,1) vs label (N,)★famousThe expensive one. These broadcast to (N,N), so your loss quietly averages N² pairs. Nothing raises — the model just never learns.dtype does not broadcastShapes stretch; dtypes do not. int32 + float32 is an error, not a promotion.
x[0] · x[:, 1] · x[..., -1]Standard Python slicing works.tf.reduce_sum(x, axis=1) · reduce_mean · reduce_max★Note thereduce_prefix — TF spells these differently from NumPy.tf.argmax(x, axis=-1)★Turning logits into predicted classes.tf.gather(x, idx) · tf.gather_nd(x, idx)Pick rows by index tensor.tf.boolean_mask(x, mask) · tf.where(c, a, b)Filter by condition; elementwise choose.tf.top_k(x, k=5) · tf.sort · tf.uniqueReturns(values, indices).tf.reduce_any · reduce_all · math.count_nonzeroBoolean reductions.tf.tensor_scatter_nd_update(x, idx, v)Functional update — tensors are immutable, so you get a new one.
a + b · a * b · a ** 2Operators are elementwise and broadcast.a @ b · tf.matmul(a, b)★Matrix multiply. Batched automatically on leading dims.tf.einsum("bij,bjk->bik", a, b)When the index bookkeeping gets hard, write it down instead.tf.math.exp · log · sqrt · absMost elementwise maths lives undertf.math.tf.clip_by_value(x, lo, hi) · clip_by_normClamping, and gradient clipping.tf.nn.softmax(x, axis=-1) · relu · sigmoidRaw activations, outside of Keras layers.tf.linalg.inv · det · svd · solveThe linear algebra namespace.tf.math.is_nan(x) · tf.debugging.assert_all_finite(x, msg)★First thing to reach for when a loss goes NaN.
v = tf.Variable(3.0)★Tensors are immutable; Variables are not. Every weight in your model is one.v.assign(5.0) · v.assign_add(1.0)★Notv = 5.0— that rebinds the Python name and throws the Variable away.v.read_value() · v.numpy()Snapshot the current value.tf.Variable(x, trainable=False)For counters and running statistics the optimizer must not touch.model.trainable_variables★What you hand totape.gradientand to the optimizer.model.variables · model.non_trainable_variablesEverything, and the frozen remainder (BatchNorm statistics live here).creating a Variable inside tf.functiontracingRaises on the second call — tracing would create it twice. Build them outside, or guard with a flag.
@tf.function★Traces the Python once per input signature into a graph, then reuses it. This is the whole performance story.it traces, then it runs★Python side effects —print, appending to a list, incrementing a counter — happen only during tracing.tf.print(x)★Prints every call, because it becomes a graph op.print(x)does not.retracing on a new signatureA new shape or dtype means a new trace. Passing Python ints instead of tensors retraces for every value.@tf.function(input_signature=[tf.TensorSpec([None, 32])])Pin the signature to stop runaway retracing.@tf.function(jit_compile=True)XLAHand the graph to XLA for fusion. Often a large win, sometimes a compile-time cost.f.get_concrete_function(spec)The single traced graph, if you want to inspect or export it.a Python loop over a tensorGets unrolled into the graph. Usetf.while_loop, or vectorise.tf.config.run_functions_eagerly(True)Turns tracing off globally so you can use a debugger. Slow — debugging only.
with tf.GradientTape() as tape:★Records every op on a watched tensor, then replays it backwards.grads = tape.gradient(loss, model.trainable_variables)★Variables are watched automatically. Constants are not.tape.watch(x)Needed for a plain tensor — for input gradients, saliency, adversarial examples.opt.apply_gradients(zip(grads, vars))Nozero_gradhere. A fresh tape each step means gradients never accumulate, unlike PyTorch.tf.GradientTape(persistent=True)The tape is consumed by the firstgradientcall unless you say this. Thendel tape.nested tapesSecond derivatives: an outer tape over an inner one.tape.stop_recording() · tf.stop_gradient(x)Exclude a region, or cut the gradient path at a tensor.grads come back as None★commonThe chain broke: a.numpy()call, an int dtype, or an op outside the tape.
tf.config.list_physical_devices("GPU")★TF grabs every visible GPU by default.with tf.device("/GPU:0"):Explicit placement. Rarely needed — TF places ops for you.tf.config.experimental.set_memory_growth(gpu, True)★Stops TF pre-allocating the entire GPU at startup. Set it before anything else runs.keras.mixed_precision.set_global_policy("mixed_float16")★One line, roughly 2× on modern GPUs.last layer: Dense(10, dtype="float32")With mixed precision, force the output layer back to float32 for numerical safety.tf.config.set_visible_devices([], "GPU")Hide the GPU to force a CPU run.CUDA_VISIBLE_DEVICES=0The environment-variable equivalent, applied before import.
tf.data.Dataset.from_tensor_slices((X, y))★Slices along the first axis — one element per row.tf.data.Dataset.from_generator(gen, output_signature=...)Python generator in. The signature is required.keras.utils.image_dataset_from_directory(path, image_size=(224,224))★Folder of class subfolders straight to a batched Dataset.keras.utils.text_dataset_from_directory(path)Same idea for text files.ds.map(fn, num_parallel_calls=tf.data.AUTOTUNE)★Always pass AUTOTUNE — without it, mapping is single-threaded.ds.filter(pred) · ds.take(n) · ds.skip(n)Subset the stream.ds.shuffle(1024).batch(32)Order matters. Shuffle before batch, or you only shuffle whole batches.ds.batch(32, drop_remainder=True)Fixed batch size — needed on TPU and for some static shapes.for x, y in ds.take(1): print(x.shape)How to actually look at one batch.ds.element_specThe shapes and dtypes the pipeline promises.
ds.prefetch(tf.data.AUTOTUNE)★Last step, always. Overlaps CPU preprocessing with GPU compute.ds.cache()★Caches after the first epoch. Put it after expensive deterministic maps, before random augmentation.ds.cache("/tmp/cache")Cache to disk when the dataset will not fit in RAM.vectorise the map, then batchA map over batches beats a map over elements. Per-element Python is the usual bottleneck.ds.interleave(fn, num_parallel_calls=AUTOTUNE)Read many shards at once instead of one file at a time.tf.data.TFRecordDataset(files)The native sharded format. Worth it once data outgrows memory.tf.data.Dataset.save(ds, path) · load(path)Snapshot a fully preprocessed pipeline.ds.shuffle(buffer_size=100)★quietA small buffer barely shuffles. It samples from a 100-element window, not the dataset.ds.repeat()Rarely needed —fit(epochs=n)already loops. An unbounded repeat plus nosteps_per_epochhangs forever.
keras.layers.Normalization(); layer.adapt(data)★Learns statistics from data, then travels with the saved model. No train/serve skew.keras.layers.Rescaling(1./255)★The one-liner every image model starts with.keras.layers.Resizing · CenterCropDeterministic image shaping.keras.layers.RandomFlip · RandomRotation · RandomZoom★Augmentation layers. Active in training, bypassed at inference automatically.keras.layers.TextVectorization(max_tokens=20000); .adapt(texts)Builds the vocabulary, then maps strings to int sequences.keras.layers.StringLookup · IntegerLookup · DiscretizationCategorical encoding and binning.keras.layers.CategoryEncoding(output_mode="one_hot")One-hot / multi-hot / count.in the pipeline, or in the model?Inds.mapit runs on CPU in parallel; inside the model it ships with the export. Augmentation in the pipeline, normalization in the model.
keras.Sequential([layers...])★A straight stack. Simplest, and enough surprisingly often.inp = keras.Input(shape=(32,)); out = Dense(1)(inp)★The Functional API. Any DAG: multiple inputs, skips, shared layers.model = keras.Model(inp, out)Close the Functional graph by naming its ends.class Net(keras.Model): call(self, x, training=None)Subclassing. Full Python control, but no static graph to inspect.model.summary()Shapes and parameter counts. Empty until the model is built.keras.utils.plot_model(model, show_shapes=True)Draws the graph. Functional models only.model.build(input_shape) · model(x)Subclassed and Sequential models are built lazily on first call.pass training through in a subclass★Forget it and Dropout stays on at inference. Functional models handle this for you.
keras.layers.Dense(128, activation="relu")★Units first. No input size — it is inferred.Conv2D(32, 3, padding="same", activation="relu")★Channels-last(N, H, W, C)— the opposite of PyTorch.MaxPooling2D(2) · GlobalAveragePooling2D()Downsample; collapse spatial dims before the head.BatchNormalization() · LayerNormalization()BatchNorm keeps running statistics — they are non-trainable variables.Dropout(0.5)Active in training only. Keras switches it for you.Embedding(vocab, dim) · LSTM · GRUSequence basics.MultiHeadAttention(num_heads=8, key_dim=64)Transformer attention, built in.Concatenate() · Add() · Lambda(fn)Merge branches; wrap an arbitrary op.out = (in + 2*pad − k) // stride + 1★The conv size formula.padding="same"keeps the size when stride is 1.
keras.losses.SparseCategoricalCrossentropy(from_logits=True)★Integer labels. The most common classification loss in TF.keras.losses.CategoricalCrossentropy()★One-hot labels. Choosing the wrong one of these two is a rite of passage.from_logits=True ↔ no final activation★pair upMatch them. A softmax layer plusfrom_logits=Truetrains on nonsense — and never errors.keras.losses.BinaryCrossentropy(from_logits=True) · MeanSquaredErrorBinary and regression.keras.optimizers.AdamW(learning_rate=1e-3, weight_decay=0.01)Decoupled decay — the modern default.keras.optimizers.schedules.CosineDecay(lr, steps)Pass a schedule object aslearning_rate.keras.metrics.SparseCategoricalAccuracy · AUC · PrecisionStateful objects: they accumulate across the epoch.loss_fn(y_true, y_pred)True first. TF orders the arguments the opposite way round from most libraries.
model.compile(optimizer=, loss=, metrics=[])★Wires the three together. Strings work:"adam","mse".model.fit(ds, epochs=10, validation_data=val_ds)★Handles batching, metrics, callbacks and the progress bar.model.fit(X, y, batch_size=32, validation_split=0.2)★The NumPy path.validation_splitworks only for arrays, not Datasets.history = model.fit(...); history.history["val_loss"]Everything for your learning curves is in that dict.model.evaluate(test_ds) · model.predict(x)predictreturns NumPy; callingmodel(x)returns a tensor and is faster for one batch.class_weight={0: 1.0, 1: 5.0}Imbalance, without touching the data.model.compile(..., jit_compile=True)fastXLA for the whole training step.steps_per_epoch=nRequired if your Dataset repeats forever.
keras.callbacks.ModelCheckpoint("best.keras", save_best_only=True)★Keep the best epoch, not the last one.keras.callbacks.EarlyStopping(patience=5, restore_best_weights=True)★Setrestore_best_weights— without it you keep the worse, later weights.keras.callbacks.ReduceLROnPlateau(factor=0.5, patience=3)Cut the learning rate when validation stalls.keras.callbacks.TensorBoard(log_dir="logs")Thentensorboard --logdir logs. Remember to pip install it on 2.21+.keras.callbacks.CSVLogger · LearningRateScheduler · BackupAndRestoreLog to file; per-epoch LR; resume after a crash.class Cb(keras.callbacks.Callback): on_epoch_end(self, epoch, logs)Your own. Alsoon_train_batch_endand friends.monitor="val_loss", mode="min"★Silently does nothing if the monitored name is not inlogs. Check your spelling.
@tf.function def train_step(x, y):★Decorate the step, not the epoch loop. That single line is most of the speed.with tf.GradientTape() as tape: loss = loss_fn(y, model(x, training=True))★training=Truematters: it switches Dropout and BatchNorm on.grads = tape.gradient(loss, model.trainable_variables)★One call gives you every gradient.opt.apply_gradients(zip(grads, model.trainable_variables))And the step is done. No zeroing required.metric.update_state(y, pred) → metric.result() → metric.reset_state()Keras metrics are stateful. Reset them each epoch or they average across all of training.model.train_step(self, data)The middle path: override just the step and keepfit()with all its callbacks.tf.function(reduce_retracing=True)For a last batch of a different size.
model.save("m.keras")★changedThe extension is mandatory in Keras 3. A bare directory path raisesValueError: Invalid filepath extension.keras.models.load_model("m.keras")★Architecture, weights, optimizer state — ready to resume.model.export("saved_model/")Keras 3This is how you get a SavedModel now. For TF Serving, LiteRT, anything outside Python.model.save_weights("m.weights.h5")Weights only. That exact suffix is required.keras.layers.TFSMLayer(path, call_endpoint="serving_default")The bridge back. Wraps any TF SavedModel as a Keras 3 layer — inference only.tf.train.Checkpoint(model=m, optimizer=o).save(path)The low-level, object-graph checkpoint. What long training runs use.custom_objects={...} · @keras.saving.register_keras_serializable()Custom layers and losses need registering or reloading fails.keras.models.load_model("saved_model/")★Keras 3Cannot load a SavedModel. Keras 3 reads only.kerasand.h5. UseTFSMLayer.
strategy = tf.distribute.MirroredStrategy()★Multi-GPU on one machine. The easiest real speedup there is.with strategy.scope(): model = build(); model.compile(...)★Build and compile inside the scope. Outside it, the variables are not mirrored.tf.distribute.MultiWorkerMirroredStrategy() · TPUStrategy()Many machines; TPU pods.global batch = per-replica × replicasKeras splits the batch you pass across replicas. Scale the learning rate accordingly.docker run -p 8501:8501 tensorflow/servingPoint it at an exported SavedModel directory and you have a REST endpoint.pip install ai-edge-litert★tf.litetf.liteis deprecated and is being removed from the TF package. On-device inference moved to the standalone LiteRT project.tf.saved_model.load(path) · obj.signatures["serving_default"]The pure-TF loader, no Keras involved.saved_model_cli show --dir path --allInspect signatures before you try to serve them.
print(x.shape, x.dtype, tf.reduce_mean(x))★The three questions. Nearly every TF error is one of them disagreeing.overfit a single batch firstIf the model cannot drive one batch to near-zero loss, the bug is in the model or the loss — not the data.tf.config.run_functions_eagerly(True)Disables tracing so you can set a breakpoint inside atf.function.model.compile(..., run_eagerly=True)Same idea, scoped to one model.tf.random.set_seed(s); random.seed(s); np.random.seed(s)Three separate RNGs. Seeding one is the usual mistake.tf.config.experimental.enable_op_determinism()Makes GPU kernels deterministic. Slower, and worth it while chasing a bug.TF_CPP_MIN_LOG_LEVEL=2Silence the informational C++ chatter at import.loss is NaN from step one★Learning rate too high, an unscaled input, or alog(0). Check the inputs before the architecture.val accuracy stuck at chance★Almost always thefrom_logits/ final-activation mismatch, or labels that do not line up with predictions.