I’ve just lately been tinkering with deep studying fashions in Tensorflow, and have accordingly been launched to managing knowledge as tensors.
As a Knowledge Engineer that works all day in tables that I can simply slice, cube, and visualize, I had completely no instinct round working with tensors, and I appeared to continually run into the identical errors that, particularly at first, went method over my head.
Nevertheless, deep diving them has taught me lots about tensors and TensorFlow, and I wished to consolidate these learnings right here to make use of as a reference.
In case you have a favourite error, resolution, or debugging tip, please go away a remark!
Earlier than we dive into the errors themselves, I wished to doc a number of of the lightweight, easy bits and items of code that I’ve discovered useful in debugging. (Though it should be acknowledged for authorized causes that we after all all the time debug with official debugging options and by no means simply dozens of print statements 🙂)
Seeing inside our Tensorflow Datasets
First off, our precise knowledge. After we print a Dataframe or SELECT * in SQL, we see the information! After we print a tensor dataset we see…
<_TensorSliceDataset element_spec=(TensorSpec(form=(2, 3), dtype=tf.int32, title=None), TensorSpec(form=(1, 1), dtype=tf.int32, title=None))>
That is all fairly helpful data, nevertheless it doesn’t assist us perceive what’s really occurring in our knowledge.
To print a single tensor inside the execution graph we will leverage tf.print. This text is a superb deep dive into tf.print that I extremely advocate in case you plan to make use of it typically: Utilizing tf.Print() in TensorFlow
However when working with Tensorflow datasets throughout growth, typically we have to see a number of values at a time. For that we will loop by way of and print particular person items of knowledge like this:
# Generate dummy 2D knowledge
np.random.seed(42)
num_samples = 100
num_features = 5
X_data = np.random.rand(num_samples, num_features).astype(np.float32)
y_data = 2 * X_data[:, 0] + 3 * X_data[:, 1] - 1.5 * X_data[:, 2] + 0.5 * X_data[:, 3] + np.random.randn(num_samples)# Flip it right into a Tensorflow Dataset
dataset = tf.knowledge.Dataset.from_tensor_slices((X_data, y_data))
# Print the primary 10 rows
for i, (options, label) in enumerate(dataset.take(10)):
print(f"Row {i + 1}: Options - {options.numpy()}, Label - {label.numpy()}")
We are able to additionally use skip to get to a particular index:
mini_dataset = dataset.skip(100).take(20)
for i, (options, label) in enumerate(mini_dataset):
print(f"Row {i + 1}: Options - {options.numpy()}, Label - {label.numpy()}")
Understanding our tensors’ specs
When working with tensors we additionally must know their form, rank, dimension, and knowledge kind (if a few of that vocabulary is unfamiliar, because it was to me initially, don’t fear, we’ll get again to it later within the article). Anyway, under are a number of strains of code to assemble this data:
# Create a pattern tensor
sample_tensor = tf.fixed([[1, 2, 3], [4, 5, 6]])# Get the scale of the tensor (complete variety of components)
tensor_size = tf.measurement(sample_tensor).numpy()
# Get the rank of the tensor
tensor_rank = tf.rank(sample_tensor).numpy()
# Get the form of the tensor
tensor_shape = sample_tensor.form
# Get the size of the tensor
tensor_dimensions = sample_tensor.form.as_list()
# Print the outcomes
print("Tensor Dimension:", tensor_size)
print("Tensor Rank:", tensor_rank)
print("Tensor Form:", tensor_shape)
print("Tensor Dimensions:", tensor_dimensions)
The above outputs:
Tensor Dimension: 6
Tensor Rank: 2
Tensor Form: (2, 3)
Tensor Dimensions: [2, 3]
Augmenting mannequin.abstract()
Lastly, its is all the time useful to have the ability to see how knowledge is shifting by way of a mannequin, and the way form adjustments all through inputs and outputs between layers. The supply of many an error might be a mismatch between these anticipated enter and output shapes and the form of a given tensor.
mannequin.abstract() after all will get the job achieved, however we will complement that data with the next snippet, which provides a bit extra context with mannequin and layer inputs and outputs:
print("###################Enter Form and Datatype#####################")
[print(i.shape, i.dtype) for i in model.inputs]
print("###################Output Form and Datatype#####################")
[print(o.shape, o.dtype) for o in model.outputs]
print("###################Layer Enter Form and Datatype#####################")
[print(l.name, l.input, l.dtype) for l in model.layers]
So let’s soar into some errors!
Rank
ValueError: Form should be rank x however is rank y….
Okay, to begin with, what’s a rank? Rank is simply the unit of dimensionality we use to explain tensors. A rank 0 tensor is a scalar worth; a rank one tensor is a vector; a rank two is a matrix, and so forth for all n dimensional constructions.
Take for instance a 5 dimensional tensor.
rank_5_tensor = tf.fixed([[[[[1, 2], [3, 4]], [[5, 6], [7, 8]]], [[[9, 10], [11, 12]], [[13, 14], [15, 16]]]],
[[[[17, 18], [19, 20]], [[21, 22], [23, 24]]], [[[25, 26], [27, 28]], [[29, 30], [31, 32]]]]])
print("nRank 5 Tensor:", rank_5_tensor.form)
Rank 5 Tensor: (2, 2, 2, 2, 2)
The code above exhibits that every dimension of the 5 has a measurement of two. If we wished to index it, we might accomplish that alongside any of those axes. To get on the final component, 32, we might run one thing like:
rank_5_tensor.numpy()[1][1][1][1][1]
The official tensor documentation has some actually useful visualizations to make this a bit extra understandable.
Again to the error: it’s simply flagging that the tensor supplied is a distinct dimension than what is anticipated to a selected perform. For instance if the error declares that the “Form should be rank 1 however is rank 0…” it implies that we’re offering a scalar worth, and it expects a 1-D tensor.
Take the instance under the place we try to multiply tensors along with the matmul technique.
import tensorflow as tf
import numpy as np
# Create a TensorFlow dataset with random matrices
num_samples = 5
matrix_size = 3
dataset = tf.knowledge.Dataset.from_tensor_slices(np.random.rand(num_samples, matrix_size, matrix_size))
mul = [1,2,3,4,5,6]# Outline a perform that makes use of tf.matmul
def matmul_function(matrix):
return tf.matmul(matrix, mul)
# Apply the matmul_function to the dataset utilizing map
result_dataset = dataset.map(matmul_function)
If we take a peek on the documentation, matmul expects not less than a rank 2 tensor, so multiplying the matrix by [1,2,3,4,5,6], which is simply an array, will elevate this error.
ValueError: Form should be rank 2 however is rank 1 for '{{node MatMul}} = MatMul[T=DT_DOUBLE, transpose_a=false, transpose_b=false](args_0, MatMul/b)' with enter shapes: [3,3], [2].
An excellent first step for this error is to dive into the documentation and perceive what the perform you might be utilizing is in search of (Right here’s a pleasant checklist of the capabilities accessible on tensors: raw_ops.
Then use the rank technique to find out what we are literally offering.
print(tf.rank(mul))
tf.Tensor(1, form=(), dtype=int32)
So far as fixes go, tf.reshape is commonly an excellent choice to start out with. Let’s take a quick second to speak just a little bit about tf.reshape, since it will likely be a devoted companion all through our Tensorflow journey: tf.reshape(tensor, form, title=None)
Reshape merely takes within the tensor we need to reshape, and one other tensor containing what we would like the form of the output to be. For instance, let’s reshape our multiplication enter:
mul = [1,2,3,4,5,6]
tf.reshape(mul, [3, 2]).numpy()
array([[1, 2],
[3, 4],
[5, 6]], dtype=int32)
Our variable will flip right into a (3,2) tensor (3 rows, 2 columns). A fast observe, tf.reshape(t, [3, -1]).numpy() will produce the identical factor as a result of the -1 tells Tensorflow to compute the scale of the dimension such that the whole measurement stays fixed. The variety of components within the form tensor is the rank.
As soon as we create a tensor with the correct rank, our multiplication will work simply tremendous!
Form
ValueError: Enter of layer is incompatible with layer….
Having an intuitive understanding of tensor form, and the way it interacts and adjustments throughout mannequin layers has made life with deep studying considerably simpler
First, getting primary vocab out of the way in which: the form of a tensor refers back to the variety of components alongside every dimension, or axis of the tensor. For instance, a 2D tensor with 3 rows and 4 columns has a form of (3, 4).
So what can go improper with form? Glad you requested, fairly a number of issues!
At the start the form and rank of your coaching knowledge should match the enter form anticipated by the enter layer. Let’s check out an instance, a primary CNN:
import tensorflow as tf
from tensorflow.keras import layers, fashions# Create a perform to generate pattern knowledge
def generate_sample_data(num_samples=100):
for _ in vary(num_samples):
options = tf.random.regular(form=(64, 64, 3))
labels = tf.one_hot(tf.random.uniform(form=(), maxval=10, dtype=tf.int32), depth=10)
yield options, labels
# Create a TensorFlow dataset utilizing the generator perform
sample_dataset = tf.knowledge.Dataset.from_generator(generate_sample_data, output_signature=(tf.TensorSpec(form=(64, 64, 3), dtype=tf.float32), tf.TensorSpec(form=(10,), dtype=tf.float32)))
# Create a CNN mannequin with an enter layer anticipating (128, 128, 3)
mannequin = fashions.Sequential()
mannequin.add(layers.Conv2D(32, (3, 3), activation='relu', input_shape=(128, 128, 3)))
mannequin.add(layers.MaxPooling2D((2, 2)))
mannequin.add(layers.Conv2D(64, (3, 3), activation='relu'))
mannequin.add(layers.MaxPooling2D((2, 2)))
mannequin.add(layers.Conv2D(64, (3, 3), activation='relu'))
mannequin.add(layers.Flatten())
mannequin.add(layers.Dense(64, activation='relu'))
mannequin.add(layers.Dense(10, activation='softmax'))
# Compile the mannequin
mannequin.compile(optimizer='adam', loss='categorical_crossentropy', metrics=['accuracy'])
# Match the mannequin utilizing the dataset
mannequin.match(sample_dataset.batch(32).repeat(), epochs=5, steps_per_epoch=100, validation_steps=20)
Making an attempt to run the code above will lead to:
ValueError: Enter 0 of layer "sequential_5" is incompatible with the layer: anticipated form=(None, 128, 128, 3), discovered form=(None, 64, 64, 3)
It’s because our mannequin is anticipating the enter tensor to be of the form (128, 128, 3) and our generated knowledge is (64, 64, 3).
In a scenario like this, our good buddy, reshape, or one other Tensorflow perform, resize, may help. If, as within the case above we’re working with photographs, we will merely run resize or change the expectations of our mannequin’s enter:
def resize_image(picture, label):
resized_image = tf.picture.resize(picture, measurement=target_shape)
return resized_image, label# Apply the resize perform to your entire dataset
resized_dataset = sample_dataset.map(resize_image)
On this context, it’s useful to know just a little about how widespread varieties of fashions and mannequin layers anticipate enter of various shapes, so let’s take just a little detour.
Deep Neural Networks of Dense layers absorb 1 dimensional tensors (or 2 dimensional, relying on whether or not you embody batch measurement, however we’ll speak about batch measurement in a bit) of the format (feature_size, ) the place feature_size is the variety of options in every pattern.
Convolutional Neural Networks absorb knowledge representing photographs, utilizing 3 dimensional tensors of (width, peak, channels) the place channels are the colour scheme, 1 for grey scale, and three for RBG.
And at last, Recurrent Neural Networks similar to LTSMs absorb 2 dimensions (time steps, feature_size)
However again to errors! One other widespread perpetrator in Tensorflow form errors has to do with how form adjustments as knowledge passes by way of the mannequin layers. As beforehand talked about, completely different layers absorb completely different enter shapes, they usually may reshape output.
Returning to our CNN instance from above, let’s break it once more, by seeing what occurs once we take away the Flatten layer. If we attempt to run the code we’ll see
ValueError: Shapes (None, 10) and (None, 28, 28, 10) are incompatible
That is the place printing all of our mannequin enter and output shapes together with our knowledge shapes turns out to be useful to assist us pinpoint the place there’s a mismatch.
mannequin.abstract() will present us
Layer (kind) Output Form Param #
=================================================================
conv2d_15 (Conv2D) (None, 126, 126, 32) 896
max_pooling2d_10 (MaxPooli (None, 63, 63, 32) 0
ng2D)
conv2d_16 (Conv2D) (None, 61, 61, 64) 18496
max_pooling2d_11 (MaxPooling2D) (None, 30, 30, 64) 0
conv2d_17 (Conv2D) (None, 28, 28, 64) 36928
flatten_5 (Flatten) (None, 50176) 0
dense_13 (Dense) (None, 64) 3211328
dense_14 (Dense) (None, 10) 650
=================================================================
Whole params: 3268298 (12.47 MB)
Trainable params: 3268298 (12.47 MB)
Non-trainable params: 0 (0.00 Byte)
And our additional diagnostic will reveal
###################Enter Form and Datatype#####################
(None, 128, 128, 3) <dtype: 'float32'>
###################Output Form and Datatype#####################
(None, 10) <dtype: 'float32'>
###################Layer Enter Form and Datatype#####################
conv2d_15 KerasTensor(type_spec=TensorSpec(form=(None, 128, 128, 3), dtype=tf.float32, title='conv2d_15_input'), title='conv2d_15_input', description="created by layer 'conv2d_15_input'") float32
max_pooling2d_10 KerasTensor(type_spec=TensorSpec(form=(None, 126, 126, 32), dtype=tf.float32, title=None), title='conv2d_15/Relu:0', description="created by layer 'conv2d_15'") float32
conv2d_16 KerasTensor(type_spec=TensorSpec(form=(None, 63, 63, 32), dtype=tf.float32, title=None), title='max_pooling2d_10/MaxPool:0', description="created by layer 'max_pooling2d_10'") float32
max_pooling2d_11 KerasTensor(type_spec=TensorSpec(form=(None, 61, 61, 64), dtype=tf.float32, title=None), title='conv2d_16/Relu:0', description="created by layer 'conv2d_16'") float32
conv2d_17 KerasTensor(type_spec=TensorSpec(form=(None, 30, 30, 64), dtype=tf.float32, title=None), title='max_pooling2d_11/MaxPool:0', description="created by layer 'max_pooling2d_11'") float32
flatten_5 KerasTensor(type_spec=TensorSpec(form=(None, 28, 28, 64), dtype=tf.float32, title=None), title='conv2d_17/Relu:0', description="created by layer 'conv2d_17'") float32
dense_13 KerasTensor(type_spec=TensorSpec(form=(None, 50176), dtype=tf.float32, title=None), title='flatten_5/Reshape:0', description="created by layer 'flatten_5'") float32
dense_14 KerasTensor(type_spec=TensorSpec(form=(None, 64), dtype=tf.float32, title=None), title='dense_13/Relu:0', description="created by layer 'dense_13'") float32
It’s loads of output, however we will see that dense_13 layer is in search of enter of (None, 50176) form. Nevertheless, conv2d_17 layer outputs (None, 28, 28, 64)
Flatten layers remodel the multi-dimensional output from earlier layers right into a one-dimensional (flat) vector that the Dense layer expects.
Conv2d and Max Pooling layers change their enter knowledge in different fascinating methods as nicely, however these are out of scope for this text. For an superior breakdown check out: Final Information to Enter form and Mannequin Complexity in Neural Networks
However what about batch measurement?! I haven’t forgotten!
If we break our code yet another time by eradicating the .batch(32) from the dataset in mannequin.match we’ll get the error:
ValueError: Enter 0 of layer "sequential_10" is incompatible with the layer: anticipated form=(None, 128, 128, 3), discovered form=(128, 128, 3)
That’s as a result of, the primary dimension of a layer’s enter is reserved for the batch measurement or variety of samples that we would like the mannequin to work by way of at a time. For a terrific deep dive learn by way of Distinction between batch and epoch.
Batch measurement defaults to None previous to becoming, as we will see within the mannequin abstract output, and our mannequin expects us to set it elsewhere, relying on how we tune the hyperparameter. We are able to additionally pressure it in our enter layer by utilizing batch_input_size as a substitute of input_size, however that decreases our flexibility in testing out completely different values.
Kind
TypeError: Didn’t convert object of kind to Tensor. Unsupported object kind
Lastly, let’s speak a bit about some knowledge kind specifics in Tensors.
The error above is one other, that, in case you’re used to working in database methods with tables constructed from all kinds of knowledge, could be a bit baffling, nevertheless it is without doubt one of the extra easy to diagnose and repair, though there are a few widespread causes to look out for.
The primary difficulty is that, though tensors help a wide range of knowledge varieties, once we convert a NumPy array to tensors (a typical circulate inside deep studying), the datatypes should be floats. The script under initializes a contrived instance of a dataframe with None and with string knowledge factors. Let’s stroll by way of some difficulty and fixes for this instance:
import numpy as np
import pandas as pd
import tensorflow as tf
from tensorflow.keras.layers import Dense
from tensorflow.keras.fashions import Sequential
knowledge = [
[None, 0.2, '0.3'],
[0.1, None, '0.3'],
[0.1, 0.2, '0.3'],
]
X_train = pd.DataFrame(knowledge=knowledge, columns=["x1", "x2", "x3"])
y_train = pd.DataFrame(knowledge=[1, 0, 1], columns=["y"])# Create a TensorFlow dataset
train_dataset = tf.knowledge.Dataset.from_tensor_slices((X_train.to_numpy(), y_train.to_numpy()))
# Outline the mannequin
mannequin = Sequential()
mannequin.add(Dense(1, input_dim=X_train.form[1], activation='sigmoid'))
mannequin.compile(optimizer='adam', loss='binary_crossentropy', metrics=['accuracy'])
# Match the mannequin utilizing the TensorFlow dataset
mannequin.match(train_dataset.batch(3), epochs=3)
Operating this code will flag to us that:
ValueError: Didn't convert a NumPy array to a Tensor (Unsupported object kind float).
The obvious difficulty is that you’re sending in a NumPy array that accommodates some non float kind, an object. In case you have an precise column of categorical knowledge, there are numerous methods to transform that to numeric knowledge (One shot encoding, and so forth) however that’s out of scope for this dialogue.
We are able to decide that if we run print(X_train.dtypes), which can inform us what’s in our dataframe that Tensorflow doesn’t like.
x1 float64
x2 float64
x3 object
dtype: object
If we’re working into non float knowledge factors, the road under will magically remedy all of our issues:
X_train = np.asarray(X_train).astype('float32')
One other factor to verify for is when you’ve got None or np.nan wherever.
To search out out we will use a number of strains of code similar to:
null_mask = X_train.isnull().any(axis=1)
null_rows = X_train[null_mask]
print(null_rows)
Which tells us that we’ve nulls on rows 0 and 1:
x1 x2 x3
0 NaN 0.2 0.3
1 0.1 NaN 0.3
In that case, and that’s anticipated/intentional we have to exchange these values with an appropriate different. Fillna may help us right here.
X_train.fillna(worth=0, inplace=True)
With these adjustments to the code under, our NumPy array will efficiently convert to a tensor dataset and we will prepare our mannequin!
I typically discover that I study probably the most a few explicit know-how when I’ve to work by way of errors, and I hope this has been considerably useful to you too!
In case you have cool ideas and methods or enjoyable Tensorflow errors please cross them alongside!
