TensorFlow Troubleshooting

Back to troubleshooting

Overview

This guide covers common TensorFlow 2.x installation and runtime issues, including:

• Installation with GPU support
• TensorRT integration problems
• Keras compatibility issues
• TensorBoard profiler bugs

---

TensorFlow 2.x Installation

Best Practices

Use pip, not conda

Install TensorFlow with pip instead of conda to avoid compatibility issues and ensure you get the latest stable release with proper CUDA support.

Step 1: Upgrade pip

Upgrade pip to latest version

pip install --upgrade pip

Step 2: Install TensorFlow

With CUDA (Recommended)

Install TensorFlow with GPU support

python3 -m pip install 'tensorflow[and-cuda]'

This automatically installs compatible CUDA libraries.

Standard

Install standard TensorFlow

pip install tensorflow

May require manual CUDA setup depending on your system configuration.

Step 3: Verify Installation

Verify TensorFlow GPU support

python3 -c "import tensorflow as tf; print(tf.config.list_physical_devices('GPU'))"

Expected output:

[PhysicalDevice(name='/physical_device:GPU:0', device_type='GPU')]

If you see an empty list [], check:

1. NVIDIA drivers are installed - see Driver Installation
2. CUDA version compatibility
3. Environment activation - see Environment Setup

---

TensorRT Integration Issues

Problem

TensorFlow cannot find TensorRT even after installation, showing CUDA errors or warnings.

Solution

Step 1: Install TensorRT

Install TensorRT for Python

pip install nvidia-pyindex
pip install nvidia-tensorrt

Step 2: Fix Library Path

Add TensorRT to library path

# Replace 'user' with your username and adjust Python version as needed
export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:"/home/user/miniconda3/envs/tf/lib/python3.11/site-packages/tensorrt_libs/"

# Make it persistent by adding to ~/.bashrc or conda environment activation script
echo 'export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:"/home/user/miniconda3/envs/tf/lib/python3.11/site-packages/tensorrt_libs/"' >> ~/.bashrc

CUDA Warnings vs Errors

Some CUDA warnings may persist in TensorFlow 2.x but are not critical errors. As long as GPU training works, these warnings can typically be ignored.

---

Keras Compatibility Issues

Error: AttributeError: module 'keras' has no attribute 'ops'

Cause: Version mismatch between Keras and TensorFlow

Solutions:

Use TF Keras (Recommended)

Import from TensorFlow

# Instead of: import keras
from tensorflow import keras

# This ensures version compatibility

Downgrade Keras

Install compatible Keras version

pip install keras==2.15.0  # Adjust based on TensorFlow version

Check Versions

Verify compatibility

import tensorflow as tf
print(f"TensorFlow: {tf.__version__}")
print(f"Keras: {tf.keras.__version__}")

---

TensorBoard Profiler Issues

Problem: Profile Data Not Showing

Symptoms: TensorBoard profiler shows “No profile data was found” even though profiling ran successfully.

Root Cause: Log file structure bug in TensorBoard profiler.

Solution:

Fix profile log structure

# Move profile logs up one directory level
# From: logs/train/plugins/profile/...
# To: logs/plugins/profile/...

cd logs
mv train/plugins/profile/* plugins/profile/ 2>/dev/null || true
mv validation/plugins/profile/* plugins/profile/ 2>/dev/null || true

The profile logs should be at the same directory level as train and validation directories, not inside them.

Known Issue

TensorBoard profiler is actively developed and bugs may vary between versions. If you encounter profiling issues:

1. Check the TensorFlow GitHub issues
2. Try updating TensorBoard: pip install --upgrade tensorboard
3. Verify it’s not an environment or installation problem

GitHub Discussion

Detailed Solution Guide

---

Related Resources

• Environment Setup - Python environment configuration
• GPU Detection - Troubleshoot GPU availability
• Driver Installation - CUDA and driver setup