Installation & Setup
NeuNet is built with pure Python and NumPy. Here's how to get started:
Prerequisites
pip install numpy matplotlib plotly networkx pandasProject Structure
NeuNet/
├── src/
│ ├── models/
│ │ └── neural_network.py # Core NeuralNetwork class
│ ├── layers/
│ │ ├── core.py # Dense layer implementation
│ │ ├── activations.py # Activation functions
│ │ ├── regularization.py # BatchNorm, Dropout
│ │ ├── losses.py # Loss functions
│ │ ├── optimisers.py # SGD, Adam optimizers
│ │ └── dataset.py # Data generation utilities
│ ├── utils/
│ │ ├── metrics.py # Performance metrics
│ │ ├── network_data.py # Network export utilities
│ │ └── Visualization.py # Network visualization
│ └── main.py # Example implementation
└── docs/ # Documentation
Basic Usage
Creating your first neural network with NeuNet is straightforward:
from src.models.neural_network import NeuralNetwork
from src.layers.core import Dense
from src.layers.activations import ReLU, Softmax
from src.layers.losses import CategoricalCrossentropy
# Create a new neural network
model = NeuralNetwork()
# Add layers
model.add(Dense(2, 64, learning_rate=0.01)) # Input: 2, Output: 64
model.add(ReLU()) # ReLU activation
model.add(Dense(64, 32, learning_rate=0.01)) # Hidden layer
model.add(ReLU()) # ReLU activation
model.add(Dense(32, 3, learning_rate=0.01)) # Output layer: 3 classes
model.add(Softmax()) # Softmax for classification
# Set loss function
model.set_loss(CategoricalCrossentropy())
# Train the model (X, Y are your data)
model.train(X, Y, epochs=100, batch_size=32)
# Make predictions
predictions = model.predict(X)
probabilities = model.predict_proba(X)
Network Architecture
The NeuralNetwork class is the core component that orchestrates the entire training process:
Key Features
- Layer Management: Add layers sequentially using the
add()method - Forward Pass: Automatic propagation through all layers
- Backward Pass: Gradient computation and backpropagation
- Training Loop: Built-in batch processing and early stopping
- History Tracking: Loss and accuracy logging during training
Training Process
# The training method handles:
# 1. Data shuffling for each epoch
# 2. Batch processing
# 3. Forward and backward passes
# 4. Early stopping based on loss improvement
# 5. Learning rate decay
# 6. Performance logging
model.train(
X, Y, # Training data
epochs=500, # Maximum epochs
batch_size=32, # Batch size (0 for full batch)
patience=30, # Early stopping patience
verbose=True # Print training progress
)
Layer Types
Dense Layer
The fully connected layer is the core building block of the network:
from src.layers.core import Dense
# Create a dense layer
layer = Dense(
n_inputs=64, # Number of input features
n_neurons=32, # Number of output neurons
learning_rate=0.01, # Learning rate
decay_rate=0.02, # Learning rate decay
momentum=0.9, # Momentum for SGD
optimizer='adam' # 'adam' or None for SGD
)
Weight Initialization
The Dense layer uses He initialization for better gradient flow:
# He initialization for ReLU networks
self.weights = np.random.randn(n_inputs, n_neurons) * np.sqrt(2. / n_inputs)
self.biases = np.zeros((1, n_neurons))
Forward and Backward Pass
# Forward pass: y = Wx + b
output = np.dot(inputs, weights) + biases
# Backward pass: compute gradients
dW = np.dot(inputs.T, dvalues)
db = np.sum(dvalues, axis=0, keepdims=True)
dinputs = np.dot(dvalues, weights.T)
Activation Functions
NeuNet provides a complete set of activation functions with proper gradient computation:
ReLU (Recommended for Hidden Layers)
from src.layers.activations import ReLU
# ReLU: max(0, x)
relu = ReLU()
output = relu.forward(inputs) # Forward pass
gradients = relu.backward(dvalues) # Backward pass
Leaky ReLU
from src.layers.activations import LeakyReLU
# Leaky ReLU: max(αx, x) where α=0.01
leaky_relu = LeakyReLU(alpha=0.01)
Softmax (For Classification Output)
from src.layers.activations import Softmax
# Softmax: e^xi / Σe^xj (probability distribution)
softmax = Softmax()
# Includes numerical stability and proper Jacobian computation
Other Activations
from src.layers.activations import Tanh, Sigmoid
# Tanh: (-1, 1) range
tanh = Tanh()
# Sigmoid: (0, 1) range with numerical stability
sigmoid = Sigmoid()
Optimizers
NeuNet includes two sophisticated optimizers for efficient training:
SGD with Momentum
# SGD with momentum helps accelerate gradients in relevant directions
# and dampens oscillations
# Velocity update:
velocity = momentum * velocity - learning_rate * gradient
# Parameter update:
weights += velocity
Adam Optimizer (Recommended)
# Adam combines momentum and adaptive learning rates
# Includes bias correction for better early training
# Momentum estimates:
m = β₁ * m + (1 - β₁) * gradient
v = β₂ * v + (1 - β₂) * gradient²
# Bias correction:
m_corrected = m / (1 - β₁^t)
v_corrected = v / (1 - β₂^t)
# Parameter update:
weights -= learning_rate * m_corrected / (√v_corrected + ε)
Usage Example
# Use Adam optimizer (recommended for most cases)
model.add(Dense(64, 32, learning_rate=0.002, optimizer='adam'))
# Use SGD with momentum (more traditional approach)
model.add(Dense(64, 32, learning_rate=0.01, optimizer=None, momentum=0.9))
Regularization Techniques
Prevent overfitting with built-in regularization methods:
Batch Normalization
from src.layers.regularization import BatchNormalization
# Normalize inputs to have zero mean and unit variance
batch_norm = BatchNormalization(
epsilon=1e-5, # Small constant for numerical stability
momentum=0.9 # Running statistics momentum
)
# Handles training vs inference modes automatically
# Maintains running mean and variance for inference
Dropout
from src.layers.regularization import Dropout
# Randomly zero out neurons during training
dropout = Dropout(rate=0.1) # Drop 10% of neurons
# Automatically handles training vs inference:
# - Training: applies random masking with scaling
# - Inference: no dropout applied
L1/L2 Regularization
from src.layers.losses import CategoricalCrossentropy
# Add L2 regularization to loss function
loss = CategoricalCrossentropy(
regularization_l2=0.0001, # L2 penalty coefficient
regularization_l1=0.0 # L1 penalty coefficient
)
# Regularization is applied during gradient computation
# Helps prevent overfitting by penalizing large weights
Training Process
The training loop includes several advanced features:
Early Stopping
# Early stopping prevents overfitting
model.train(
X, Y,
epochs=1000, # Maximum epochs
patience=30, # Stop if no improvement for 30 epochs
batch_size=32
)
# Monitors validation loss and stops when:
# 1. Loss doesn't improve for 'patience' epochs
# 2. Minimum 100 epochs have passed
Learning Rate Decay
# Automatic exponential learning rate decay
# learning_rate = initial_lr * exp(-decay_rate * epoch)
layer = Dense(64, 32,
learning_rate=0.01, # Initial learning rate
decay_rate=0.02 # Decay coefficient
)
Batch Processing
# Efficient batch processing with data shuffling
model.train(
X, Y,
batch_size=32, # Process 32 samples at once
# batch_size=0 # Use full batch (all data)
)
# Each epoch:
# 1. Shuffles training data
# 2. Processes data in batches
# 3. Updates parameters after each batch
Training History
# Access training history
history = model.history
print(f"Loss history: {history['loss']}")
print(f"Accuracy history: {history['accuracy']}")
# Logged every 20 epochs during training
Model Evaluation
Comprehensive evaluation metrics are available:
Basic Metrics
from src.utils.metrics import calculate_accuracy
# Get predictions
predictions = model.predict(X_test) # Class predictions
probabilities = model.predict_proba(X_test) # Class probabilities
# Calculate accuracy
accuracy = calculate_accuracy(y_true, probabilities)
print(f"Accuracy: {accuracy:.4f}")
Advanced Metrics
from src.utils.metrics import confusion_matrix, precision_recall_f1
# Confusion matrix
cm = confusion_matrix(y_true, predictions, num_classes=3)
print("Confusion Matrix:")
print(cm)
# Precision, Recall, F1-score
precision, recall, f1 = precision_recall_f1(y_true, predictions, num_classes=3)
print(f"Precision: {precision}")
print(f"Recall: {recall}")
print(f"F1-score: {f1}")
Visualization
NeuNet includes powerful visualization tools:
Network Visualization
from src.utils.network_data import export_network
from src.utils.Visualization import network_visualization
# Export network structure (first 4 dense layers)
dense_layers = [layer for layer in model.layers if hasattr(layer, 'weights')]
export_network(*dense_layers[:4])
# Create interactive visualization
fig = network_visualization("src/utils/network_data.json")
# Saves as 'neural_network_visualization.html'
Dataset Visualization
from src.layers.dataset import create_data
# Create and visualize synthetic dataset
X, Y = create_data(samples=100, classes=3, plot=True)
# Automatically saves scatter plot as 'scatter_plot.png'
Complete Examples
Classification on Synthetic Data
import sys
import os
sys.path.append(os.path.abspath(os.path.join(os.path.dirname(__file__), '..')))
from src.models.neural_network import NeuralNetwork
from src.layers.core import Dense
from src.layers.activations import ReLU, Softmax
from src.layers.regularization import BatchNormalization, Dropout
from src.layers.losses import CategoricalCrossentropy
from src.layers.dataset import create_data
from src.utils.metrics import calculate_accuracy
# Create synthetic dataset
X, Y = create_data(samples=100, classes=3, noise=0.2)
# Build the network
model = NeuralNetwork()
# Layer 1: Input -> 128 neurons
model.add(Dense(2, 128, learning_rate=0.002, optimizer='adam'))
model.add(BatchNormalization())
model.add(ReLU())
model.add(Dropout(0.1))
# Layer 2: 128 -> 64 neurons
model.add(Dense(128, 64, learning_rate=0.002, optimizer='adam'))
model.add(BatchNormalization())
model.add(ReLU())
model.add(Dropout(0.1))
# Layer 3: 64 -> 32 neurons
model.add(Dense(64, 32, learning_rate=0.002, optimizer='adam'))
model.add(BatchNormalization())
model.add(ReLU())
model.add(Dropout(0.1))
# Output layer: 32 -> 3 classes
model.add(Dense(32, 3, learning_rate=0.002, optimizer='adam'))
model.add(Softmax())
# Set loss function with L2 regularization
model.set_loss(CategoricalCrossentropy(regularization_l2=0.0001))
# Train the model
print("Training neural network...")
model.train(X, Y, epochs=500, batch_size=32, patience=30)
# Evaluate the model
predictions = model.predict_proba(X)
accuracy = calculate_accuracy(Y, predictions)
print(f"\nFinal Accuracy: {accuracy:.4f}")
# Export and visualize the network
from src.utils.network_data import export_network
dense_layers = [layer for layer in model.layers if hasattr(layer, 'weights')]
if len(dense_layers) >= 4:
export_network(*dense_layers[:4])
print("Network exported successfully!")
Custom Training Loop
# For more control, you can implement custom training
def custom_training_loop(model, X, Y, epochs=100):
for epoch in range(epochs):
# Forward pass
output = model.forward(X, training=True)
# Calculate loss
loss = model.loss_function.calculate(output, Y)
# Backward pass
loss_gradient = model.loss_function.backward(output, Y)
model.backward(loss_gradient, epoch)
# Log progress
if epoch % 20 == 0:
val_output = model.forward(X, training=False)
accuracy = calculate_accuracy(Y, val_output)
print(f"Epoch {epoch}, Loss: {loss:.6f}, Accuracy: {accuracy:.4f}")
# Use custom training
custom_training_loop(model, X, Y, epochs=200)