SSHadow: Security-Focused SSH Monitoring for Bastion Architectures
The Problem with Blind Bastion Hosts
Run SSH bastion hosts long enough and you'll start asking uncomfortable questions. Who's connected right now? What certificate are they using? Did someone just attempt a brute force from that sketchy IP range? How many sessions does that contractor have open?
Traditional SSH logging wasn't built to answer these questions. You get disconnected log events scattered across auth.log. You grep through files trying to correlate usernames with source IPs. You have no real-time view of what's actually happening on your infrastructure right now. Sure, you can piece together a timeline after an incident, but that's cold comfort when you're trying to detect an ongoing attack.
SSHadow changes this by giving you a live feed of SSH connections with full authentication context. This is not a replacement for traditional logging. This is a real-time intelligence layer that makes sense of what's happening as it happens.
What SSHadow Does
SSHadow is a Go-based SSH connection monitor that adapts to your environment through three distinct operating modes. The flexibility matters because not every organization can drop a proxy into their production SSH path, and not every security team is comfortable with passive-only monitoring.
Check it out at github.com/Splat/SSHadow.
Three Monitoring Modes
The first mode is active proxy interception. SSHadow sits inline between clients and your SSH service, performing a man-in-the-middle inspection during handshake. This is the most invasive approach but also the most powerful because you get real-time extraction of authentication details with zero delay. The proxy captures everything during the handshake, then forwards the connection to your actual sshd and gets out of the way.
The second mode flips the script entirely. Instead of touching the connection path, SSHadow runs as a passive log monitor. Point it at your "auth.log" files or "systemd journald", and it parses authentication events as they occur. This approach is perfect for environments where you can't modify the infrastructure or where compliance requirements demand non-invasive monitoring. The trade-off is slightly delayed visibility and you're beholden to log rotation and write frequency.
The third mode combines both approaches in hybrid operation. You get the detailed real-time data from proxy interception plus the additional context from log monitoring. This is defense-in-depth monitoring: two independent data streams feeding a unified view. If an attacker somehow bypasses one monitoring channel, the other catches it. It's overkill for some environments and essential for others.
The choice of mode should be the result of a threat model rather than just technicals. How invasive can you be? How much latency can you accept? Do you need defense-in-depth or is single-source monitoring sufficient? SSHadow gives you the tools to implement whichever approach fits your requirements.
Regardless of which mode you run, SSHadow tracks the same core data: usernames, source IPs, authentication types (password, public key, or certificate), key fingerprints for public keys, and the full certificate context including key IDs, principals, serial numbers, and issuing CA. All of this feeds into multiple export formats including a web dashboard with auto-refresh, native Prometheus metrics, and a JSON API for custom integrations.
Architecture and Deployment Flexibility
The architecture is deliberately simple. In proxy mode, the data flow looks like this: client connects to SSHadow, which performs the SSH handshake to extract authentication metadata, then establishes a second connection to your real SSH service and proxies all traffic between them. The metrics server runs on a separate port, exposing the dashboard and monitoring endpoints. Once the initial handshake completes, SSHadow is just forwarding bytes.
Logmon mode inverts this completely. Your existing SSH service handles all connections normally. SSHadow sits off to the side, tailing auth.log or pulling from journald, parsing authentication events as they arrive. No connections flow through SSHadow at all. It's purely observational. The metrics server still exposes the same dashboard and endpoints, just fed from a different data source.
Hybrid mode runs both paths simultaneously. Connections flow through the proxy while SSHadow also monitors the logs. This gives you two independent views of the same authentication events, which is useful for correlation and validation. If the proxy sees a connection but the logs don't reflect it, something's wrong. If the logs show authentication attempts that never made it to the proxy, you're seeing direct attacks on the underlying SSH service.
Choosing your deployment mode comes down to constraints and priorities. Use proxy mode when you're building something new or when you need the absolute lowest latency on detection. The sub-second visibility is worth the architectural changes. Use logmon mode when you're dealing with existing infrastructure that can't be easily modified, or when organizational politics make inserting a proxy difficult. Use hybrid mode when you're building a proper security operations capability and need defense-in-depth monitoring with correlation across multiple data sources.
Deployment Example: Proxy Mode (Transparent)
The cleanest proxy deployment makes SSHadow completely transparent to clients:
- Configure your existing sshd to listen on a different port (e.g., 2222)
- Run SSHadow on port 22 (the standard SSH port)
- Point SSHadow at localhost:2222
Clients connect normally to port 22, completely unaware they're hitting a proxy. You get full monitoring without changing a single client configuration.
# Move sshd to port 2222
echo "Port 2222" >> /etc/ssh/sshd_config
systemctl restart sshd
# Run SSHadow on port 22
./SSHadow -hostkey /etc/SSHadow/host_key -listen :22 -target localhost:2222 -metrics :9090Certificate Authentication: The Real Power
SSH certificates are finally gaining traction in production environments, and for good reason. Traditional public key authentication is a pain to manage at scale while you're distributing and rotating keys, managing authorized_keys files across hundreds of servers, and building custom tooling for revocation. Certificates solve this by moving trust to a CA and embedding time bounds and authorization context directly into the credential.
SSHadow understands certificates as first-class authentication objects. When a user authenticates with a certificate, SSHadow extracts the key ID, which should be a human-readable identifier like "prod-deploy-key-1" or "alice-workstation-2024". It also grabs the list of principals, which define what identities the certificate asserts. A certificate might have principals like ["alice", "developers", "deployments"], meaning it can authenticate as any of those identities depending on what the server accepts.
Beyond that, SSHadow tracks the certificate serial number and the fingerprint of the issuing CA. This gives you a complete audit trail. You're not just seeing "user bob authenticated". Instead you're seeing "user bob authenticated with certificate prod-deploy-key-1, principals [bob, deployments], serial 12345, issued by internal-ca-sha256:xyz".
This level of detail transforms incident response. During a security event, you can immediately identify which certificate was used, who issued it, what authorization scope it had, and whether it's the expected credential for that user. You can detect certificate misuse, a deployment key being used for interactive sessions, or a certificate with overly broad principals being used where a more restricted one should apply. Moreover this information makes it easy to enforce the KRL offered with SSH certificates for easy revocation and ubiquitous remediation.
In environments using certificate-based authentication, SSHadow provides the visibility that makes the architecture actually secure rather than just operationally convenient.
Monitoring and Observability
SSHadow exposes three endpoints because different people need different views of the same data. The web dashboard at the root path gives you a human-readable interface with automatic refresh. This is useful for throwing up on a NOC display or for quick manual checks during an incident. It shows total active connections, per-user breakdowns with source IPs and authentication types, certificate key IDs and principals, and historical connection counts. Nothing fancy, just the information you need formatted readably.
The Prometheus metrics endpoint speaks the native format. You get counters for active connections, per-user active connections with labels for username, source IP, auth type, and key ID, plus total connection counters with the same labels. This integrates directly into existing monitoring infrastructure. It's possible to graph connection trends over time, alert when a single user has too many concurrent sessions, track authentication method distribution across your infrastructure, monitor certificate usage patterns to detect rotation issues.
The JSON export is for everything else such as feeding data to a SIEM, building custom dashboards, automated compliance reporting, or integration with other security tools. The structure is straightforward: each connection entry includes source IP, username, auth type, key ID (for certificates), principals, active count, total count, first seen timestamp, and last seen timestamp. Everything you need to build whatever analysis or visualization makes sense for your environment.
Detecting Attack Patterns
Real-time connection tracking naturally lends itself to attack detection. The monitoring feed shows patterns that indicate malicious activity, and the speed of detection depends on which mode you're running.
Volume-based attacks are the easiest to spot. Brute force attempts show up as rapid connection spikes from single source IPs with repeated authentication failures. Distributed attacks spread the attempts across multiple IPs but still create recognizable patterns in the aggregate connection data. Credential stuffing generates high volumes of authentication attempts with varied usernames but common passwords. Connection flooding attempts overwhelm the bastion with connection requests, which shows up immediately in the active connection counts.
Behavioral anomalies require more context but are often more interesting. For instance, if a user normally authenticates with certificates suddenly switches to password authentication. Connections during maintenance windows or outside normal business hours from users who don't have on-call responsibilities. Source IPs from geographic regions where your organization doesn't operate. Certificate key IDs that don't match your naming conventions or principals that include unexpected roles.
The hybrid monitoring mode is particularly effective for attack detection because it gives you two independent data streams. If you see authentication attempts in the logs that never made it through the proxy, someone's attacking the underlying SSH service directly. If the proxy shows connections but the logs are silent, something's bypassing normal authentication paths.
Live Attack Simulation
SSHadow includes a built-in attack simulator for testing your detection thresholds. Run go test -v -tags=livedemo -run TestLiveDemoSimulateAttack ./monitor/ and it spins up a temporary SSHadow instance, generates realistic attack traffic (brute force patterns, distributed attempts, various failure modes), and automatically opens your browser to the dashboard so you can watch the detection unfold in real-time.
This is useful for more than just demos. Before deploying SSHadow to production, run the simulator and tune your alert thresholds. It's fairly straightforward to adopt the simulator data to something more akin to your production environment. Train your security team on what attack patterns look like in the interface. Show executives or compliance auditors concrete evidence of detection capabilities. Validate that your SIEM integration is actually pulling the data and triggering the right alerts.
The simulator generates patterns that mirror real attacks but are not perfect replicas. Close enough to test your detection logic and familiarize people with what malicious traffic looks like in SSHadow's monitoring interface.
Performance and Security Considerations
Performance
Proxy mode adds about one millisecond of latency to connection establishment. Once the handshake completes, SSHadow is just forwarding bytes between sockets so there's no packet inspection or protocol parsing in the data path. Memory footprint sits around 10MB for the base process, scaling linearly with active connection count. CPU usage is negligible unless you're handling hundreds of simultaneous handshakes, at which point you've probably bottlenecked sshd anyway.
Logmon mode has zero impact on connection latency because it's not in the connection path at all. Memory usage is lower, around 5MB, plus whatever buffers are needed for tailing log files. CPU usage depends on log volume and parsing complexity, but even busy bastions rarely generate enough auth events to stress a modern processor.
Hybrid mode combines both overheads at around 15MB memory and the proxy latency. The trade-off is comprehensive visibility with redundant data collection. Whether that's worth the overhead depends on your threat model and operational requirements.
All three modes use proper concurrent locking and are safe for high-traffic environments. The code was written with production deployments in mind, not as a proof-of-concept.
Security Notes
Proxy mode sees authentication credentials during the handshake. This is inherent to how SSH works. To proxy a connection, you need to participate in the authentication exchange. SSHadow doesn't log passwords or private keys, but it does observe them in memory during the auth process. If this makes you uncomfortable, use logmon mode. If it doesn't bother you because you're using certificate authentication exclusively, proxy mode gives you better visibility.
Logmon mode only sees what's written to auth logs. It has read access to those files or to the journald socket, but it never touches credentials directly. The trade-off is slightly delayed detection and dependency on log rotation and write frequency. If your threat model includes insider threats with root access to the bastion, logmon mode doesn't help you because someone with root can modify logs before SSHadow sees them. Logmon is necessary for certificate monitoring because of the security challenge issued in certificate auth when proxying.
Clients connecting through proxy mode see SSHadow's host key, not your real sshd's host key. This is fine if you generate a new key for SSHadow and distribute it to clients. It's also fine if you copy your existing sshd host key to SSHadow so clients don't notice the difference. What's not fine is ignoring this entirely and letting clients blindly accept whatever key SSHadow presents which trains them to accept man-in-the-middle attacks. That being said, normal human hygiene around paying attention to host keys is near non-existent. This is a good teachable moment.
The metrics endpoint exposes usernames, source IPs, and connection patterns. This is sensitive data. Put it behind a firewall, use TLS, or at minimum restrict access to your monitoring network. Don't expose the metrics endpoint to the internet unless you enjoy explaining to your CISO why connection patterns for your entire infrastructure are publicly visible.
SSHadow can be run as a non-root user. If you need it to bind to privileged ports (anything below 1024), use systemd socket activation to hand off the bound socket to a non-privileged process. Restrict host key file permissions to 600. Standard operational security applies. Just because SSHadow is a monitoring tool doesn't mean it requires running with elevated privileges everywhere.
Upcoming Features
The current version handles basic connection monitoring and authentication tracking. The next major feature is SSH agent forwarding support, specifically the -A flag that forwards your local SSH agent through the bastion to downstream systems.
Agent forwarding creates an interesting security surface. When you connect with ssh -A, you're letting the bastion host use your SSH agent to authenticate to other systems. The bastion never sees your private keys, but it can request signatures from your agent. This is convenient for hopping through the bastion to production servers without copying keys around. It's also dangerous if the bastion is compromised.
SSHadow will intercept agent forwarding traffic and extract metadata about the keys being used downstream. When a certificate is forwarded through the agent, SSHadow will capture the key ID, principals, and serial number. This gives you visibility into the entire authentication chain: user connects to bastion with certificate A, then forwards agent to production server and authenticates with certificate B. It's also immune to manipulations of log files.
The implementation will support certificate middling by inspecting and logging certificate details as they flow through the agent channel. This isn't the same as modifying certificates or blocking specific key IDs, though that's certainly possible. The initial implementation focuses on visibility: knowing which certificates are being used where, building an audit trail of agent-forwarded authentication events.
Beyond agent forwarding, there are smaller features in the pipeline: session recording integration, geo-IP lookups for source addresses, connection duration tracking, idle session detection, configurable alerting thresholds without requiring external tools, and a database backend for storing historical data beyond what fits in memory.
Getting Started
SSHadow is open source and offers flexible deployment options:
# Clone the repository
git clone https://github.com/Splat/SSHadow
cd SSHadow
# Build
go build -o SSHadow .
# Generate a host key (for proxy/hybrid mode)
ssh-keygen -t ed25519 -f ssh_host_key -N ""Running in Proxy Mode
./SSHadow -mode proxy -hostkey ssh_host_key -listen :2222 -target localhost:22 -metrics :9090Running in Logmon Mode
# Monitor auth.log file
./SSHadow -mode logmon -logfile /var/log/auth.log -metrics :9090
# Or monitor journald
./SSHadow -mode logmon -journald -metrics :9090Running in Hybrid Mode
Try the Attack Simulator
# Run the live demo with simulated attacks
go test -v -tags=livedemo -run TestLiveDemoSimulateAttack ./monitor/
# Your browser will automatically open to http://localhost:9090
# Watch as the simulated attack unfolds in real-timeThe project includes:
- Comprehensive unit tests
- Docker Compose setup for quick testing
- Grafana dashboard templates
- Example Prometheus configuration
- systemd service files
Why SSHadow Matters
Modern infrastructure is complex. Users connect through bastion hosts to reach production systems. They use certificates issued by internal CAs. They forward SSH agents to hop through multiple layers.
Traditional SSH logging wasn't built for this world. SSHadow gives you the real-time visibility you need to:
- Understand who's accessing your infrastructure right now
- Track certificate usage and rotation
- Detect anomalous connection patterns
- Comply with audit requirements
- Investigate security incidents with concrete data
It's a simple tool that solves a real problem. No complicated deployment. No heavyweight dependencies. Just clean Go code that does one thing well: monitoring SSH connections with certificate awareness.
Conclusion
If you're running SSH bastion hosts and you don't have real-time visibility into who's connected and how they're authenticating, you're flying blind. Traditional logging gives you forensics after the fact. SSHadow gives you intelligence during the event.
The three operating modes (proxy, logmon, and hybrid) exist because different environments have different constraints. Some organizations can drop a proxy into the connection path. Others can't touch existing infrastructure. Some need defense-in-depth monitoring with correlation across multiple data sources. SSHadow doesn't force you into one approach.
Certificate authentication is finally gaining adoption in production, and SSHadow's certificate awareness makes that architecture actually auditable. Knowing which key IDs are in use, what principals they carry, and when they're being used transforms certificate-based auth from operationally convenient to properly secure.
The project is open source, written in Go, and focused on doing one thing well. If you need SSH connection monitoring for bastion hosts, it's worth evaluating. If you're already running certificate-based authentication, it's probably essential.
Check it out at github.com/Splat/SSHadow.
