limelight3a

name: limelight3a description: Expert guidance for using Limelight 3A vision camera in FTC robotics. Use when working with AprilTag detection, robot localization, MegaTag, pipeline configuration, coordinate transformations, or vision-based targeting.

Limelight 3A FTC Vision Skill

This skill provides comprehensive guidance for using the Limelight 3A vision camera in FTC robotics for AprilTag detection, robot localization, and vision-based targeting.

Overview

The Limelight 3A is a compact USB vision camera optimized for FTC with:

• OV5647 sensor: 640×480 @ 90 FPS
• Field of View: 54.5° horizontal, 42° vertical
• AprilTag tracking with MegaTag/MegaTag2 robot localization
• 10 hot-swappable pipelines for different vision tasks
• Web-based configuration UI at http://limelight.local:5801
• USB-C connection to Control Hub USB 3.0 port
• No LED illumination (rely on ambient light or external LEDs)

Power: 4.1V-5.75V via USB, max 4W consumption Weight: 0.20 lb Dimensions: 2.839" × 1.894" × 0.661"

FTC API Overview

Initialization

import com.qualcomm.hardware.limelightvision.Limelight3A
import com.qualcomm.hardware.limelightvision.LLResult
import com.qualcomm.hardware.limelightvision.LLResultTypes

// In init()
val limelight = hardwareMap.get(Limelight3A::class.java, "limelight")
limelight.pipelineSwitch(0)     // Switch to pipeline 0
limelight.setPollRateHz(100)    // Poll 100 times/sec
limelight.start()               // Start processing

Key Limelight3A Methods

LLResult Object

Always check validity first:

val result = limelight.latestResult
if (result != null && result.isValid) {
    // Use result data
}

IMPORTANT: getLatestResult() always returns an LLResult object, but isValid() will be false if no valid targets are detected. Always check isValid() before accessing data.

Basic Targeting Data:

result.tx              // Horizontal angle to target (degrees)
result.ty              // Vertical angle to target (degrees)
result.ta              // Target area (0-100% of image)
result.isValid         // true if data is valid
result.getStaleness()  // Age of data in milliseconds
result.pipelineIndex   // Current pipeline number

Robot Localization (MegaTag):

result.botpose              // Pose3D: MegaTag 1 position
result.botpose_MT2          // Pose3D: MegaTag 2 with IMU fusion
result.botposeTagCount      // Number of AprilTags used for pose

// Access pose components:
val pose = result.botpose
pose.position.x             // X position (meters)
pose.position.y             // Y position (meters)
pose.position.z             // Z position (meters)
pose.orientation.yaw        // Yaw angle (degrees)
pose.orientation.pitch      // Pitch angle (degrees)
pose.orientation.roll       // Roll angle (degrees)

AprilTag Detection:

result.fiducialResults      // List<FiducialResult>

for (tag in result.fiducialResults) {
    tag.fiducialId          // AprilTag ID
    tag.targetXDegrees      // Horizontal angle to tag
    tag.targetYDegrees      // Vertical angle to tag
    tag.targetArea          // Tag area percentage
    tag.robotPose           // Pose3D of robot relative to tag
    tag.targetPose          // Pose3D of tag relative to camera
}

Other Result Types:

result.colorResults         // Color blob detection
result.barcodeResults       // QR code/barcode data
result.classifierResults    // Neural network classifications
result.detectorResults      // Object detection bounding boxes
result.pythonOutput         // Data from Python pipelines (double[])

Pipeline Configuration

Accessing Web UI

1. Connect Limelight via USB to Control Hub
2. Connect Driver Station to Control Hub WiFi
3. Navigate to http://limelight.local:5801 in browser
4. Or use Limelight Hardware Manager application

Pipeline Types

Limelight 3A supports 10 unique pipelines (0-9):

1. AprilTag Pipelines - Fiducial marker detection
2. Color Pipelines - Color blob tracking (90 FPS)
3. Neural Network Pipelines - CPU inference (~10 FPS)
4. Python Pipelines - Custom OpenCV processing
5. Barcode Pipelines - QR code/barcode reading

AprilTag Pipeline Configuration

Input Tab Settings:

Fiducial Tab Settings:

3D Settings:

• Enable "Full 3D" mode for robot localization
• Configure camera pose relative to robot center
• Upload field map with AprilTag locations

Pipeline Switching

Switch pipelines dynamically for different tasks:

// Pipeline 0: AprilTag localization
limelight.pipelineSwitch(0)

// Pipeline 1: Sample detection (color)
limelight.pipelineSwitch(1)

// Pipeline 2: Specimen detection
limelight.pipelineSwitch(2)

Robot Localization with MegaTag

MegaTag vs MegaTag2

MegaTag 1 (botpose):

• Uses only AprilTag detections
• No IMU fusion
• Works with 1+ tags visible
• More stable with multiple tags

MegaTag2 (botpose_MT2):

• Fuses AprilTag data with IMU
• Requires updateRobotOrientation(yaw) calls
• More accurate with IMU fusion
• Resilient to temporary tag loss

Recommendation: Use MegaTag2 for best results in FTC.

Setting Up Localization

1. Configure Camera Pose in Web UI:

• Navigate to "3D" tab in AprilTag pipeline
• Enter camera position relative to robot center
• Example: X=0, Y=0, Z=0.2m (20cm above center)
• Enter camera orientation (pitch, yaw, roll)

2. Upload Field Map:

• Use pre-made FTC field maps from Limelight
• Or create custom map with AprilTag positions
• Upload via "Field Map" button in web UI

3. Enable in Code:

// Update IMU heading every loop
limelight.updateRobotOrientation(imu.robotYawPitchRollAngles.getYaw(AngleUnit.DEGREES))

val result = limelight.latestResult
if (result != null && result.isValid) {
    if (result.botposeTagCount >= 2) {  // Use 2+ tags for reliability
        val pose = result.botpose_MT2    // MegaTag2 with IMU fusion

        // Use pose for odometry correction
        val x = pose.position.x
        val y = pose.position.y
        val heading = Math.toRadians(pose.orientation.yaw)
    }
}

Coordinate System Transformations

Limelight Coordinate System (from camera perspective):

• X-axis: Right is positive
• Y-axis: Down is positive
• Z-axis: Forward (away from camera) is positive

FTC Field Coordinate System (standard):

• X-axis: Varies by alliance
• Y-axis: Varies by alliance
• Heading: Varies by starting orientation

Common Transformation Pattern (from this codebase):

// Limelight returns pose in field coordinates
val botX = -botpose.position.y  // Swap and negate Y
val botY = botpose.position.x   // Use X
val botRot = ((botpose.orientation.yaw * Math.PI) / 180.0) - Math.PI  // Convert to radians, adjust

Important: Always test coordinate transformations empirically. Place robot at known position, read Limelight pose, and adjust transformation until it matches.

Integration Patterns

Pattern 1: Motif Detection (Game Element Patterns)

Detect specific AprilTag IDs to determine game element configuration:

enum class Motif {
    GPP,  // Green-Purple-Purple
    PGP,  // Purple-Green-Purple
    PPG   // Purple-Purple-Green
}

fun detectMotif(fiducialResults: List<LLResultTypes.FiducialResult>): Motif {
    var currentMotif = Motif.GPP  // Default

    for (tag in fiducialResults) {
        currentMotif = when (tag.fiducialId) {
            21 -> Motif.GPP
            22 -> Motif.PGP
            23 -> Motif.PPG
            else -> currentMotif
        }
    }

    return currentMotif
}

Pattern 2: Continuous Localization Update

Update robot odometry with vision data:

class LimelightIO(val limelight: Limelight3A) {
    var localization = Pose2D(DistanceUnit.METER, 0.0, 0.0, AngleUnit.RADIANS, 0.0)
        private set

    var localizationTags = 0
        private set

    fun run(imuYaw: Double) {
        // Provide IMU for MegaTag2
        limelight.updateRobotOrientation(imuYaw)

        val result = limelight.latestResult
        if (result != null && result.isValid) {
            localizationTags = result.botposeTagCount

            // Only use pose if 2+ tags visible (more reliable)
            if (result.botposeTagCount >= 2) {
                updateBotPosition(result.botpose_MT2)
            }
        }
    }

    private fun updateBotPosition(botpose: Pose3D) {
        // Transform from Limelight to field coordinates
        val botX = -botpose.position.y
        val botY = botpose.position.x
        val botRot = ((botpose.orientation.yaw * Math.PI) / 180.0) - Math.PI

        localization = Pose2D(DistanceUnit.METER, botX, botY, AngleUnit.RADIANS, botRot)
    }
}

Pattern 3: Vision-Assisted Targeting

Use tx/ty for alignment assistance:

fun isAlignedWithTarget(result: LLResult, toleranceDegrees: Double = 2.0): Boolean {
    if (!result.isValid) return false
    return abs(result.tx) < toleranceDegrees
}

fun getSteeringCorrection(result: LLResult, kP: Double = 0.05): Double {
    if (!result.isValid) return 0.0
    return -result.tx * kP  // Proportional control
}

Pattern 4: Odometry Fusion with Vision

Combine vision with odometry using weighted averaging or Kalman filter:

fun fuseVisionWithOdometry(
    odomPose: Pose2D,
    visionPose: Pose2D,
    visionWeight: Double = 0.15  // 15% vision, 85% odometry
): Pose2D {
    val fusedX = odomPose.x * (1 - visionWeight) + visionPose.x * visionWeight
    val fusedY = odomPose.y * (1 - visionWeight) + visionPose.y * visionWeight

    // Heading fusion (handle wrapping)
    val odomHeading = odomPose.heading
    val visionHeading = visionPose.heading
    val headingDiff = normalizeAngle(visionHeading - odomHeading)
    val fusedHeading = normalizeAngle(odomHeading + headingDiff * visionWeight)

    return Pose2D(DistanceUnit.METER, fusedX, fusedY, AngleUnit.RADIANS, fusedHeading)
}

Best Practices

Performance Optimization

1. Use appropriate resolution: 320×240 for 2D detection (90 FPS), 640×480 for 3D localization
2. Minimize exposure: Reduce motion blur during robot movement (start at 1-2 ms)
3. Increase detector downscale: Trades range for framerate (1.5-2.0 works well)
4. Filter by tag count: Only use pose estimates with 2+ tags for reliability
5. Set poll rate: 100 Hz is good balance for FTC (don't exceed loop rate)

Reliability

1. Always check isValid() before accessing result data
2. Check staleness: Reject old data (getStaleness() > 50ms)
3. Validate tag count: More tags = more reliable pose (botposeTagCount >= 2)
4. Handle missing targets: Have fallback behavior when no tags detected
5. Test transformations: Empirically verify coordinate transformations

Camera Mounting

1. Height: Mount above or below tag height for better detection
2. Angle: Slight tilt (10-20°) prevents head-on ambiguity
3. Stable mount: Minimize vibration and movement
4. USB routing: Ensure USB-C cable won't disconnect during movement
5. Field of view: Position to see multiple tags simultaneously

Debugging

1. Use web UI: Live view shows detections in real-time
2. Capture snapshots: Save problem frames for offline analysis
3. Check telemetry: Monitor tx, ty, tag count, staleness
4. Verify field map: Ensure AprilTag positions match physical field
5. Test camera pose: Verify robot-to-camera transform is correct

Common Issues and Solutions

Issue: No AprilTags Detected

Solutions:

• Check lighting (too bright or too dark)
• Reduce exposure to minimize motion blur
• Increase sensor gain for low light
• Verify tag size configuration matches physical tags
• Check ID filters aren't excluding your tags
• Ensure tags are flat and not damaged

Issue: Unreliable Pose Estimates

Solutions:

• Only use poses with 2+ tags (botposeTagCount >= 2)
• Check camera pose configuration in web UI
• Verify field map matches physical tag locations
• Use MegaTag2 with IMU fusion (updateRobotOrientation())
• Reject poses with low tag count or high staleness
• Implement outlier rejection (check if pose jumped >1m)

Issue: Wrong Coordinate Transformation

Solutions:

• Test transformation empirically (place robot at known position)
• Check coordinate system documentation
• Verify camera orientation in robot pose config
• Use telemetry to debug X, Y, heading separately
• Consider alliance-specific transformations

Issue: Low Frame Rate

Solutions:

• Use lower resolution (320×240 instead of 640×480)
• Increase detector downscale (1.5-2.0)
• Reduce number of tags in ID filter
• Use simpler pipeline type (AprilTag is faster than neural nets)
• Check USB connection quality

Issue: Tags Detected But Wrong IDs

Solutions:

• Increase quality threshold to filter spurious detections
• Use ID filters to restrict to expected tags
• Check tag family setting (should be 36h11 for FTC)
• Ensure tags are correctly printed/sized
• Improve lighting conditions

Integration with This Codebase

Current Architecture

LimelightIO.kt - Abstraction layer:

class LimelightIO(limelight3A: Limelight3A, physicalHardware: HardwareIO) {
    // Manages:
    // - Motif detection (GPP, PGP, PPG)
    // - Robot localization from AprilTags
    // - Coordinate transformations
    // - IMU fusion via updateRobotOrientation()
}

Limelight.kt - Subsystem:

class Limelight(robot: Robot): SubsystemBase() {
    // Periodic updates:
    // - Displays detected motif in telemetry
    // - Can provide vision measurements to odometry
}

HardwareIO.kt - Hardware interface:

override fun limelightDetectedMotif(): Limelight.Motif
override fun limelightLocalizationAprilTagCount(): Int
override fun limelightLocalization(): Pose2D

Usage in Autonomous

// In FarAuto.kt
val motif = robot.hardware.limelightDetectedMotif()
ShootMotif(robot.spiceRack, motif)

Usage in TeleOp

Display targeting info:

telemetry.addData("Limelight Tags", hardware.limelightLocalizationAprilTagCount())
telemetry.addData("Motif", hardware.limelightDetectedMotif())

Resources

• Official Docs: https://docs.limelightvision.io/docs/docs-limelight/getting-started/limelight-3a
• FTC API Guide: https://docs.limelightvision.io/docs/docs-limelight/apis/ftc-programming
• AprilTag Detection: https://docs.limelightvision.io/docs/docs-limelight/pipeline-apriltag/apriltags
• Robot Localization: https://docs.limelightvision.io/docs/docs-limelight/pipeline-apriltag/apriltag-robot-localization
• Pipeline Setup: https://docs.limelightvision.io/docs/docs-limelight/getting-started/pipelines
• Web UI: http://limelight.local:5801

Quick Reference

Remember:

• Always check result.isValid before accessing data
• Use 2+ tags for reliable localization
• Update IMU heading for MegaTag2
• Test coordinate transformations empirically
• Access web UI at http://limelight.local:5801