hcovguard

A coverage ratchet tool for Haskell projects

hcovguard reads HPC-generated coverage data and checks it against configurable thresholds defined in a TOML file.

Quick Start

# 1. Run your tests with coverage enabled
stack test --coverage
# or: cabal test --enable-coverage

# 2. Create a minimal config file
cat > hcovguard.toml << 'EOF'
[forAnyModule]

[forAnyModule.expression]
minimumCovered = 1
EOF

# 3. Run hcovguard with auto-discovery
hcovguard --tix $(stack path --local-hpc-root)/combined/all/all.tix --auto-discover

# 4. Adjust thresholds based on your current coverage and goals

Features

• Configurable Thresholds: Set minimum covered counts or maximum uncovered counts for expressions, top-level bindings, alternatives, and local bindings
• Per-Module Overrides: Override thresholds for specific modules by exact name or glob patterns
• Pattern Matching: Use * (single level) and ** (recursive) glob patterns to match multiple modules
• Ignore Modules: Mark test helpers or generated code to be excluded from checks
• TOML Configuration: Simple, readable configuration format
• CI-Friendly: Returns non-zero exit code when thresholds aren't met
• Configuration Validation: Warns when patterns don't match any modules, helping catch typos and stale config

Installation

From Source

cabal install hcovguard

Or with Stack:

stack install hcovguard

Installing the Manual Page

The man page is included in the package and can be installed manually:

# Install the man page system-wide (typically requires root access)
mkdir -p /usr/local/share/man/man1
cp man/hcovguard.1 /usr/local/share/man/man1/
mandb  # Update man database (on some systems)

# Then view it with:
man hcovguard

To view the man page without installing, run from the repository:

man man/hcovguard.1

Usage

First, generate coverage data by running your tests with coverage enabled:

# With Stack
stack test --coverage

# With Cabal
cabal test --enable-coverage

Then run hcovguard against the generated .tix file:

hcovguard \
  --tix $(stack path --local-hpc-root)/combined/all/all.tix \
  --auto-discover \
  --config hcovguard.toml

Auto-Discovery of Mix Directories

The --auto-discover flag automatically searches for .mix files in common build output directories:

• .stack-work/ (Stack)
• dist-newstyle/ (Cabal)
• .hpc/ (direct HPC output)

This is the recommended approach for most projects, as it eliminates the need to manually specify --mix-dir paths that change between platforms and GHC versions.

If you need to specify mix directories manually (e.g., for non-standard build configurations), use --mix-dir:

hcovguard \
  --tix path/to/coverage.tix \
  --mix-dir /custom/path/to/mix \
  --mix-dir /another/mix/directory

Command Line Options

Note: You must specify either --auto-discover or at least one --mix-dir. If neither is provided, hcovguard will exit with an error.

Generating a Baseline Config

Use --baseline to generate a configuration file from your current coverage:

hcovguard --tix coverage.tix --auto-discover --baseline > hcovguard.toml

This outputs TOML with thresholds set to your current coverage counts—useful for:

• Initial setup when adopting hcovguard
• Resetting thresholds after a major refactor
• Establishing a new baseline after significant changes

Dry Run Mode

Use --dry-run to see which patterns match which modules without running threshold checks:

hcovguard --tix coverage.tix --auto-discover --dry-run

Example output:

MyProject.Core: matched rule #1 (module = "MyProject.Core")
MyProject.Utils: matched rule #2 (pattern = "MyProject.*")
MyProject.Internal.Helpers: matched rule #3 (pattern = "**.Internal.**") (ignored)
Test.MyProject.CoreSpec: using [forAnyModule] defaults

This helps debug pattern matching issues and verify your configuration.

Configuration

Create a hcovguard.toml file in your project root, for example:

# Default thresholds for all modules
[forAnyModule]

[forAnyModule.expression]
minimumCovered = 50
maximumUncovered = 10

[forAnyModule.topLevel]
minimumCovered = 30

[forAnyModule.alternative]
maximumUncovered = 5

# Override thresholds for specific modules (exact match)
[[forSpecifiedModules]]
module = "MyProject.Core"

[forSpecifiedModules.expression]
minimumCovered = 100

[forSpecifiedModules.topLevel]
minimumCovered = 50

# Override thresholds using pattern matching
[[forSpecifiedModules]]
pattern = "MyProject.Internal.**"

[forSpecifiedModules.expression]
minimumCovered = 20

# Ignore specific modules
[[forSpecifiedModules]]
module = "Test.Helpers"
ignore = true

# Ignore modules matching a pattern
[[forSpecifiedModules]]
pattern = "**.Test.**"
ignore = true

Threshold Options

Thresholds control the minimum coverage requirements. Both options are optional and can be used independently or together.

Coverage Categories

Thresholds can be set for four coverage categories, each as a nested table:

[forAnyModule]

Default thresholds applied to all modules unless overridden by [[forSpecifiedModules]].

Example:

[forAnyModule]

[forAnyModule.expression]
minimumCovered = 50

[forAnyModule.topLevel]
maximumUncovered = 5

[[forSpecifiedModules]]

Per-module overrides. Each entry uses one of two forms:

Form 1: Exact Module Match

Match a specific module by its exact name.

Example:

[[forSpecifiedModules]]
module = "MyProject.Core"

[forSpecifiedModules.expression]
minimumCovered = 100

Form 2: Pattern Match

Match multiple modules using glob patterns.

Example:

[[forSpecifiedModules]]
pattern = "MyProject.Internal.**"

[forSpecifiedModules.expression]
minimumCovered = 20

Note: Each [[forSpecifiedModules]] entry must use exactly one form. Specifying both module and pattern is an error.

Pattern Syntax

• * matches any characters except . (single module level)
• ** matches any characters including . (multiple module levels)

Examples:

• MyProject.* matches MyProject.Foo but not MyProject.Foo.Bar
• MyProject.** matches MyProject.Foo, MyProject.Foo.Bar, etc.
• **.Internal matches Foo.Internal, Foo.Bar.Internal, etc.

Pattern Match Order

When multiple [[forSpecifiedModules]] entries could match a module, the first matching entry wins. Entries are checked in the order they appear in the configuration file.

This means you should order your patterns from most specific to least specific:

# CORRECT: Specific patterns first
[[forSpecifiedModules]]
module = "MyProject.Core.Critical"  # Exact match checked first
[forSpecifiedModules.expression]
minimumCovered = 200

[[forSpecifiedModules]]
pattern = "MyProject.Core.*"        # Then specific pattern
[forSpecifiedModules.expression]
minimumCovered = 100

[[forSpecifiedModules]]
pattern = "MyProject.**"            # Then broad pattern
[forSpecifiedModules.expression]
minimumCovered = 50

If no [[forSpecifiedModules]] entry matches, the [forAnyModule] defaults apply.

Why hcovguard?

• Count-Based Thresholds: Set minimum covered counts or maximum uncovered counts rather than percentages. This makes coverage ratcheting straightforward - incrementally require more coverage as you improve your test suite, without worrying about percentage fluctuations as code is added or removed.

• Per-Module Configuration: Different parts of your codebase deserve different standards. Require higher coverage for core business logic, lower thresholds for internal utilities, and ignore generated code entirely.

• Glob Pattern Matching: Configure thresholds for entire module hierarchies with patterns like MyProject.Internal.** instead of listing every module individually.

• Pure Haskell: No C library dependencies means straightforward builds across platforms and GHC versions.

Troubleshooting

"Mix file not found" error

HPC generates two types of files:

• .tix files contain coverage counts (which code paths were executed)
• .mix files contain coverage metadata (mapping counts to source locations)

If you see this error, hcovguard found the .tix file but cannot locate the corresponding .mix files. Solutions:

1. Use --auto-discover: This searches common build directories automatically.

2. Check your build output: Mix files are typically located in:

   • Stack: .stack-work/install/<platform>/<snapshot>/hpc/
   • Cabal: dist-newstyle/build/<platform>/<compiler>/<package>/hpc/

3. Verify mix files exist: Run find . -name "*.mix" to locate them, then use --mix-dir with the containing directory.

4. Rebuild with coverage: Mix files are only generated when building with coverage enabled. Re-run stack test --coverage or cabal test --enable-coverage.

No modules checked / empty output

This typically means:

1. Wrong tix file path: The .tix file might be empty or point to a different test run. Verify the file exists and has recent modification time.

2. Module name mismatch: The modules in your .tix file don't match any patterns in your config. Run with --verbosity 2 to see which modules are being processed.

3. All modules ignored: Check that your ignore = true patterns aren't too broad.

Pattern not matching expected modules

Remember:

• * matches within a single module level (stops at .)
• ** matches across multiple levels (includes .)

Common mistakes:

• Using MyProject.* when you meant MyProject.** (the former won't match MyProject.Foo.Bar)
• Using *.Test when you meant **.Test (the former only matches Foo.Test, not Foo.Bar.Test)

Run with --verbosity 2 to see exactly which modules match which patterns.

Coverage counts seem wrong

HPC counts can be surprising because:

• Expressions are very granular (each subexpression counts separately)
• Alternatives include both case branches AND guard clauses
• Local bindings include let and where bindings

Use --verbosity 2 to see the actual counts per category for each module, then adjust your thresholds accordingly.

"No .mix directories specified" error

This error occurs when you don't specify how hcovguard should find .mix files. You must either:

1. Use --auto-discover to search common build directories automatically, or
2. Specify at least one --mix-dir path explicitly

"Pattern matched no modules" warning

If you see a warning like:

Warning: [Hcovguard-8451] Pattern matched no modules
  pattern = "MyProject.Internal.**"

This means a [[forSpecifiedModules]] entry in your config didn't match any modules in the .tix file. This often indicates:

1. Typo in the pattern: Double-check the module name or pattern syntax.
2. Modules not included in coverage: The modules might not have been compiled with coverage enabled.
3. Stale configuration: The modules may have been renamed or removed.

This is a warning, not an error — hcovguard will continue checking other modules. However, you should review your configuration to ensure patterns are intentional.

License

MIT License - see LICENSE for details.

Contributing

Contributions are welcome! Please feel free to submit issues and pull requests.