Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
The table of contents is too big for display.
Diff view
Diff view
  •  
  •  
  •  
24 changes: 24 additions & 0 deletions CREDITS.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,24 @@
# Credits — Demo & Test Images

The sample images used in the iOS demo app and the Kit test fixtures are
**CC0 1.0 (Public Domain Dedication)** images from Wikimedia Commons. CC0
requires no attribution and permits commercial use; the sources are listed here
as a courtesy.

| File | Source (Wikimedia Commons) | License |
|---|---|---|
| `cc0_cherry_blossom` | [Brightin Star 35mm f0.95 - Cherry Blossom](https://commons.wikimedia.org/wiki/File:Brightin_Star_35mm_f0.95_-_Cherry_Blossom.jpg) | CC0 1.0 |
| `cc0_flower_bokeh` | [Beautiful Flower covered in an artistic Bokeh](https://commons.wikimedia.org/wiki/File:Beautiful_Flower_covered_in_an_artistic_Bokeh_(54028221085).jpg) | CC0 1.0 |
| `cc0_ant_sunflower` | [Ant on sunflower with snail](https://commons.wikimedia.org/wiki/File:Ant_on_sunflower_with_snail.jpg) | CC0 1.0 |
| `cc0_highway_bokeh` | [Highway Bokeh](https://commons.wikimedia.org/wiki/File:Highway_Bokeh_(49256321736).jpg) | CC0 1.0 |
| `cc0_misty_river` | [Blurry misty river](https://commons.wikimedia.org/wiki/File:Blurry_misty_river.jpg) | CC0 1.0 |
| `cc0_street_bokeh` | [Street bokeh (Unsplash)](https://commons.wikimedia.org/wiki/File:Street_bokeh_(Unsplash).jpg) | CC0 1.0 |
| `cc0_morning_coffee` | [Morning coffee (Unsplash)](https://commons.wikimedia.org/wiki/File:Morning_coffee_(Unsplash).jpg) | CC0 1.0 |
| `cc0_love_in_a_cup` | [Love in a cup (Unsplash)](https://commons.wikimedia.org/wiki/File:Love_in_a_cup_(Unsplash).jpg) | CC0 1.0 |
| `cc0_dew_grass` | [Dew on Grass, beautiful greenery](https://commons.wikimedia.org/wiki/File:Dew_on_Grass,_beautiful_greenery.jpg) | CC0 1.0 |
| `cc0_panning_gulls` | [Panning gulls](https://commons.wikimedia.org/wiki/File:Panning_gulls.jpg) | CC0 1.0 |
| `cc0_clover_closeup` | [Trifolium subterraneum close-up](https://commons.wikimedia.org/wiki/File:Trifolium_subterraneum_close-up.jpg) | CC0 1.0 |
| `cc0_icicles` | [Icicles on a winter morning](https://commons.wikimedia.org/wiki/File:Icicles_on_a_winter_morning_(30982282464).jpg) | CC0 1.0 |

Images were downscaled to 1024 px (longest side) for use in the demo; no other
modifications were made to their licensing status.
3,664 changes: 38 additions & 3,626 deletions Demo/iOS/BlurDiscriminatorKit/BlurDiscriminatorKit.xcodeproj/project.pbxproj

Large diffs are not rendered by default.

Original file line number Diff line number Diff line change
@@ -1,48 +1,99 @@
// SPDX-License-Identifier: Apache-2.0
//
// BlurDiscriminator.swift
// BlurDiscriminatorKit
//
// Created by syjdev on 2021/09/23.
//

import Foundation
import CoreGraphics
import BlurDiscriminatorKit.Private
import Foundation

/// An error thrown by ``BlurDiscriminator/predict(input:)``.
public enum BlurDiscriminatorError: Error {
/// The Core ML model failed to load from the framework bundle.
case modelLoadFailed(underlying: Error)
/// The input image could not be prepared (letterboxed) for the model.
case imagePreprocessingFailed
/// The Vision request failed while running the model.
case inferenceFailed(underlying: Error)
/// The model produced an output the framework does not understand.
case unexpectedOutput
}

public final class BlurDiscriminator: @unchecked Sendable {
private let interpreter: InterpreterWrapper
private let inputConverter: InputConverter = RGBConverter(inputWidth: Constants.inputWidth,
inputHeight: Constants.inputHeight)
private let outputConverter: OutputConverter = GrayscaleConverter(inputWidth: Constants.inputWidth,
inputHeight: Constants.inputHeight)
/// What ``BlurDiscriminator`` needs from its inference engine — a seam for
/// injecting a test double in place of the Core ML-backed predictor.
internal protocol BlurPredicting: Sendable {
func predict(input: CGImage) throws -> BlurObservation
}

private let inferenceQueue = DispatchQueue(label: "com.blurdiscriminatorkit.inference", qos: .userInitiated)
/// A deep-learning based detector that classifies the clear and blurry regions of an image.
///
/// `BlurDiscriminator` wraps a Core ML segmentation model (`BlurSegmentation.mlpackage`, bundled
/// inside this framework). Given a `CGImage`, it produces a ``BlurObservation`` describing, per
/// pixel, how blurry the input is.
///
/// The input image is letterboxed to the model's fixed input size (currently 224×224) internally —
/// aspect ratio is preserved with black padding — and the resulting blur map is cropped back to the
/// original image's aspect ratio. Any image size may be passed. `BlurDiscriminator` is an actor:
/// concurrent calls are serialized by actor isolation, so a single instance is safe to share across
/// concurrency domains.
///
/// ```swift
/// let discriminator = BlurDiscriminator()
///
/// let observation = try await discriminator.predict(input: cgImage)
/// imageView.image = UIImage(cgImage: observation.blurMap)
/// ```
public actor BlurDiscriminator {
private let loadPredictor: @Sendable () async throws -> any BlurPredicting
private var predictor: (any BlurPredicting)?
private var loadTask: Task<any BlurPredicting, Error>?

public init(modelPath: String, numberOfThread: UInt8) {
interpreter = InterpreterWrapper(modelPath: modelPath, andNumberOfThread: numberOfThread)
/// Creates a discriminator backed by the bundled Core ML model.
///
/// Creation is cheap: the model is loaded lazily on the first call to ``predict(input:)``,
/// so initializing the discriminator never blocks the calling thread.
public init() {
self.loadPredictor = { try await CoreMLBlurPredictor() }
}

public func predict(input: CGImage) -> BlurObservation? {
return inferenceQueue.sync { runPrediction(input: input) }
/// Test seam: creates a discriminator that loads its predictor from the given closure.
internal init(loadPredictor: @escaping @Sendable () async throws -> any BlurPredicting) {
self.loadPredictor = loadPredictor
}

public func predict(input: CGImage) async -> BlurObservation? {
return await withCheckedContinuation { continuation in
inferenceQueue.async { [self] in
continuation.resume(returning: runPrediction(input: input))
}
}
/// Runs blur detection on the given image.
///
/// The Core ML model is loaded on the first call and cached for subsequent calls.
/// Inference is actor-isolated: concurrent calls are executed one at a time, and the
/// calling task is suspended (not blocked) until inference completes.
///
/// - Parameter input: The image to analyze. It is resized to the model input size internally.
/// - Returns: A ``BlurObservation`` describing the blur map.
/// - Throws: ``BlurDiscriminatorError`` if the model could not be loaded, inference failed,
/// or the model produced an unexpected output.
public func predict(input: CGImage) async throws -> BlurObservation {
return try await loadedPredictor().predict(input: input)
}

private func runPrediction(input: CGImage) -> BlurObservation? {
guard let convertedData = inputConverter.convert(cgImage: input),
let outputData = interpreter.interpret(withInputData: convertedData)
else {
return nil
/// Returns the cached predictor, loading the model on first use. A single in-flight load
/// task is shared between reentrant callers so the model is never loaded twice; a failed
/// load is not cached, so a later call may retry.
private func loadedPredictor() async throws -> any BlurPredicting {
if let predictor {
return predictor
}
if let loadTask {
return try await loadTask.value
}

let task = Task { [loadPredictor] in try await loadPredictor() }
loadTask = task
defer { loadTask = nil }

return outputConverter.convert(data: outputData,
originImageWidth: input.width,
originImageHeight: input.height)
let loaded = try await task.value
predictor = loaded
return loaded
}
}
87 changes: 87 additions & 0 deletions Demo/iOS/BlurDiscriminatorKit/BlurDiscriminatorKit/BlurMap.swift
Original file line number Diff line number Diff line change
@@ -0,0 +1,87 @@
// SPDX-License-Identifier: Apache-2.0
//
// BlurMap.swift
// BlurDiscriminatorKit
//
// Helpers that turn the Core ML sigmoid output into an 8-bit grayscale blur map.
//

import Accelerate
import CoreGraphics
import Foundation


internal enum PixelConverter {
/// Converts a float sigmoid buffer in [0, 1] to UInt8 [0, 255] with vDSP:
/// scale by 255, clip to the valid range, then round-to-nearest into UInt8.
/// Replaces per-pixel `UInt8(max(0, min(255, v * 255)))` loops.
static func floatToUInt8<Buffer: AccelerateBuffer>(_ source: Buffer) -> [UInt8] where Buffer.Element == Float {
let scaled = vDSP.clip(vDSP.multiply(255, source), to: 0 ... 255)

var pixels = [UInt8](repeating: 0, count: scaled.count)
vDSP.convertElements(of: scaled, to: &pixels, rounding: .towardNearestInteger)
return pixels
}
}


internal enum BlurMapGeometry {
/// The un-padded region of a letterboxed square output, matching the original image's aspect
/// ratio. Mirrors the top-left aspect-fit placement used by ``CGImage/makeAspectFitPadded(width:height:)``,
/// so the returned size is the slice of the model output that corresponds to real image content.
static func croppedSize(outputWidth: Int, outputHeight: Int,
originalWidth: Int, originalHeight: Int) -> (width: Int, height: Int) {
guard originalWidth > 0, originalHeight > 0 else { return (outputWidth, outputHeight) }

if originalWidth > originalHeight {
return (outputWidth, Int(Float(outputHeight) * Float(originalHeight) / Float(originalWidth)))
} else if originalWidth < originalHeight {
return (Int(Float(outputWidth) * Float(originalWidth) / Float(originalHeight)), outputHeight)
} else {
return (outputWidth, outputHeight)
}
}

/// Crops the top-left `cropWidth`×`cropHeight` block out of a row-major, `sourceWidth`-wide map.
static func cropTopLeft(_ pixels: [UInt8], sourceWidth: Int,
cropWidth: Int, cropHeight: Int) -> [UInt8] {
guard sourceWidth > 0 else { return pixels }
let sourceHeight = pixels.count / sourceWidth
guard cropWidth < sourceWidth || cropHeight < sourceHeight else { return pixels }

var cropped = [UInt8]()
cropped.reserveCapacity(cropWidth * cropHeight)
for row in 0 ..< cropHeight {
let start = row * sourceWidth
cropped.append(contentsOf: pixels[start ..< start + cropWidth])
}
return cropped
}
}


internal enum BlurMapRenderer {
/// Builds a grayscale CGImage from width×height UInt8 pixels in row-major order.
static func makeBlurMap(grayscaledPixels: [UInt8], width: Int, height: Int) -> CGImage? {
guard width > 0, height > 0, grayscaledPixels.count == width * height else { return nil }

// Keep every use of the pixel buffer inside the closure: `makeImage()` copies the
// bitmap, so the context never sees the pointer beyond this scope. (Passing `&pixels`
// straight to `CGContext(data:)` would leave the context holding a pointer that is
// only valid for the duration of the initializer call.)
var pixels = grayscaledPixels
return pixels.withUnsafeMutableBytes { buffer -> CGImage? in
guard let context = CGContext(
data: buffer.baseAddress,
width: width,
height: height,
bitsPerComponent: 8,
bytesPerRow: width,
space: CGColorSpaceCreateDeviceGray(),
bitmapInfo: CGImageAlphaInfo.none.rawValue
) else { return nil }

return context.makeImage()
}
}
}
Original file line number Diff line number Diff line change
@@ -1,34 +1,44 @@
// SPDX-License-Identifier: Apache-2.0
//
// BlurObservation.swift
// BlurDiscriminatorKit
//
// Created by syjdev on 2021/11/03.
//

import Foundation
import CoreGraphics


public final class BlurObservation {
/// The result of running ``BlurDiscriminator/predict(input:)`` on an image.
///
/// A `BlurObservation` describes, per pixel, how blurry the analyzed image is. Pixel values
/// range from `0` (clear) to `255` (blurry), matched to the original image's dimensions.
public struct BlurObservation: Sendable {
/// A grayscale blur map sized to the original image, where brighter pixels are blurrier.
///
/// Suitable for direct display, e.g. `UIImage(cgImage: observation.blurMap)`.
public let blurMap: CGImage

/// The raw per-pixel blur intensities of ``blurMap``, from `0` (clear) to `255` (blurry),
/// in row-major order.
public let grayscaledPixels: [UInt8]
public let data: Data

internal init(blurMap: CGImage,
grayscaledPixels: [UInt8],
data: Data = Data()) {
internal init(blurMap: CGImage, grayscaledPixels: [UInt8]) {
self.blurMap = blurMap
self.grayscaledPixels = grayscaledPixels
self.data = data
}

public func blurRatio(threshold: Int) -> Float {
let numberOfBlur: Float = grayscaledPixels.reduce(into: 0) { result, element in
if element > threshold {
result += 1
}
}

return numberOfBlur / Float(grayscaledPixels.count)

/// The fraction of pixels considered blurry.
///
/// A pixel counts as blurry when its intensity in ``grayscaledPixels`` is strictly greater
/// than `threshold`. Use the returned ratio to decide whether an image is blurry overall,
/// e.g. treat it as blurry when the ratio exceeds some cutoff.
///
/// - Parameter threshold: The intensity cutoff; higher values count fewer pixels as blurry.
/// Defaults to `127`, the midpoint of the intensity range.
/// - Returns: A value in `0...1` — the number of blurry pixels divided by the total.
public func blurRatio(threshold: UInt8 = 127) -> Float {
let numberOfBlur = grayscaledPixels.count { $0 > threshold }
return Float(numberOfBlur) / Float(grayscaledPixels.count)
}
}
Binary file not shown.
Binary file not shown.
Original file line number Diff line number Diff line change
@@ -0,0 +1,18 @@
{
"fileFormatVersion": "1.0.0",
"itemInfoEntries": {
"15F4E827-89F3-4D2A-AFF3-A1609A536263": {
"author": "com.apple.CoreML",
"description": "CoreML Model Specification",
"name": "model.mlmodel",
"path": "com.apple.CoreML/model.mlmodel"
},
"4FAA7EEF-8FF4-4185-B2F6-2461CB59E676": {
"author": "com.apple.CoreML",
"description": "CoreML Model Weights",
"name": "weights",
"path": "com.apple.CoreML/weights"
}
},
"rootModelIdentifier": "15F4E827-89F3-4D2A-AFF3-A1609A536263"
}
Original file line number Diff line number Diff line change
@@ -1,16 +1,22 @@
// SPDX-License-Identifier: Apache-2.0
//
// CGImage+resize.swift
// BlurDiscriminatorKit
//
// Created by syjdev on 2021/10/07.
//

import Foundation
import CoreGraphics


extension CGImage {
func makeAspectFitPaddedContext(width: Int, height: Int) -> CGContext? {
/// Returns a `width`×`height` image with `self` aspect-fit into the top-left corner and the
/// remaining area padded with black.
///
/// This reproduces the letterbox preprocessing the model was trained with: the aspect ratio is
/// preserved (no stretching), and the padding is placed at the bottom/right so the content is
/// top-left aligned. ``BlurMapGeometry`` later crops the model output back to this same region.
func makeAspectFitPadded(width: Int, height: Int) -> CGImage? {
guard self.width > 0, self.height > 0,
let context = CGContext(
data: nil,
Expand All @@ -32,6 +38,7 @@ extension CGImage {
let scaledWidth = CGFloat(self.width) * scale
let scaledHeight = CGFloat(self.height) * scale

// Context origin is bottom-left, so `y = height - scaledHeight` pins the content to the top.
let drawRect = CGRect(x: 0,
y: CGFloat(height) - scaledHeight,
width: scaledWidth,
Expand All @@ -40,6 +47,6 @@ extension CGImage {
context.interpolationQuality = .high
context.draw(self, in: drawRect)

return context
return context.makeImage()
}
}
14 changes: 0 additions & 14 deletions Demo/iOS/BlurDiscriminatorKit/BlurDiscriminatorKit/Constants.swift

This file was deleted.

Loading