Cleaning up garbled UTF-8 data
Background
With Swift, I'm trying to get the HTML through URLSession
rather than loading it in first WKWebView
, since I only need HTML and none of the sub-resources. I am facing a problem with some pages that work when loading to WKWebView
, but loading via URLSession
(or even simple NSString(contentsOf: url, encoding String.Encoding.utf8.rawValue)
) UTF-8 conversion fails.
How to reproduce
This fails (prints "nil"):
print(try? NSString(contentsOf: URL(string: "http://www.huffingtonpost.jp/techcrunch-japan/amazon-is-gobbling-whole-foods-for-a-reported-13-7-billion_b_17171132.html?utm_hp_ref=japan&ir=Japan")!, encoding: String.Encoding.utf8.rawValue))
But changing the url on the site home page succeeds:
print(try? NSString(contentsOf: URL(string: "http://www.huffingtonpost.jp")!, encoding: String.Encoding.utf8.rawValue))
Question
How can I "clear" the data returned from a URL that contains invalid UTF-8? I would like to either remove or replace any invalid sequences in invalid UTF-8 so that the rest can be viewed. WKWebView is capable of rendering the page very well (and claims to be UTF-8 content), as you can see by visiting the url: http://www.huffingtonpost.jp/techcrunch-japan/amazon-is-gobbling-whole- foods-for-a-reported-13-7-billion_b_17171132.html? utm_hp_ref = japan & ir = Japan
source to share
Here's an approach to create String
from (possibly) malformed UTF-8 Data:
- Read the site content into an object
Data
. - Add a byte
0
to make it a "C string" - Use
String(cString:)
for conversion. This initializer replaces malformed UTF-8 code unit sequences with the Unicode replacement character ("\u{FFFD}"
). - Optional: Remove all occurrences of the replacement character.
An example for the "cleaning" process:
var data = Data(bytes: [65, 66, 200, 67]) // malformed UTF-8
data.append(0)
let s = data.withUnsafeBytes { (p: UnsafePointer<CChar>) in String(cString: p) }
let clean = s.replacingOccurrences(of: "\u{FFFD}", with: "")
print(clean) // ABC
Swift 5:
var data = Data([65, 66, 200, 67]) // malformed UTF-8
data.append(0)
let s = data.withUnsafeBytes { p in
String(cString: p.bindMemory(to: CChar.self).baseAddress!)
}
let clean = s.replacingOccurrences(of: "\u{FFFD}", with: "")
print(clean) // ABC
This can of course be defined as a custom init method:
extension String {
init(malformedUTF8 data: Data) {
var data = data
data.append(0)
self = data.withUnsafeBytes { (p: UnsafePointer<CChar>) in
String(cString: p).replacingOccurrences(of: "\u{FFFD}", with: "")
}
}
}
Swift 5:
extension String {
init(malformedUTF8 data: Data) {
var data = data
data.append(0)
self = data.withUnsafeBytes{ p in
String(cString: p.bindMemory(to: CChar.self).baseAddress!)
}.replacingOccurrences(of: "\u{FFFD}", with: "")
}
}
Using:
let data = Data(bytes: [65, 66, 200, 67])
let s = String(malformedUTF8: data)
print(s) // ABC
Cleaning can be done "directly" transcode
using
extension String {
init(malformedUTF8 data: Data) {
var utf16units = [UInt16]()
utf16units.reserveCapacity(data.count) // A rough estimate
_ = transcode(data.makeIterator(), from: UTF8.self, to: UTF16.self,
stoppingOnError: false) { code in
if code != 0xFFFD {
utf16units.append(code)
}
}
self = String(utf16CodeUnits: utf16units, count: utf16units.count)
}
}
This is essentially what it String(cString:)
really is, compare
CString.swift and
StringCreate.swift .
Another option is to use the codec method UTF8
decode()
. and ignore errors:
extension String {
init(malformedUTF8 data: Data) {
var str = ""
var iterator = data.makeIterator()
var utf8codec = UTF8()
var done = false
while !done {
switch utf8codec.decode(&iterator) {
case .emptyInput:
done = true
case let .scalarValue(val):
str.unicodeScalars.append(val)
case .error:
break // ignore errors
}
}
self = str
}
}
source to share