Prepare password hashes from a flat file

A password hash only works on the target if it is read exactly as the source computed it: the right algorithm, the right encoding, and the salt, if any, put on the right side of the password. Get one of them wrong and the user is imported without error but can never sign in.

This guide shows how to describe your file’s hashes in a mapping.yaml, use iamigrate prepare to check them, and confirm a hash independently. For the mapping format itself, see the prepare reference.

What to find out about your hashes

Look at one row of your file and answer:

1. Does the hash describe itself? bcrypt ($2b$10$...), PHC strings ($argon2id$v=19$..., $pbkdf2-sha256$i=...) and LDAP ({SSHA}...) do: map the column to password and you are done. A bare digest (32 hex characters, a base64 string) doesn’t.
2. Which algorithm? md5, sha1, sha256, sha512… in a column, spelled however the source spells it (SHA-256 is fine), or the same for everyone.
3. Which encoding? Hex or base64. Base64 may use the URL-safe alphabet and have no = padding; both are accepted.
4. Is there a salt? If so, where is it stored, how is it encoded (hex, base64 or plain text), and was it hashed before or after the password? Ask whoever wrote the source’s login code: sha256(salt + password) and sha256(password + salt) are different hashes.

Attention

Only hashes of the form digest(salt + password) or digest(password + salt) can be imported. Schemes that hash twice, add a site-wide secret, or mix the salt in another way are not supported: for those users, give a random password with set-passwords and let them reset it.

Example 1: hex digests, salt in its own column

id,email,hash,salt,salt_order,hash_type
s1,jane@example.com,EFE240D7...6B72,00112233445566778899aabbccddeeff,after,SHA-256
s2,john@example.com,78355d36...dfbf,ffeeddccbbaa99887766554433221100,before,sha512

The algorithm changes from row to row and is spelled differently; the salt is hex; its position is a column that says before or after.

fields:
  - {source: id, cmf: source_id}
  - {source: email, cmf: email}
  - {source: hash, cmf: password}
  - {source: hash_type, cmf: password_algorithm}
  - {cmf: password_encoding, default: hex}
  - {source: salt, cmf: password_salt}
  - {cmf: password_salt_encoding, default: hex}
  - source: salt_order
    cmf: password_salt_position
    values: {before: prefix, after: suffix}

default with no source is a constant for every row. values translates the source’s words into prefix and suffix.

Example 2: base64 digests in nested JSON

[
  {"uid": "b1", "mail": "jane@example.com", "credentials": {"scheme": "MD5", "digest": "DRB9CfW75Ayt495ccenptw=="}},
  {"uid": "b2", "mail": "john@example.com", "credentials": {"scheme": "SHA256", "digest": "y3HNP1mQlTK3gyV_X5fQUCryWCMfnRfL52J5nAu0oPY"}},
  {"uid": "b3", "mail": "ann@example.com", "credentials": null}
]

The first digest is standard padded base64, the second URL-safe and unpadded. Both are fine once the encoding is stated. A null is a user with no password.

fields:
  - {source: uid, cmf: source_id}
  - {source: mail, cmf: email}
  - {source: credentials.digest, cmf: password}
  - {source: credentials.scheme, cmf: password_algorithm}
  - {cmf: password_encoding, default: base64}

Example 3: everything packed in one column

id,email,password
d1,jane@example.com,sha1$Xk29fa$6cb40f8e1040a12cdcba26d36c1679ce8af7eeaf
d2,john@example.com,md5$q8Zr1b$d5747b5512589470f79cf6da9dbeae32

A pattern cuts each field out of the column: the part of the expression in parentheses is what is kept.

fields:
  - {source: id, cmf: source_id}
  - {source: email, cmf: email}
  - {source: password, cmf: password_algorithm, pattern: '^([a-z0-9]+)\$'}
  - {source: password, cmf: password_salt, pattern: '^[a-z0-9]+\$([^$]*)\$'}
  - {source: password, cmf: password, pattern: '\$([0-9a-f]+)$'}
  - {cmf: password_encoding, default: hex}
  - {cmf: password_salt_encoding, default: utf8}
  - {cmf: password_salt_position, default: prefix}

The salt here is the text Xk29fa itself, hashed before the password: utf8 and prefix.

Example 4: several schemes in one column

login,pwd,pwd_type
m1,d8578edf8458ce06fbc5bb76a58c5ca4,1
m2,r4l4sXl7cqz/+VlaWio3PsPZEG0=,2
m3,$2b$06$fIzRDRdEG3f1wFccB8oNku7/bvEb06u7xyNJr219JZ81vG0SiHVLO,3
m4,{SSHA}kDI9SWgzBrBbTHhec0FiIJWjKS0BAgMEBQYHCA==,4

A numeric code picks the scheme. Rows 1 and 2 are raw digests (md5 in hex, sha1 in base64), and rows 3 and 4 describe themselves.

fields:
  - {source: login, cmf: source_id}
  - {source: pwd, cmf: password}
  - source: pwd_type
    cmf: password_algorithm
    values: {"1": md5, "2": sha1, "3": bcrypt, "4": ldap}

No encoding is given, so hex or base64 is detected from the characters. That is safe when the encodings can’t be confused (here a 32-character hex string and a base64 string with =); state the encoding as in the other examples whenever you can.

Run prepare

iamigrate prepare --in users.csv --mapping mapping.yaml --out users-prepared.csv

prepared 3 records -> users-prepared.csv
iamigrate: 1 records will be skipped by export:
  s2: password normalization failed: hash: "78355d36..." is not a md5 digest (16 bytes expected, got 64): is the algorithm or the encoding wrong?

prepare checks every hash the way the import will read it. The usual findings:

Message	Cause
is not a md5 digest (16 bytes expected, got 64)	The algorithm column is wrong for that row, or the digest isn’t encoded as you said
is not valid hex / is not valid base64	The encoding is wrong, or the column holds something else (a prefix, whitespace)
password_salt needs password_salt_encoding	A salt can’t be guessed to be hex, base64 or text: say which
not a valid bcrypt hash	A bare digest labeled bcrypt, argon2… Those carry their own parameters
unknown algorithm "..."	The algorithm isn’t one of the supported names; map it with values

The file is written even when records are flagged, so you can inspect it. Open it: a prepared file has the conventional columns (source_id, email, password, password_algorithm, password_encoding, password_salt, …) and nothing else to interpret. Then export it as usual:

iamigrate export --source flatfile --in users-prepared.csv --out ./export/
iamigrate validate --in ./export/users.cmf.jsonl.gz --target <target>

validate tells you which algorithms the target accepts. A user whose algorithm isn’t accepted can’t be converted, since a hash can’t be re-hashed without the password: use set-passwords for them.

Check a hash yourself

prepare can tell that a hash is well-formed, not that it is the hash of the user’s password. For that, recompute one from a password you know, with tools that have nothing to do with iamigrate, and compare it to the source’s hash. From Example 1, user s1 (password Tr0ub4dor&3, salt hex 0011...eeff hashed after the password):

{ printf '%s' 'Tr0ub4dor&3'; printf '%s' 00112233445566778899aabbccddeeff | xxd -r -p; } | sha256sum
# efe240d7bce44b10807f4f903caed452ddf3028166d397fd4e247bb6d3a46b72  (the same digest as in the file, in lowercase)

From Example 3, user d1 (password swordfish, salt Xk29fa text, hashed before the password):

{ printf '%s' 'Xk29fa'; printf '%s' 'swordfish'; } | sha1sum
# 6cb40f8e1040a12cdcba26d36c1679ce8af7eeaf

If the digest you compute matches the source’s hash, then the algorithm, the salt and the salt position you put in the mapping are the right ones. Change the order, and the digest changes: that is how to find out which one the source used.

The end-to-end proof is the one in Generate test fixtures: after the import, sign a test user in on the target with a password you know.